license: apache-2.0 base_model: Qwen/Qwen3-ASR-1.7B pipeline_tag: automatic-speech-recognition library_name: transformers language:

en tags:
automatic-speech-recognition
speech
audio
qwen3-asr
reinforcement-learning
gspo
model-soup
wise-ft
research
oruk model-index:
name: oruk/orukqwen-asr-1.7b results:
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: AMI-Cleaned type: hf-audio/open-asr-leaderboard config: ami_cleaned split: test metrics:
  - type: wer value: 7.92 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: Earnings22-Cleaned-AA-chunked type: ArtificialAnalysis/Earnings22-Cleaned-AA-chunked split: test metrics:
  - type: wer value: 5.88 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: Gigaspeech-Cleaned type: hf-audio/open-asr-leaderboard config: gigaspeech_cleaned split: test metrics:
  - type: wer value: 7.12 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: LS Clean type: hf-audio/open-asr-leaderboard config: librispeech split: test.clean metrics:
  - type: wer value: 1.2 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: LS Other type: hf-audio/open-asr-leaderboard config: librispeech split: test.other metrics:
  - type: wer value: 2.92 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: SPGISpeech type: hf-audio/open-asr-leaderboard config: spgispeech split: test metrics:
  - type: wer value: 2.74 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: URGENT2024 type: hf-audio/open-asr-leaderboard config: urgent2024 split: test metrics:
  - type: wer value: 9.44 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: Voice Arena Monsoon type: VoiceArena/Monsoon_en_IN_test split: test metrics:
  - type: wer value: 3.51 name: Test WER
- task: type: automatic-speech-recognition name: Automatic Speech Recognition dataset: name: Voxpopuli-AA-Cleaned type: hf-audio/open-asr-leaderboard config: voxpopuli_cleaned_aa split: test metrics:
  - type: wer value: 2.07 name: Test WER

orukqwen-asr-1.7b

A research artifact from oruk. orukqwen-asr-1.7b is Qwen3-ASR-1.7B [1] post-trained with our method: reinforcement learning with GSPO [2] and Levenshtein token-level credit assignment, followed by weight-space blending, i.e. a uniform soup of three sibling RL runs [3] interpolated 50/50 with the base model, in the style of WiSE-FT [4]. The RL training ran on a single NVIDIA A100 40GB GPU at a time, about 14 GPU-hours in total (plus a one-time data-selection pass of at most 7.5 GPU-hours), drawing from a 54.6-hour pool of transcribed English audio. Architecture, tokenizer, prompt format and inference code are the same as Qwen/Qwen3-ASR-1.7B-hf; only the weights differ.

Authors: Nathan Roll, Vianney Grenez, Busra Marsan, Irene Yi (oruk)

Open ASR Leaderboard (public H200 reproduction, November 6): nine-set mean WER 4.76, aggregate RTFx 718.84 (artifacts); maintainer private-set evaluation is pending.
Size: 2.04B parameters, bf16 safetensors, transformers -hf layout. License: Apache-2.0 (inherited from the base model).
Research article: Teaching a speech model what counts as wrong on oruk.ai covers the word-aligned rewards, the weight averaging and the experiments behind this release.

Method

1. RL with token-level credit assignment (our implementation)

Rollouts: for each training recording, 64 transcripts are sampled from the current policy (temperature 0.8).
Reward: each transcript gets r = 1 − min(WER, 1) against the reference, computed on normalized text. A per-word normalizer drops fillers and non-speech tags. There is no learned judge.
Advantage: leave-one-out mean baseline, divided by the group's reward standard deviation: A_i = (r_i − mean_{j≠i} r_j) / std(r). This is the group-normalized estimator used in NVIDIA's Nemotron GSPO recipe [5]. Groups in which all 64 rewards are identical are discarded, and each update uses 20 informative groups (1,280 transcripts).
Objective: GSPO-token [2]. It uses the sequence-level importance ratio (geometric mean over tokens), carried per token so that advantages can differ between tokens. The clip range is 3e-4 / 4e-4, the loss is averaged per rollout (1/|y|), and there is no KL or SFT term.
Token-level credit assignment: each sampled transcript is aligned to its reference with a word-level Levenshtein alignment on normalized words.
- In transcripts with positive advantage, the tokens of hypothesis words that are errors (substitutions or inserted words) get zero advantage. So a mostly good transcript does not reinforce the words it got wrong.
- Deleted reference words have no hypothesis token. For those, the first token after the gap (or the end-of-sequence token) is masked instead, so that dropping uncertain words is not rewarded.
- Token spans come from the sampled token ids, never from re-tokenized text. In one of the sibling runs (below), the mirror rule is also applied to negative-advantage transcripts, zeroing the penalty on their correct words.
- This is a lower-variance estimator of a deliberately different (biased) objective. We do not present it as a variance reduction of the original one.
Schedule:
- First run: 400 updates from the base model (AdamW, lr 1e-6 after 10 warmup updates, gradient-norm clip 1.0, 2 optimizer steps per update).
- Then three 150-update continuations with gradient-norm clip 3.0:
  - E2 continued run 1 (lr 1e-6, AdamW state carried over).
  - E6 and E7 each continued from E2's final weights: E6 with lr 2e-6, E7 with the negative-side mirror mask.
- The three runs' final weights are the "siblings" that get souped.
Trained parameters: all LLM decoder blocks, the last two audio-encoder layers and the audio projector (1.44B of 2.04B parameters). Embeddings, lm_head and the other encoder layers are frozen. Training uses bf16 autocast with fp32 master weights.

2. Weight-space blending

Soup: W_soup is the uniform average of the final weights of E2, E6 and E7 (model soup [3]).
Blend: the released weights are W = W_base + 0.5 · (W_soup − W_base), computed in fp32 and stored as bf16. This is a 50/50 interpolation toward the base model in the WiSE-FT style [4]. Frozen tensors are identical to the base model's.

Compute footprint

Every number below comes from the run logs: the Weights & Biases runtimes, which include the periodic evaluations inside each run, and the data-build logs.

stage	hardware	time
Data selection: a one-time 16-sample pilot of the base model over the candidate pool, used to pick the training clips	3× A100 40GB in parallel, ~2.5 h wall clock	≤ 7.5 A100-hours (shared by all runs)
RL run 1, 400 updates	1× A100 40GB	6.67 h
Sibling runs E2 / E6 / E7, 150 updates each	1× A100 40GB each	3.95 h / 1.80 h / 1.84 h (+0.55 h for a crashed first attempt of E7)
Soup + blend	CPU	minutes
Lineage total (training)	one A100 40GB at a time	14.3 A100-hours (14.8 with the crashed attempt), over ~19 h of wall clock

Training audio: a pool of 20,000 clips (54.6 h) of transcribed English speech. The first run drew 11,304 of them (8,000 informative groups); the continuations kept drawing from the rest of the pool in the same fixed order, wrapping around at the end. The audio is used only as RL prompts with references; no training data is redistributed with this model.
Whole study, for context: the three-recipe study this artifact comes from, including every exploratory run, used 87 A100-hours of logged run time (48 W&B runs; training plus in-run evaluation).

Results

HF Jobs H200 reproduction — November 6, 2026

4.76% mean WER and 718.84 RTFx over the nine public scoring columns. The full English short-form public suite was run on HF Jobs, including both LibriSpeech splits and the additional URGENT2024-Clean set. URGENT2024-Clean is excluded from the nine-set mean. These are contributor-run public results; maintainer private-set evaluation is pending.

Raw predictions, scores, checksums, runtime environments, provenance and launcher. Each run is linked below.

Dataset	Segments	WER (%)	RTFx	HF Job
AMI-Cleaned	7,715	7.92	650.77	run
Earnings22-Cleaned-AA-chunked	341	5.88	436.47	run
Gigaspeech-Cleaned	18,757	7.12	706.04	run
LibriSpeech test-clean	2,620	1.20	599.03	run
LibriSpeech test-other	2,939	2.92	561.57	run
SPGISpeech	39,341	2.74	792.07	run
URGENT2024	1,000	9.44	637.74	run
Voice Arena Monsoon	2,102	3.51	532.10	run
Voxpopuli-AA-Cleaned	628	2.07	382.79	run
URGENT2024-Clean (extra)	1,000	4.20	649.53	run
Nine-set mean / aggregate speed	75,443	4.76	718.84

Model revision 5925458e58e7b0ac81b9fc36e56d91800d21d987; harness 74497c5ebf1d01b5d39982a12f5b2c399da90e3a (upstream a768a2c7 plus this PR). Official transformers Space, batch 256 on every set, bf16, sdpa, greedy decoding, max_new_tokens=512, the same language="en" prompt, default warmup_steps=10 (up to ten untimed batches), no compilation. Package versions and dataset revision details are in the artifacts. The standard harness filters apply; Earnings22 is scored at parent-session level.

The normalizer/runtime differ from the original A100 run below. No new paired base-model H200 run is claimed.

Original A100 evaluation — November 3, 2026

Official harness (huggingface/open_asr_leaderboard @ edd684247365, transformers/run_eval.py with the flags of transformers/submit_jobs_qwen3asr.sh, repo normalizer and score_results), greedy decoding, language="en", bf16. WER %. Our base-model run reproduces the published leaderboard numbers to within 0.02 WER points. The Δ column is a paired per-utterance bootstrap (2,000 resamples) vs base.

set	base (Qwen3-ASR-1.7B-hf)	this model	Δ [95% CI]
AMI-Cleaned	8.32	7.92	-0.40 [-0.50, -0.30]*
Earnings22-Cleaned-AA-chunked	5.84	5.85	+0.01 [-0.40, +0.64]
Gigaspeech-Cleaned	7.22	7.11	-0.11 [-0.14, -0.08]*
LS Clean	1.25	1.20	-0.05 [-0.09, -0.01]*
LS Other	2.94	2.89	-0.05 [-0.12, +0.01]
SPGISpeech	2.58	2.75	+0.17 [+0.15, +0.19]*
URGENT2024	9.93	9.48	-0.45 [-0.72, -0.20]*
Voice Arena Monsoon	3.48	3.50	+0.02 [-0.06, +0.10]
Voxpopuli-AA-Cleaned	2.86	2.11	-0.75 [-0.95, -0.56]*
Average (9 public sets)	4.94	4.76	-0.18 [-0.24, -0.10]*
URGENT2024-Clean (extra, not in average)	4.29	4.21
RTFx (9 public sets; higher is faster)	409	404	-1.2%

* the 95% interval excludes zero.

RTFx was measured on 1x NVIDIA A100-SXM4-40GB (driver 580.178.04; base and this model ran on two VMs of the same type, batch 256 with 128 fallback on OOM). The leaderboard measures RTFx on an NVIDIA H200, so absolute RTFx is not comparable.

Held-out English evaluation

This is a held-out English test set from the same sources as training, disjoint by audio hash and recording/speaker group, plus LibriSpeech test-clean and test-other. WER % with the per-word normalizer; paired bootstrap 95% CIs vs base.

test set	utterances	base	ours	Δ [95% CI]
English pooled (18 sets)	3,004	10.09	9.09	-1.00 [-1.20, -0.81]*
English macro (mean of 18 sets)	3,004	10.45	9.48	-0.97 [-1.35, -0.67]*
LibriSpeech test-clean	2,620	2.19	1.89	-0.30 [-0.39, -0.22]*
LibriSpeech test-other	2,939	3.90	3.56	-0.34 [-0.44, -0.24]*
AMI Meeting Corpus (IHM)	200	12.80	9.21	-3.59 [-5.71, -1.94]*
Human Voices: Trust and Intent	13	8.33	5.56	-2.78 [-8.70, +0.00]
SLUE-HVB (Harper Valley Bank)	200	5.89	3.67	-2.22 [-3.27, -1.32]*
ProsodyEmoji	119	8.14	6.12	-2.01 [-4.55, -0.34]*
People's Speech (clean, CC-BY-SA subset)	200	8.75	7.22	-1.54 [-1.98, -1.13]*
DiPCo (Dinner Party Corpus, mixed channel)	184	30.82	29.61	-1.20 [-2.27, -0.32]*
Nationwide Speech Project	200	1.31	0.23	-1.08 [-1.84, -0.46]*
SLURP (real recordings)	200	12.18	11.23	-0.95 [-1.92, +0.00]
CAMEO: JL Corpus	200	7.75	6.88	-0.87 [-1.50, -0.34]*
Switchboard-1 Release 2 + transcripts	200	10.23	9.39	-0.84 [-1.31, -0.39]*
LibriSpeech train-clean-100	200	4.75	4.30	-0.45 [-0.79, -0.13]*
Helsinki Prosody (LibriTTS lineage)	200	2.24	1.98	-0.26 [-0.65, +0.10]
CAMEO: CREMA-D	200	0.42	0.25	-0.17 [-0.52, +0.00]
CAMEO: eNTERFACE	130	27.48	27.37	-0.11 [-0.34, +0.00]
English Dialects (Google crowdsourced UK/IE)	200	4.08	4.00	-0.07 [-0.37, +0.22]
speechocean762	200	11.36	11.36	+0.00 [-0.59, +0.61]
CAMEO: RAVDESS	60	0.83	1.11	+0.28 [+0.00, +0.83]
MINDS-14 (English)	98	30.74	31.06	+0.32 [-0.07, +0.85]

Multilingual check

RL used English audio only, so we checked that the other languages did not regress. We ran the leaderboard's multilingual script (transformers/run_eval_ml.py, language given explicitly, max_new_tokens=256, multilingual scorer) on a small sample: the first 200 test utterances of 5 FLEURS and 2 MLS sets. WER %, paired bootstrap 95% CI vs base.

set (first 200 test utterances)	base	this model	Δ [95% CI]
FLEURS German	2.82	2.76	-0.07 [-0.37, +0.21]
FLEURS French	4.25	4.23	-0.02 [-0.40, +0.37]
FLEURS Spanish	2.30	2.28	-0.02 [-0.21, +0.17]
FLEURS Italian	1.46	1.53	+0.06 [-0.08, +0.21]
FLEURS Portuguese	4.05	3.88	-0.16 [-0.47, +0.12]
MLS Spanish	3.49	3.43	-0.06 [-0.27, +0.14]
MLS French	5.62	5.46	-0.16 [-0.47, +0.15]
mean of 7	3.43	3.37	-0.06

We found no measurable regression. No set changed significantly, and no output was empty or in the wrong script. Language auto-detection (no language hint) still returns English and Chinese correctly on the base model's demo clips. This is a small-sample check, not a full multilingual evaluation.

Quickstart

Transformers

pip install "transformers==5.17.0" torch accelerate soundfile librosa

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "oruk/orukqwen-asr-1.7b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

inputs = processor.apply_transcription_request(
    audio="https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    language="en",
).to(model.device, model.dtype)

output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
print(processor.decode(generated_ids, return_format="transcription_only")[0])

Batch inference (pass a list of audio paths/URLs/arrays; same prompt for every item):

from transformers import AutoProcessor, AutoModelForMultimodalLM

model_id = "oruk/orukqwen-asr-1.7b"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, device_map="auto")

audio = [
    "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav",
    "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
]
inputs = processor.apply_transcription_request(audio, language="en").to(model.device, model.dtype)
output_ids = model.generate(**inputs, max_new_tokens=256)
generated_ids = output_ids[:, inputs["input_ids"].shape[1]:]
for text in processor.decode(generated_ids, return_format="transcription_only"):
    print(text)

vLLM (offline)

pip install "vllm[audio]==0.31.0" librosa

import librosa
from huggingface_hub import hf_hub_download
from vllm import LLM, SamplingParams

llm = LLM(model="oruk/orukqwen-asr-1.7b", max_model_len=2048, limit_mm_per_prompt={"audio": 1})
prompt = "<|im_start|>user\n<|audio_start|><|audio_pad|><|audio_end|><|im_end|>\n<|im_start|>assistant\nlanguage English<asr_text>"
path = hf_hub_download("bezzam/audio_samples", "librispeech_mr_quilter.wav", repo_type="dataset")
audio, sr = librosa.load(path, sr=16000)
out = llm.generate({"prompt": prompt, "multi_modal_data": {"audio": (audio, sr)}},
                   SamplingParams(temperature=0.0, max_tokens=256))
print(out[0].outputs[0].text)

vLLM server (OpenAI-compatible transcription endpoint)

vllm serve oruk/orukqwen-asr-1.7b --max-model-len 2048 --port 8000

curl -s -L -o sample.wav https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav
curl -s http://localhost:8000/v1/audio/transcriptions \
  -F file=@sample.wav -F model=oruk/orukqwen-asr-1.7b -F language=en

Long audio

The model is trained and evaluated on clips of up to 30 s. Longer recordings can still be passed in one call. We tested 70 s, 136 s and 322 s recordings (concatenated LibriSpeech test-clean). With transformers, apply_transcription_request handles them as-is (WER 1.2-2.2%, the same as the base model). With vLLM they work once max_model_len is raised (we used 8192 for 5 minutes; the 2048 in the snippets above is enough for short-form audio). For hour-long audio, split it into segments of up to a few minutes, preferably at pauses.

Timestamps

Word-level timestamps come from the separate forced aligner, exactly as for the base model (Qwen/Qwen3-ForcedAligner-0.6B-hf). The aligner is unchanged and works on this model's transcripts exactly as on the base model's. The snippet below is run verbatim in our deployment tests.

import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM, AutoModelForTokenClassification

asr_model_id = "oruk/orukqwen-asr-1.7b"
aligner_model_id = "Qwen/Qwen3-ForcedAligner-0.6B-hf"
asr_processor = AutoProcessor.from_pretrained(asr_model_id)
asr_model = AutoModelForMultimodalLM.from_pretrained(asr_model_id, device_map="auto")
aligner_processor = AutoProcessor.from_pretrained(aligner_model_id)
aligner_model = AutoModelForTokenClassification.from_pretrained(aligner_model_id, dtype=torch.bfloat16, device_map="auto")

audio_url = "https://huggingface.co/datasets/bezzam/audio_samples/resolve/main/librispeech_mr_quilter.wav"
inputs = asr_processor.apply_transcription_request(audio=audio_url, language="en").to(asr_model.device, asr_model.dtype)
output_ids = asr_model.generate(**inputs, max_new_tokens=256)
transcript = asr_processor.decode(output_ids[:, inputs["input_ids"].shape[1]:], return_format="transcription_only")[0]

aligner_inputs, word_lists = aligner_processor.prepare_forced_aligner_inputs(
    audio=audio_url, transcript=transcript, language="English",
)
aligner_inputs = aligner_inputs.to(aligner_model.device, aligner_model.dtype)
with torch.inference_mode():
    outputs = aligner_model(**aligner_inputs)
timestamps = aligner_processor.decode_forced_alignment(
    logits=outputs.logits, input_ids=aligner_inputs["input_ids"], word_lists=word_lists,
    timestamp_token_id=aligner_model.config.timestamp_token_id,
)[0]
for item in timestamps:
    print(f"{item['text']:<20} {item['start_time']:>8.3f}s -> {item['end_time']:>8.3f}s")

Intended use

English speech-to-text for short-form audio (up to ~30 s per request; see Long audio): meetings, calls, read and conversational speech. The base model is multilingual, while the RL post-training used English audio only (see Limitations). This model is released as a research artifact for studying RL post-training and weight-space blending for ASR. It is not intended for speaker identification or for inferring sensitive attributes from voice.

Limitations

English only. Other languages still work and showed no measurable regression on a small FLEURS/MLS sample (see Multilingual check). They were not optimized, though, and were checked only for de/fr/es/it/pt, so use the base model for non-English work.
Verbatim style and other weak spots: (1) Verbatim style on edited financial transcripts. RL taught the model to transcribe repetitions, false starts and "you know" verbatim. This helps on verbatim-reference sets (AMI −0.40, VoxPopuli −0.75, URGENT −0.45) and costs on SPGISpeech, whose references are edited (+0.17, significant). Earnings22 is unchanged (+0.01, n.s.). (2) Script switching on non-English names. On English speech with Chinese named entities or currency, the model can occasionally write them in Chinese characters ("RMB" → 人民币, Chinese city names). This happened in 5 of the 341 Earnings22 chunks, vs 1 for the base model. If you need Latin-only output, filter or re-decode non-Latin spans. (3) Indian English (Voice Arena Monsoon) is unchanged (+0.02, n.s.).
Formatting: number and formatting conventions follow the base model. The reward was computed on normalized text, so casing and punctuation were not trained.
Long audio: training and evaluation used clips of up to 30 s. In our tests, recordings of up to 5 minutes transcribed well in one call; split longer audio into segments (see Long audio).

License

Apache-2.0, inherited from the base model Qwen/Qwen3-ASR-1.7B. No training data is redistributed with this model.

Citation

@misc{orukqwen2026,
  title  = {orukqwen-asr-1.7b: GSPO with Levenshtein token-level credit assignment and weight-space blending for speech recognition},
  author = {Roll, Nathan and Grenez, Vianney and Marsan, Busra and Yi, Irene},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/oruk/orukqwen-asr-1.7b}},
  note   = {oruk research artifact}
}

References

Qwen Team. Qwen3-ASR Technical Report. arXiv:2601.21337, 2026.
Zheng et al. Group Sequence Policy Optimization. arXiv:2507.18071, 2025.
Wortsman et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022, arXiv:2203.05482.
Wortsman et al. Robust fine-tuning of zero-shot models (WiSE-FT). CVPR 2022, arXiv:2109.01903.
NVIDIA. NVIDIA Nemotron 3: Efficient and Open Intelligence. arXiv:2512.20856, 2025. The group-normalized, leave-one-out GSPO post-training recipe.

Acknowledgements

Built on Qwen3-ASR by the Qwen team. The training audio comes from: People's Speech / MLCommons, LibriSpeech / LibriTTS, the LDC (Switchboard-1, Nationwide Speech Project), AMI, SLURP, SLUE, DiPCo, speechocean762, MINDS-14, CAMEO and others. Evaluation uses the Hugging Face Open ASR Leaderboard harness and datasets.