license: apache-2.0 pipeline_tag: automatic-speech-recognition library_name: transformers language:

en
fr
de
es
it
pt
nl base_model:
google/gemma-4-26B-A4B-it
openai/whisper-large-v3 tags:
speech
asr
speech-to-text
non-streaming
instruction-following track_downloads: true

SOV-Transcribe-1

SOV-Transcribe-1 is Salesforce's open-weight speech-augmented language model built for non-streaming automatic speech recognition (ASR). It combines leading English transcription accuracy with multilingual support and natural-language control over how speech is transcribed.

The model ranks #1 among open-weight models for non-streaming speech-to-text on Artificial Analysis (#7 overall) and #2 of 23 systems on VoiceCodeBench ( besimple AI ) for entity accuracy and task success as of November 12th, 2026.

Developers and enterprises can deploy and adapt SOV-Transcribe-1 on their own infrastructure, retaining control over their audio data and transcription behavior.

Key features

Accurate entity recognition. Captures details such as names, phone numbers and email addresses that matter in dictation and voice applications.
Robustness to accents and noise. Handles diverse English accents and challenging audio conditions, including background noise.
Transcription style control. Use natural-language instructions to choose verbatim or cleaned-up transcripts and control punctuation, capitalization and number formatting.
Context biasing. Supply names, domain-specific vocabulary or a short description of the topic to help recognize difficult words and technical terms.

Model summary

Property	Details
Total parameters	26.6B total (4.6B active) including the audio encoder
Architecture	Whisper large-v3 audio encoder, audio projector and Gemma 4 26B A4B MoE
Precision	BF16 (MoE router in FP32)
Maximum audio input	Up to 30 minutes (tested)
Inputs / outputs	16 kHz Audio with optional text instructions and context / transcript text
Languages	English, French, German, Spanish, Italian, Portuguese and Dutch
Tasks	Verbatim or written-form transcription, multilingual ASR, style control and contextual transcription
Inference frameworks	vLLM and Hugging Face Transformers
Speculative decoding	DFlash draft model in its own repository, `salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash`
License	Apache 2.0

Benchmark results

Benchmark results for SOV-Transcribe-1

Usage

Native support in vLLM and Hugging Face Transformers is being upstreamed. Until it is released, a vLLM plugin adds native support to vLLM 0.32.0, and this repository ships its modeling code for Transformers.

Installation and hardware requirements

# Transformers
pip install "transformers>=5.19,<5.23" torch librosa accelerate

# vLLM: the plugin in this repository's vllm_plugin/ folder; it installs vLLM 0.32.0
hf download salesforce-sovereign-intelligence/SOV-Transcribe-1 --include "vllm_plugin/*" --local-dir SOV-Transcribe-1
pip install ./SOV-Transcribe-1/vllm_plugin

The hf command comes with pip install huggingface_hub.

Hugging Face Transformers

import librosa
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "salesforce-sovereign-intelligence/SOV-Transcribe-1"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained(model_id, trust_remote_code=True, device_map="auto")

files = ["meeting.wav", "call.mp3"]
audio = [librosa.load(f, sr=16000, mono=True)[0] for f in files]
inputs = processor.apply_transcription_request(audio).to(model.device, dtype=model.dtype)
outputs = model.generate(**inputs, max_new_tokens=8192)
transcripts = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(transcripts[0])

The recordings are transcribed in one batch. apply_transcription_request(audio, prompt="...") takes any instruction from the prompt guide, or a list with one per recording.

For speculative decoding, load the DFlash draft from its own repository as the assistant model. Speculative decoding in Transformers runs one recording at a time:

from transformers import AutoModel

draft = AutoModel.from_pretrained(
    "salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash", trust_remote_code=True, dtype=model.dtype, device_map=model.device
)
for a in audio:
    inputs = processor.apply_transcription_request(a).to(model.device, dtype=model.dtype)
    outputs = model.generate(**inputs, assistant_model=draft, max_new_tokens=8192)
    print(processor.decode(outputs[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

If your environment does not allow remote code, copy this self-contained function instead:

def transcribe_hf(model, audio_files, prompt="Transcribe the audio verbatim, outputting only spoken words in sequence."):
    """
    Transcribe audio files (up to 30 minutes each) with Transformers; returns one transcript
    per file. model: a local folder or a Hugging Face repo id.
    """
    import json
    import math
    from pathlib import Path

    import librosa
    import torch
    from huggingface_hub import snapshot_download
    from safetensors import safe_open
    from transformers import (AutoConfig, AutoModelForCausalLM, AutoTokenizer, GenerationConfig, WhisperConfig,
                              WhisperFeatureExtractor)
    from transformers.models.gemma4.modeling_gemma4 import Gemma4TextRouter
    from transformers.models.whisper.modeling_whisper import WhisperEncoder

    model = Path(model).expanduser() if Path(model).expanduser().is_dir() else Path(snapshot_download(model))
    config = json.loads((model / "config.json").read_text())
    tokens = json.loads((model / "processor_config.json").read_text())
    tokenizer = AutoTokenizer.from_pretrained(model)
    features = WhisperFeatureExtractor.from_pretrained(model)
    index = json.loads((model / "model.safetensors.index.json").read_text())["weight_map"]

    def weights(prefix):
        state = {}
        for shard in sorted({v for k, v in index.items() if k.startswith(prefix)}):
            with safe_open(str(model / shard), framework="pt") as f:
                state.update({k[len(prefix):]: f.get_tensor(k) for k in f.keys() if k.startswith(prefix)})
        return state

    # Decoder: Gemma 4's text model, built from its config and filled shard by shard, as the
    # training model builds it. Its MoE router runs in FP32, as in training (router_dtype); the
    # checkpoint stores the router in FP32, so it is converted before the weights are loaded.
    text_config = dict(config["text_config"])
    text_config = AutoConfig.for_model(text_config.pop("model_type"), **text_config)
    with torch.device("cuda"):
        decoder = AutoModelForCausalLM.from_config(text_config, dtype=torch.bfloat16)
    # Rotary frequencies computed on the CPU, as in training (on the GPU their last bits differ).
    decoder.model.rotary_emb = type(decoder.model.rotary_emb)(text_config).to("cuda")
    for router in decoder.modules():
        if isinstance(router, Gemma4TextRouter):
            router.float()
            router.forward = lambda hidden, forward=router.forward: forward(hidden.float())
    for shard in sorted(set(index.values())):
        with safe_open(str(model / shard), framework="pt") as f:
            part = {("model." + k[len("model.language_model."):] if k.startswith("model.language_model.") else k):
                    f.get_tensor(k) for k in f.keys() if k.startswith(("model.language_model.", "lm_head."))}
        decoder.load_state_dict(part, strict=False)
    decoder.eval()
    # Whisper encoder and projector (stack 4 encoder frames -> Linear -> GELU -> Linear).
    audio_config = WhisperConfig(**config["audio_config"])
    audio_config._attn_implementation = "sdpa"
    encoder = WhisperEncoder(audio_config)
    encoder.load_state_dict(weights("model.audio_tower."))
    encoder = encoder.to("cuda", torch.bfloat16).eval()
    factor, width, hidden = config["downsample_factor"], audio_config.d_model, text_config.hidden_size
    projector = torch.nn.Sequential()
    projector.add_module("linear_1", torch.nn.Linear(width * factor, hidden, bias=False))
    projector.add_module("act", torch.nn.GELU())
    projector.add_module("linear_2", torch.nn.Linear(hidden, hidden, bias=False))
    projector.load_state_dict(weights("model.embed_audio."))
    projector = projector.to("cuda", torch.bfloat16).eval()
    tokens_per_window = features.nb_max_frames // 2 // factor  # 375 per 30 s window
    generation = GenerationConfig.from_pretrained(model)  # greedy; the training stop tokens
    generation.max_new_tokens = 8192

    transcripts = []
    for path in audio_files:
        # Pad to whole 30 s windows, extract features over the full waveform, then split by window.
        audio, _ = librosa.load(path, sr=16000, mono=True)
        windows = math.ceil(len(audio) / features.n_samples)
        mel = features([audio], sampling_rate=16000, padding="max_length", max_length=windows * features.n_samples,
                       truncation=False, return_tensors="pt")["input_features"][0]
        mel = mel.reshape(mel.shape[0], windows, features.nb_max_frames).permute(1, 0, 2).to("cuda", torch.bfloat16)
        # Chat prompt; its one audio marker becomes begin + 375 audio tokens per window + end.
        conversation = [{"role": "user", "content": [{"type": "text", "text": prompt}, {"type": "audio"}]}]
        text = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True,
                                             enable_thinking=False)
        span = tokens["audio_begin_token"] + tokens["audio_token"] * (windows * tokens_per_window) + tokens["audio_end_token"]
        ids = torch.tensor([tokenizer.encode(text.replace(tokens["audio_token"], span), add_special_tokens=False)],
                           device="cuda")
        with torch.inference_mode():
            # Encoder in batches of 8 windows, as the training model ran it.
            audio_embeds = torch.cat([projector(encoder(part).last_hidden_state.reshape(
                len(part), -1, width * factor)) for part in mel.split(8)])
            embeds = decoder.get_input_embeddings()(ids)  # Gemma scales token embeddings itself
            embeds = embeds.masked_scatter(ids.eq(config["audio_token_id"])[..., None], audio_embeds.reshape(-1))
            output = decoder.generate(inputs_embeds=embeds, attention_mask=torch.ones_like(ids),
                                      generation_config=generation)
        transcripts.append(tokenizer.decode(output[0], skip_special_tokens=True).strip())
    return transcripts

transcripts = transcribe_hf("salesforce-sovereign-intelligence/SOV-Transcribe-1", ["meeting.wav"])

vLLM

Recommended. With the plugin installed (vllm_plugin/ in this repository), vLLM loads this repository natively: offline with LLM, or as an OpenAI-compatible server with vllm serve.

from vllm import LLM, SamplingParams

llm = LLM(model="salesforce-sovereign-intelligence/SOV-Transcribe-1", max_model_len=32768, limit_mm_per_prompt={"audio": 1},
          allowed_local_media_path="/path/to/audio")

prompt = "Transcribe the audio verbatim, outputting only spoken words in sequence."
files = ["/path/to/audio/meeting.wav", "/path/to/audio/call.mp3"]  # up to 30 minutes each
messages = [[{"role": "user", "content": [{"type": "text", "text": prompt},
                                          {"type": "audio_url", "audio_url": {"url": f"file://{f}"}}]}] for f in files]
outputs = llm.chat(messages, SamplingParams(temperature=0.0, max_tokens=8192),
                   chat_template_kwargs={"enable_thinking": False})
print(outputs[0].outputs[0].text)

All files are transcribed in one batch. As a server, with the OpenAI-compatible transcription endpoint:

vllm serve salesforce-sovereign-intelligence/SOV-Transcribe-1 --max-model-len 32768
curl http://localhost:8000/v1/audio/transcriptions -F model=salesforce-sovereign-intelligence/SOV-Transcribe-1 -F file=@meeting.wav -F temperature=0

The endpoint uses the verbatim instruction by default; pass -F prompt="..." for any other from the prompt guide. It accepts up to 600 seconds of audio per request unless VLLM_MAX_AUDIO_DECODE_DURATION_S is raised (1800 for 30 minutes).

Speculative decoding

The DFlash draft model, in its own repository salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash, proposes 15 tokens at a time for the main model to verify. For one request at a time it speeds up decoding about 2–3× (less for clips under 10 seconds), with accuracy within run-to-run variation of decoding without it. Enable it in LLM(...):

llm = LLM(model="salesforce-sovereign-intelligence/SOV-Transcribe-1", max_model_len=32768, limit_mm_per_prompt={"audio": 1},
          allowed_local_media_path="/path/to/audio", max_num_seqs=64,
          speculative_config={"method": "dflash", "model": "salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash", "num_speculative_tokens": 15})

or for the server:

vllm serve salesforce-sovereign-intelligence/SOV-Transcribe-1 --max-model-len 32768 --max-num-seqs 64 \
  --speculative-config '{"method": "dflash", "model": "salesforce-sovereign-intelligence/SOV-Transcribe-1-DFlash", "num_speculative_tokens": 15}'

max_num_seqs=64 keeps vLLM's start-up memory profiling within a 96 GB GPU; vLLM's default of 1,024 concurrent sequences runs out of memory with the draft.

Mean time per request on one NVIDIA RTX PRO 6000 Blackwell, 96 GB, one request at a time:

Audio length	without draft	with draft
up to 10 s	0.16 s	0.09 s
10–30 s	0.44 s	0.15 s
30–60 s	0.88 s	0.31 s
1–5 min	2.78 s	1.26 s

vLLM 0.32.0, BF16, greedy decoding, measured with an earlier vLLM integration of the model; clips cut from Earnings-22 calls.

Serving performance

With native vLLM support (vllm serve and its OpenAI-compatible transcription endpoint), one RTX PRO 6000 Blackwell Server Edition GPU transcribes up to 1,039 seconds of audio per second (BF16 with the DFlash draft) and returns the transcript of a 30-second recording in 0.22 seconds. We aim to upstream native support to vLLM and Hugging Face Transformers.

Serving performance of SOV-Transcribe-1 on one RTX PRO 6000 Blackwell

Measured with the same model implementation as the plugin, on a vLLM development build (November 2026).

Prompt guide

Place the instruction in the user message. Replace {language} with the language, and {keywords} with a comma-separated list of terms.

Goal	User message
Verbatim transcript	`Transcribe the audio verbatim, outputting only spoken words in sequence.`
Using keywords	`This audio may contain the following words or phrases: {keywords}. Transcribe the audio into text.`
Automatic language	`Transcribe the audio verbatim in the language's standard writing system, using punctuation and capitalization. Output only the transcript.`
Language hint	`Transcribe the {language} audio into text, ensuring all punctuation marks are included.`

Evaluation protocol and reproducibility

Public benchmark reproduction uses the vLLM setup above with greedy decoding and DFlash disabled, following the dataset-specific prompts and scoring procedures below, with repetition cleanup where applicable.

Dataset / condition	Prompt	Metric
Open ASR: AMI, Earnings22, GigaSpeech, URGENT24 and VoxPopuli	`Transcribe the audio verbatim, including fillers and repetitions. Use punctuation and capitalization. Output only the transcript.`	Word error rate (WER) ↓
AppTek	`Transcribe the audio verbatim, including fillers and repetitions. Use punctuation and capitalization. Output only the transcript.`	WER ↓
HALAS	`Transcribe the audio verbatim, including fillers and repetitions. Use punctuation and capitalization. Output only the transcript.`	WER ↓
Open ASR: Monsoon English	`Transcribe the audio verbatim, including fillers and repetitions. Use punctuation and capitalization. Output only the transcript. Use Latin script only.`	WER
Open ASR: SPGISpeech	`Transcribe the audio, omitting fillers and accidental repetitions while preserving the remaining words. Use punctuation and capitalization. Output only the transcript.`	WER
Open ASR: multilingual	`Transcribe the audio verbatim in the language's standard writing system, using punctuation and capitalization. Output only the transcript.`	WER · normalizer
ContextASR: no context	`Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER
Contextual Earnings-22: no context	`Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER · normalizer
ContextASR: coarse context	`This audio belongs to the {domain_label} field. Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER
ContextASR: fine context	`This audio belongs to the {domain_label} field and may contain the following words or phrases: {entity_list}. Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER
Contextual Earnings-22: local keywords	`This audio may contain the following words or phrases: {keyword_list}. Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER · normalizer
Contextual Earnings-22: global keywords	`This audio may contain the following words or phrases (some may not be present in the clip): {keyword_list}. Transcribe the English audio into text, ensuring all punctuation marks are included.`	WER · normalizer
Preference-ASR: default	`Transcribe this recording.`	Preference-aware WER ↓ · normalizer
Preference-ASR: instructed (prompt varies by sample; one example shown)	`Perform speech recognition for provided audio. Output the transcription. Inverse Normalize numbers to written form. Example: In written form, use numerical digits as is instead of in words like 3rd for third and 22 for twenty two.`	Preference-aware WER ↓ · normalizer

Best Practices

Be explicit about style. Specify verbatim or cleaned-up text, number formatting and whether to preserve fillers.
Keep context focused. Provide relevant names, terms or a short domain description as hints.
Use the model's preprocessing and template. Prepare 16 kHz mono audio, place the instruction in the user message and keep thinking disabled.
Sampling parameters. Use greedy decoding: temperature=0, top_p=1.0, top_k=0, and repetition_penalty=1.0 in vLLM (do_sample=False in Transformers).
Allow enough output tokens. For long recordings, split at speech boundaries when needed and concatenate in order. Check for truncated or repetitive output.
Repetition cleanup. In rare cases, the model keeps repeating a character or phrase until it reaches the token limit. We remove these loops with a simple post-processing step adapted from the Qwen3-ASR toolkit.

License

Apache 2.0.