VieGem-ASR
5CD-AI/VieGem-ASR
- 作者
- 5CD-AI
- 任务
- 语音识别
- 下载量
- 0
- 点赞数
- 10
- 趋势分
- 4
- 许可证
- cc-by-nc-sa-4.0
- 收录时间
- 2026/11/6
language:
- vi
- en license: cc-by-nc-sa-4.0 library_name: pytorch pipeline_tag: automatic-speech-recognition base_model: google/embeddinggemma-2 tags:
- ctc
- vietnamese
- code-switching
- streaming
- embeddinggemma
- asr metrics:
- wer
VieGem-ASR 🇻🇳💎
Fast Vietnamese and Vietnamese–English speech recognition: a 359M-parameter streaming CTC model on Google's EmbeddingGemma 2 audio encoder. Inspired by Quan Nguyen's QUAN-Transcribe-1.0.
- Fast: about 3,500× real time on one H100 (batched, greedy); streams in 0.48 s chunks with no look-ahead.
- Vietnamese: mean WER 6.34 over 8 common test sets (5.29 with the included n-gram LM).
- Code-switching: ViMedCSS WER 14.3; 66% of English terms written exactly.
Demo
Results
Vietnamese test sets
WER %, lower is better, greedy unless marked "+ LM". VieGem-ASR, QUAN-Transcribe-1.0 and Qwen3-ASR-0.6B were run by us with one normalizer; † = reported in the QUAN-Transcribe-1.0 model card (our scorer matches its numbers within 0.1).
| Test set | What it contains | VieGem-ASR | VieGem-ASR + LM | QUAN-Transcribe-1.0 | Qwen3-ASR-0.6B | Gemini-Flash-3.8 † | ElevenLabs Scribe v2 † | PhoWhisper-large † | Whisper large-v3 † |
|---|---|---|---|---|---|---|---|---|---|
| VIVOS | read speech, quiet room | 6.63 | 4.78 | 7.52 | 10.22 | 9.36 | 9.14 | 4.52 | 9.18 |
| FLEURS vi | read Wikipedia sentences | 7.33 | 5.85 | 7.88 | 8.40 | 5.90 | 5.60 | 5.31 | 7.09 |
| Common Voice 17 | read, crowd-sourced microphones | 8.61 | 6.21 | 9.69 | 16.07 | 10.30 | 10.50 | 9.12 | 18.43 |
| PhoAudiobook | audiobooks | 1.39 | 1.04 | 1.39 | 3.23 | 1.60 | 1.45 | 0.87 | 1.85 |
| LSVSC | large mixed-domain corpus | 5.16 | 4.71 | 5.35 | 7.78 | 5.70 | 6.25 | 6.81 | 13.96 |
| Bud500 | mixed domains, tightly cut clips | 3.08 | 2.75 | 1.82 | 6.79 | 1.80 | 3.15 | 9.51 | 64.03 |
| GigaSpeech2 vi | YouTube, spontaneous | 9.50 | 8.52 | 11.60 | 11.37 | 10.70 | 10.50 | 10.14 | 14.19 |
| ViMD | dialects of 63 provinces | 9.05 | 8.46 | 9.84 | 13.98 | 10.00 | 10.05 | 11.38 | 15.56 |
| Mean of the 8 | 6.34 | 5.29 | 6.89 | 9.73 | 6.92 | 7.08 | 7.21 | 18.04 |
Bold = best greedy result per row. Text is lowercased, without punctuation, numbers as words.
Vietnamese–English code-switching (ViMedCSS)
| System | Test WER | Hard WER |
|---|---|---|
| Whisper-large-v3 (paper) | 34.47 | 39.35 |
| PhoWhisper-large (paper) | 31.24 | 37.37 |
| PhoWhisper-small + AG, fine-tuned (paper) | 23.67 | 33.73 |
| QUAN-Transcribe-1.0 | 22.26 | 26.56 |
| VieGem-ASR | 14.30 | 21.08 |
| VieGem-ASR + LM | 13.00 | 19.84 |
"Paper" = numbers from the ViMedCSS paper (its own normalization, so approximate).
Streaming and speed
- Streaming (0.48 s chunks): VIVOS 8.73, CV17 10.84, FLEURS 9.02, LSVSC 5.84, ViMD 10.00, ViMedCSS 15.99.
- A 10-minute video is transcribed in 0.49 s on one H100 (median of 20 runs, ~1,220× real time).
Usage
import torch, soundfile as sf
from transformers import AutoTokenizer
from asr.model import CTCModel, ModelConfig, greedy_decode # asr/ folder of this repo
from asr.text import Units
ck = torch.load("viegem_asr.pt", map_location="cpu")
cfg = ModelConfig.from_dict(ck["model_config"])
units = Units(ck["gemma_ids"])
tok = AutoTokenizer.from_pretrained(cfg.encoder_id) # google/embeddinggemma-2 (accept its terms)
model = CTCModel(cfg, units.gemma_ids)
model.load_state_dict(ck["model"], strict=False)
model = model.cuda().eval()
wav, sr = sf.read("audio.wav", dtype="float32") # 16 kHz mono, ≤ 30 s
x = torch.from_numpy(wav)[None].cuda()
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
logits, lens = model(x, torch.tensor([x.shape[1]], device="cuda"), streaming=False) # streaming=True: 0.48 s chunks
print(units.decode(tok, greedy_decode(logits, lens)[0]))
With the LM (pip install kenlm):
from asr.lm_decode import LMDecoder
words = open("lm/viegem_lm.unigrams.txt", encoding="utf-8").read().split()
dec = LMDecoder("lm/viegem_lm.bin", unigrams=words, alpha=0.35, beta=1.5)
pieces = ["", ""] + tok.convert_ids_to_tokens(units.gemma_ids)
lp = torch.log_softmax(logits[0, : lens[0]].float(), -1).cpu().numpy()
print(units.decode(tok, dec.decode(lp, pieces, beam_width=32)))
Model
- Encoder: EmbeddingGemma 2 audio tower (12 conformer layers, 25 frames/s); top 4 layers fine-tuned.
- Reader: layer mix + 6 transformer blocks (d=768) with chunked attention, so one model serves offline and streaming.
- Head: 21,031 Gemma tokens (Vietnamese, English, single characters), initialized from Gemma's text embeddings.
- Files:
viegem_asr.pt(fp16 weights, 311 MB),asr/(code),lm/(4-gram KenLM from our training transcripts, 71 MB),demo/.
Training data
About 5,800 hours, every set filtered against all test and dev sentences.
| Type | Sources | Hours |
|---|---|---|
| Audiobooks | PhoAudiobook, InfoRe2 | ~1,900 |
| Social media and YouTube | VietSpeech, viVoice, GigaSpeech2 (subset), VietSuperSpeech | ~2,820 |
| Mixed domains | Bud500, VLSP2020, LSVSC, ViMD, FPT FOSD, InfoRe1, FLEURS, Common Voice 17 | ~780 |
| English | LibriSpeech-100 | ~100 |
| Vietnamese–English (medical) | ViMedCSS | ~23 |
Limitations
- Output is lowercase, without punctuation; numbers are written as words.
- Clips cut in the middle of a word (as in Bud500) lose the last word more often.
- English is meant for terms mixed into Vietnamese; plain English is weak (LibriSpeech test-clean 10.7).
License
Non-commercial research use (CC-BY-NC-SA-4.0), because the training data includes Bud500, viVoice and PhoAudiobook. Use must also follow the Gemma Terms of Use and Prohibited Use Policy.
Credits
Google DeepMind (EmbeddingGemma 2); Quan Nguyen (QUAN-Transcribe-1.0, the inspiration for this work); the authors of VIVOS, Common Voice, FLEURS, LSVSC, ViMD, Bud500, PhoAudiobook, VietSpeech, viVoice, GigaSpeech2, VietSuperSpeech, VLSP2020, FPT FOSD, InfoRe, ViMedCSS and LibriSpeech. ViMedCSS baselines: Nguyen et al., LREC 2026.
内容来源：Hugging Face 模型卡
在 Hugging Face 查看