Vietnamese Model Speech-to-Text combined with Diarization — MOSS-Transcribe-Diarize-Vietnamese-6000h

🔍 Overview

The MOSS Transcribe Diarize Vietnamese 6000h model is a specialized version of the MOSS (OpenMOSS Team) architecture, finetuned specifically for Vietnamese audio transcription with speaker diarization.

This model excels at multi-speaker conversational speech, achieving 11.07% WER on Vietnamese podcast content — a 34.5% improvement over the base pretrained model.

⚙️ Model Information

Base Model: OpenMOSS-Team/MOSS-Transcribe-Diarize
Architecture: MOSS-Transcribe-Diarize 0.9B
Language: Vietnamese
Task: Automatic Speech Recognition (ASR) + Speaker Diarization
Finetuning Data: ~6000 hours of Vietnamese audio dataset public
Optimized for: Multi-speaker conversations (2-3 speakers typical)

🧠 Training Data

The model was finetuned on approximately 6000 hours of high-quality Vietnamese speech collected from various public datasets:

Dataset
VLSP2020	VLSP2021	VLSP2023-voting-pseudo-labeled
VLSP2023	FPT	VIET_BUD500
VietSpeech	FLEURS	VietMed_Labeled
Sub-GigaSpeech2-Vi	ViVoice	Sub-PhoAudioBook

🧪 Evaluation Results

Test Data: VnExpress Podcasts

The test set consists of 150 samples from VnExpress podcasts, specifically:

Source: VnExpress talk show series

Business and entrepreneurship discussions
Expert interviews on technology, economics, and innovation
Multi-speaker conversational format (2-3 speakers)
Professional studio recording quality
Natural conversational Vietnamese with industry-specific terminology

Note: Test data is completely separate from training data. While training used public ASR datasets (VLSP, VietSpeech, etc.), the test set is from real-world VnExpress podcast content to evaluate generalization to new domains.

Performance Results (150 test samples)

Metric	MOSS-Finetuned-6000h	MOSS-Pretrained	Improvement
WER (Word Error Rate)	11.07% ✅	16.91%	-34.5%
CER (Character Error Rate)	7.64% ✅	13.14%	-41.9%
DER (Diarization Error Rate)	5.12% ✅	10.35%	-50.5%

Lower is better

Optimal Use Cases

🎙️ Podcast transcription (Vietnamese)
📻 Radio/interview transcription
🗣️ Multi-speaker meeting transcription
📹 Video content subtitling
🎧 Audio content analysis

Limitations

⚠️ Halucination: Using Whisper encoder, Hallucinations can occur in speech recognition.

⚠️ Speaker Count: Designed for 2-3 speakers; larger groups not extensively tested

📝 Output Format

The model outputs transcriptions with:

Timestamps in [seconds] format
Speaker labels as [S01], [S02], [S03], etc.
Pattern: [start_time][speaker]text[end_time]

Example Output

[0.20][S01]bạn đang lắng nghe talkshow nguy cơ được thực hiện bởi vnexpress và sworld[5.48][6.56][S02]xin chào các bạn, hôm nay chúng ta sẽ bàn về chuyển đổi số[12.30]

🛠️ Usage

Python Example

from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
import torch

# Load model and processor
model_id = "hynt/MOSS-Transcribe-Diarize-Vietnamese-6000h"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id)

# Load audio
import librosa
audio, sr = librosa.load("test.wav", sr=16000)

# Transcribe
inputs = processor(audio, sampling_rate=16000, return_tensors="pt")
with torch.no_grad():
    generated_ids = model.generate(**inputs)

transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(transcription)

Using with SGLang-Omnni, VLLM

Read: https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize for more information.

📄 License

This model is released under the CC BY-NC-SA 4.0 License (Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International).

You are free to:

✅ Share — copy and redistribute the material
✅ Adapt — remix, transform, and build upon the material

Under the following terms:

🚫 NonCommercial — You may not use the material for commercial purposes
↩️ ShareAlike — If you remix or transform, you must distribute under the same license

The VnExpress podcast data is used for research and evaluation purposes under fair use.

🔗 Resources

Base Model: OpenMOSS-Team/MOSS-Transcribe-Diarize
Paper: arXiv:2601.01554