voicetut-egyptian-tashkeel

VoiceTut-TTS (OmniVoice 0.6B, Egyptian Arabic) fine-tuned on Rabe3/egyptian-arabic-tts-diacritized: 17,546 Egyptian Arabic clips (58.6 hours, 9 speakers) whose text carries short vowels (tashkeel) on about half of the words. Write the vowels in the text and the model reads them. The voice prompt was kept in training, so VoiceTut's 19 built-in voices and cloning from a reference clip still work.

Listen

The sentences are the ones on the card of the emotion model. The vowels in the table were written by hand (every word). Plain text without vowels works as before.

Morning with coffee (voice: Ahmed)

#	Text (with vowels)	This model, vowels on every word	This model, vowels on every other word
1	صَبَاحْ الْخِيرْ، انَا لِسَّهْ مِشْ قَادِرْ افْتَحْ عَيْنِي.	صَبَاحْ الْخِيرْ، انَا لِسَّهْ مِشْ قَادِرْ افْتَحْ عَيْنِي.	صَبَاحْ الخير، انَا لسه مِشْ قادر افْتَحْ عيني.
2	اهْ، رِيحِةْ الْقَهْوَةْ دِي بِتْصَحِّي الْوَاحِدْ.	اهْ، رِيحِةْ الْقَهْوَةْ دِي بِتْصَحِّي الْوَاحِدْ.	اهْ، ريحة الْقَهْوَةْ دي بِتْصَحِّي الواحد.
3	اوِّلْ رَشْفِةْ كِدَهْ وِالدُّنْيَا اتْحَسَّنِتْ خَالِصْ.	اوِّلْ رَشْفِةْ كِدَهْ وِالدُّنْيَا اتْحَسَّنِتْ خَالِصْ.	اوِّلْ رشفة كِدَهْ والدنيا اتْحَسَّنِتْ خالص.
4	اسْتَنَّى! هُوَّ السَّاعَةْ بَقِتْ تِسْعَةْ؟	اسْتَنَّى! هُوَّ السَّاعَةْ بَقِتْ تِسْعَةْ؟	اسْتَنَّى! هو السَّاعَةْ بقت تِسْعَةْ؟
5	يَا نَهَارْ ابْيَضْ، الاجْتِمَاعْ بَعْدْ نُصّ سَاعَةْ، لَازِمْ اجْرِي.	يَا نَهَارْ ابْيَضْ، الاجْتِمَاعْ بَعْدْ نُصّ سَاعَةْ، لَازِمْ اجْرِي.	يَا نهار ابْيَضْ، الاجتماع بَعْدْ نص سَاعَةْ، لازم اجْرِي.

Shopping with friends (voice: Mahmoud)

#	Text (with vowels)	This model, vowels on every word	This model, vowels on every other word
1	يَلَّا يَا جَمَاعَةْ، النَّهَارْدَهْ هَنْقَلِبْ الْمُولْ كُلُّهْ!	يَلَّا يَا جَمَاعَةْ، النَّهَارْدَهْ هَنْقَلِبْ الْمُولْ كُلُّهْ!	يَلَّا يا جَمَاعَةْ، النهارده هَنْقَلِبْ المول كُلُّهْ!
2	دِي الْجَاكِيتْ اللِّي كُنْتِي بِتْحَلَّمِي بِيهَا، جَرَّبِيهَا بِسُرْعَةْ.	دِي الْجَاكِيتْ اللِّي كُنْتِي بِتْحَلَّمِي بِيهَا، جَرَّبِيهَا بِسُرْعَةْ.	دِي الجاكيت اللِّي كنتي بِتْحَلَّمِي بيها، جَرَّبِيهَا بسرعة.
3	ايهْ دَهْ؟ سِعْرْهَا تَلَاتْ تَلَافْ جِنِيهْ؟	ايهْ دَهْ؟ سِعْرْهَا تَلَاتْ تَلَافْ جِنِيهْ؟	ايهْ ده؟ سِعْرْهَا تلات تَلَافْ جنيه؟
4	لَا طَبْعًا، دَهْ انَا مُمْكِنْ اشْتِرِي بِيهْ عَرَبِيَّةْ.	لَا طَبْعًا، دَهْ انَا مُمْكِنْ اشْتِرِي بِيهْ عَرَبِيَّةْ.	لَا طبعا، دَهْ انا مُمْكِنْ اشتري بِيهْ عربية.
5	مَبْسُوطَةْ اوِي اننَا خَرَجْنَا سَوَا، وِحْشِتُونِي.	مَبْسُوطَةْ اوِي اننَا خَرَجْنَا سَوَا، وِحْشِتُونِي.	مَبْسُوطَةْ اوي اننَا خرجنا سَوَا، وحشتوني.

Joking with friends (voice: Tareq)

#	Text (with vowels)	This model, vowels on every word	This model, vowels on every other word
1	يَا عَمّ انْتَ كُلّ مَرَّةْ تِقُولْ هَعْمِلْ دَايِتْ وِتْخَلَّصْ طَبَقْ الْكُشَرِي لِوَحْدَكْ.	يَا عَمّ انْتَ كُلّ مَرَّةْ تِقُولْ هَعْمِلْ دَايِتْ وِتْخَلَّصْ طَبَقْ الْكُشَرِي لِوَحْدَكْ.	يَا عم انْتَ كل مَرَّةْ تقول هَعْمِلْ دايت وِتْخَلَّصْ طبق الْكُشَرِي لوحدك.
2	اهْ طَبْعًا، انْتَ اللِّي هَتِكْسَبْ الْمَاتْشْ، دَهْ انْتَ بِتْنَهَّجْ مِنْ السِّلِّمْ.	اهْ طَبْعًا، انْتَ اللِّي هَتِكْسَبْ الْمَاتْشْ، دَهْ انْتَ بِتْنَهَّجْ مِنْ السِّلِّمْ.	اهْ طبعا، انْتَ اللي هَتِكْسَبْ الماتش، دَهْ انت بِتْنَهَّجْ من السِّلِّمْ.
3	اسْتَنُّوا اسْتَنُّوا، لِسَّهْ الْحِتَّةْ الْحِلْوَةْ فِي الْحِكَايَةْ جَايَةْ!	اسْتَنُّوا اسْتَنُّوا، لِسَّهْ الْحِتَّةْ الْحِلْوَةْ فِي الْحِكَايَةْ جَايَةْ!	اسْتَنُّوا استنوا، لِسَّهْ الحتة الْحِلْوَةْ في الْحِكَايَةْ جاية!
4	خَلَاصْ وَاللهِ ضَحَّكْتُونِي، بَطْنِي وَجَعِتْنِي مِنْ الضِّحِكْ.	خَلَاصْ وَاللهِ ضَحَّكْتُونِي، بَطْنِي وَجَعِتْنِي مِنْ الضِّحِكْ.	خَلَاصْ والله ضَحَّكْتُونِي، بطني وَجَعِتْنِي من الضِّحِكْ.
5	يَا جَمَاعَةْ رَكِّزُوا بَقَى، الاكْلْ هَيِبْرَدْ!	يَا جَمَاعَةْ رَكِّزُوا بَقَى، الاكْلْ هَيِبْرَدْ!	يَا جماعة رَكِّزُوا بقى، الاكْلْ هيبرد!

Remembering someone who passed (voice: Mariam)

#	Text (with vowels)	This model, vowels on every word	This model, vowels on every other word
1	اللهْ يِرْحَمْهَا، كَانِتْ اطْيَبْ وَاحْدَةْ فِي الْعِيلَةْ كُلِّهَا.	اللهْ يِرْحَمْهَا، كَانِتْ اطْيَبْ وَاحْدَةْ فِي الْعِيلَةْ كُلِّهَا.	اللهْ يرحمها، كَانِتْ اطيب وَاحْدَةْ في الْعِيلَةْ كلها.
2	فَاكْرِينْ لَمَّا كَانِتْ بِتِسْتَنَّانَا عَلَى الْبَابْ كُلّ جُمْعَةْ؟	فَاكْرِينْ لَمَّا كَانِتْ بِتِسْتَنَّانَا عَلَى الْبَابْ كُلّ جُمْعَةْ؟	فَاكْرِينْ لما كَانِتْ بتستنانا عَلَى الباب كُلّ جمعة؟
3	مَكُنْتِشْ مِتْخَيِّلْ انْ الْبِيتْ هَيْبْقَى فَاضِي كِدَهْ مِنْ غَيْرْهَا.	مَكُنْتِشْ مِتْخَيِّلْ انْ الْبِيتْ هَيْبْقَى فَاضِي كِدَهْ مِنْ غَيْرْهَا.	مَكُنْتِشْ متخيل انْ البيت هَيْبْقَى فاضي كِدَهْ من غَيْرْهَا.
4	ادْعُولْهَا كْتِيرْ، هِيَّ مُحْتَاجَةْ دُعَانَا دِلْوَقْتِي.	ادْعُولْهَا كْتِيرْ، هِيَّ مُحْتَاجَةْ دُعَانَا دِلْوَقْتِي.	ادْعُولْهَا كتير، هِيَّ محتاجة دُعَانَا دلوقتي.
5	رَبِّنَا يِصَبَّرْنَا وِيجَمَّعْنَا بِيهَا فِي الْجَنَّةْ.	رَبِّنَا يِصَبَّرْنَا وِيجَمَّعْنَا بِيهَا فِي الْجَنَّةْ.	رَبِّنَا يصبرنا وِيجَمَّعْنَا بيها فِي الجنة.

Usage

Install OmniVoice from github.com/k2-fsa/OmniVoice, then:

import json, torch, soundfile as sf
from huggingface_hub import snapshot_download
from omnivoice import OmniVoice

refs = snapshot_download("mohammedaly22/VoiceTut-TTS", allow_patterns=["reference_speakers/*"])
voice = json.load(open(f"{refs}/reference_speakers/references.json", encoding="utf-8"))[0]    # any of the 19 built-in voices, or your own clip and its transcript
model = OmniVoice.from_pretrained("masterofaudio2077/voicetut-egyptian-tashkeel", device_map="cuda:0", dtype=torch.float16)
text = "صَبَاحْ الْخِيرْ، انَا لِسَّهْ مِشْ قَادِرْ افْتَحْ عَيْنِي."      # vowels on some or all words
audio = model.generate(text=text, language="arz", ref_audio=f"{refs}/{voice['audio_path']}", ref_text=voice["reference_text"])[0]
sf.write("out.wav", audio, 24000)

Vowels can be added to any text with a diacritizer. The training text had vowels on about half of the words (median 0.52), so partly voweled text is the case this model was trained for.

Results

24 held-out sentences read with the 19 built-in voices in turn, plus 20 pairs of words with the same consonants and different vowels (two voices), scored on the output audio.

	VoiceTut-TTS	This model
Character error rate (Whisper large-v3, vowels ignored)	0.046	0.045
Diacritic error rate vs the marks in the input	0.062	0.080
Same, without word-final marks	0.041	0.069
Word pairs where the heard vowels match the input	0.84	0.55
Speaker similarity to the voice's reference clip (WavLM)	0.944	0.836
Lines where the model returned no audio	0	9

The "heard" vowels come from Cohere-Speech-Tashkeel-2B, which has its own error (6.6% DER on its test set), so the rates are relative, not absolute.

Training

Full fine-tune of VoiceTut-TTS, 13500 steps on 2x T4, Lion optimizer (lr 3e-6, weight decay 0.03), fp16. This is step 10000.
17,546 clips, 58.6 h, text column text_merged; 396 clips held out (their sentences removed from training). Hours per speaker:
lahgtna_eg_spk00000: 6.6 h
lahgtna_eg_spk00001: 6.5 h
lahgtna_eg_spk00002: 6.5 h
lahgtna_eg_spk00003: 6.5 h
lahgtna_eg_spk00004: 6.5 h
lahgtna_eg_spk00005: 6.5 h
lahgtna_eg_spk00006: 6.5 h
masri_3shwa: 6.5 h
masri_7akawi: 6.5 h
OmniVoice's stock voice prompt (first 0-30% of a clip) and mask ratio 0-1; no instruction text.

Limitations

The vowels in the training text are machine-made (Cohere-Speech-Tashkeel-2B) and cover about half of the words; the model learned that labelling, including its mistakes.
lahgtna clips are 16 kHz upsampled to 24 kHz; masri is natively 24 kHz.
Few speakers (the training data has 9); the other voices come from VoiceTut's own training.
Not for impersonating a real person or deceiving listeners.

Acknowledgements

VoiceTut-TTS, OmniVoice by k2-fsa, the Higgs audio tokenizer v2, Rabe3/egyptian-arabic-tts-diacritized and its upstream sources, and Cohere-Speech-Tashkeel-2B.