Worklog for task "Develop a custom Vietnamese text-to-speech pipeline and audio dictionary"

Full Interim Summary on TTS and Local Vietnamese Voiceover

This work relates to the task «Develop a custom Vietnamese voiceover pipeline and audio dictionary».

Initial Problem

Initially, browser speechSynthesis was considered, followed by generation via OpenRouter + Google Gemini.

Browser speechSynthesis for Vietnamese proved unsuitable as a basic mechanism for the course: voice installation is opaque, depends on the specific browser/provider, can freeze on Downloading voices ..., and requiring users to manually install language voices is not acceptable.

Next, a TTS pipeline was implemented via OpenRouter. Overall, it works for regular text, but a critical limitation was discovered with minimal inputs: when trying to voice a single separate letter, OpenRouter/Gemini returns HTTP 400 or fails to produce a usable result. For regular TTS this might be acceptable, but for a primer, it is a blocking problem because we need individual letters, graphemes, short syllables, and minimal phonetic units.

Because of this, a separate technical branch was created: «Research and implement local TTS voiceover via a local server».

---

Transition to Local Generation

The VieNeu-TTS project has been set up locally in Docker.

Version 2 is currently being used, which was the subject of previous experiments; a newer version 3 also exists.

Key practical result: VieNeu-TTS is able to stably and clearly voice individual Vietnamese letters/graphemes. This fundamentally distinguishes it from the current Gemini pipeline.

Why This Matters Specifically for a Primer

For Vietnamese, we need not just a universal "text reader", but a generator of minimal educational units:

• individual vowels;
• individual graphemes with tones;
• short syllables;
• diphthongs/triphthongs;
• minimal contrasts;
• individual words;
• followed by short phrases.

In other words, the TTS must support targeted requests like "voice this specific grapheme/syllable," rather than requiring sentence context.

VieNeu-TTS already demonstrates a significant advantage on this criterion.

The theoretical basis regarding which specific units need to be voiced is documented in the concepts:

• Vietnamese alphabet and orthography;
• Phonetics and sound system of the Vietnamese language;
• Tones in the Vietnamese language;
• Structure of the Vietnamese syllable, diphthongs, and triphthongs.

Example: The 18 Forms of Group A

For the basic Latin form a in Vietnamese, there are three different vowels:

• a;
• ă;
• â.

Each of them can carry one of six tones, yielding 18 written forms:

• a á à ả ã ạ;
• ă ắ ằ ẳ ẵ ặ;
• â ấ ầ ẩ ẫ ậ.

VieNeu-TTS is capable of separately voicing such minimal units.

For the course, this means a natural structure:

grapheme → saved audio recording.

After generation, each variant can be saved to file storage and reused without querying the TTS on every playback.

Acoustic Value of the Synthetic Set

The spectrogram of the generated set shows that individual utterances are well-separated and possess a distinguishable internal structure.

For further analysis, it is useful to view the 18 variants not as 18 completely independent classes, but as a combination of two factors:

1. vowel quality: a / ă / â;
2. tone: ngang / sắc / huyền / hỏi / ngã / nặng.

Theoretically, this allows analyzing the features separately:

• F1/F2 and duration → which vowel;
• F0 contour, duration, energy, voicing, and phonation features → which tone;
• then combining the result into a specific grapheme.

This can be useful not only for generation, but also as a basis for a synthetic ground truth for future experiments on "sound → phonetic features/grapheme."

Why VieNeu-TTS Looks Promising

The project is specialized specifically for the Vietnamese language, developed in Vietnam, and trained on a large volume of Vietnamese speech. According to available information, version 2 used over 10,000 hours of Vietnamese recordings.

This does not replace validation, but it makes the model much more relevant to our task than a universal multilingual TTS.

Quality Validation

The final quality of individual sounds and tones will be checked with native speakers.

The primary things to check are:

• tone distinguishability;
• correctness of hỏi/ngã;
• brevity of ă/â;
• naturalness of individual letters/syllables out of context;
• regional accent;
• absence of artifacts on very short inputs.

Thus, the trust architecture is currently envisioned as follows:

VieNeu-TTS = main local candidate generator

native speakers = quality validators

our database = stores confirmed/accepted audio recordings

The requirements for this part of the system are also gathered in the concept «Voiceover and audio layer of the Vietnamese language course».

Storage Architecture

Voiceover does not need to be generated every time.

For each educational unit, the following must be stored:

• text/grapheme/syllable;
• unit type;
• audio file;
• engine and model version;
• generation parameters;
• pronunciation variant/dialect if necessary;
• verification status;
• capability for re-generation;
• capability to replace a synthetic recording with a live native speaker recording.

It makes sense to keep statuses along the lines of:

• generated;
• needs_review;
• approved;
• rejected.

Current Practical Conclusion

For Vietnamguru, local VieNeu-TTS currently looks like a much more suitable base mechanism for the primer than Google Gemini via OpenRouter, because it works stably on minimal inputs and can voice individual letters/graphemes.

Even if another engine turns out to be of higher quality for long speech later on, the ability to reliably synthesize minimal units is a separate critical criterion for our project.

The entire methodological framework of the project is gathered in the concept «Vietnamese language: system and learning methodology», and practical course development is carried out in the task «Develop a Vietnamese language learning system for an adult audience».