Worklog for task "Research qualitative programmatic human voice synthesis in the browser"

Web Speech API / SpeechSynthesis as a Separate Solution Class

As part of the search for existing implementations, the built-in browser SpeechSynthesis was tested.

Minimal example:

const utterance = new SpeechSynthesisUtterance('Hello, this is speech synthesis!');
utterance.lang = 'en-US';
const voices = speechSynthesis.getVoices();
utterance.voice = voices.find(v => v.lang === 'en-US');
speechSynthesis.speak(utterance);

This code demonstrates an important fact: a modern browser is already capable of reproducing quite normal human speech from text without our own DSP implementation.

However, architecturally this is a completely different class of system, and it is not suitable for our primary task.

What the API Actually Does

SpeechSynthesisUtterance takes text and settings like language/voice, and then passes them to the built-in speech engine of the browser/operating system.

Conditional diagram:

text
→ SpeechSynthesisUtterance
→ language / voice selection
→ hidden TTS engine
→ audio

The result is available to the user, but not the internal sound generation program.

Critical Limitation 1. No Direct Control Over the Sound Itself

The API accepts text, not an articulatory or acoustic trajectory.

For example:

mama

is usually pronounced normally.

However, trying to write:

mmmmmmaaamama

does not mean the following to the engine:

hold /m/
→ smoothly transition to /a/
→ continue the syllable

The engine interprets the string as text and may start reading it roughly as a sequence of letter names or syllables:

em-em-em-em... ah... ma-ma...

Similarly:

shhhhhhhh

may turn into something like:

sha-sha-sha-sha...

That is, through this API, you cannot reliably set:

• the duration of a specific consonant;
• the holding of a phoneme;
• a smooth transition between sounds;
• the shape of the articulatory transition;
• a separate time trajectory for noise, voicing, formants, and other parameters.

Therefore, the API is practically unsuitable for experiments of the type:

M ─────────→ A

where the transition itself is the object of study.

Critical Limitation 2. Singing and Arbitrary Temporal Structure

Since the input is text rather than a sound scenario, you cannot freely specify:

• stretching of individual sounds;
• arbitrary syllable durations;
• the melodic trajectory of each sound;
• vocal transitions;
• non-standard rhythmics;
• normal "singing" through direct phoneme control.

Changing the general rate or pitch does not solve this problem: it changes the behavior of the entire utterance, rather than providing control over internal sound events.

Critical Limitation 3. Language Binding

SpeechSynthesisUtterance relies on lang and a specific voice.

This means that the engine expects text within the framework of a certain language system.

Problematic areas include:

• mixing multiple languages within a single phrase;
• arbitrary cross-language sounds;
• artificial words;
• non-standard letter sequences;
• sounds that are not words of the language at all.

Even if the engine tries to pronounce something, it does so through its own text interpretation rules, rather than as a universal generator of arbitrary sound.

Critical Limitation 4. Pronunciation Language Model Constraint

It is more accurate to speak not only of a "dictionary" in the literal sense, but of a broader limitation:

the engine knows how to pronounce what it is able to interpret within the framework of its pronunciation language model.

That is, it may try to read an unknown string, but it will still be an interpretation as text, rather than a direct reproduction of a given acoustic object.

For our task, this is fundamentally important.

We need a mechanism at the level of:

sound scenario
→ exact parameter trajectories
→ sound

whereas here the following is used:

text
→ hidden linguistic interpretation
→ hidden TTS model
→ sound

Critical Limitation 5. Black Box

Even if the engine pronounces a word with high quality, it does not output a structural description of how exactly the sound was constructed.

There is no accessible representation of the type:

initial state
+ timeline
+ source parameters
+ noise parameters
+ resonances
+ transitions

Therefore, SpeechSynthesis does not help solve the central research problem:

to obtain a compact controllable description of sound and be able to transition between sound and this description.

It only demonstrates the system's ability to reproduce good speech from text.

Why It Is Still Useful to Document This

Despite its architectural unsuitability for our task, SpeechSynthesis is useful to consider as a separate ready-made option for other projects.

If the task sounds simply like:

text → normal speech

then the browser API can completely fulfill it without its own synthesizer.

It is especially useful where:

• controlling individual phonemes is not required;
• non-standard sounds are not needed;
• singing is not needed;
• decryption or access to the internal representation is not required;
• standard TTS in a supported language is sufficient.

Conclusion for Our Project

SpeechSynthesis shows that high-quality speech in the browser as a final result is already available today.

However, it does not suit us architecturally because:

1. the input is text, not a sound scenario;
2. there is no precise control over the internal temporal structure of the sound;
3. there is no proper control over the duration of individual phonemes;
4. it is poorly suited for singing and arbitrary vocal trajectories;
5. it depends on the language and a specific voice engine;
6. it is not a universal generator of arbitrary sounds;
7. it does not expose the internal representation;
8. it does not solve the inverse problem of sound → scenario.

Therefore, for our research task, this is not a competitor to VoiceScenario, but a separate ready-made TTS tool for a much narrower class of tasks.