Task: Research qualitative programmatic human voice synthesis in the browser

Research Progress

We tested the hypothesis that a practically applicable example of the following form should exist for our task:

real audio/syllable recording → automatic analysis → compact parametric trajectory of a universal generator → inverse synthesis of recognizable sound

Why Such an Example Seemed to Exist

1. Procedural synthesis exists on its own. Web Audio provides oscillators, noise, filters, and AudioWorklet; Pink Trombone shows a browser-native physical vocal tract; formant demos allow obtaining individual vowel-like sounds from a small number of parameters.

2. Regular waveform representation is very large. About 1 second of microphone recording can take around 200 KB in a Float32Array. Even after aggressive μ-law @ 8 kHz, it leaves about 11 KB and over 10,000 values per second, although speech is still perceived normally. This creates a strong intuition that the perceived sound structure should have a substantially more compact representation.

3. There are many speech analysis methods in literature. LPC, source-filter analysis, formant tracking, spectral-envelope estimation, vocoders, analysis-by-synthesis, etc. By their names and descriptions, they look close to the required mechanism.

4. LLMs initially evaluated the task as overly optimistic. The conceptual scheme looks simple: generator + parameters + target sound + optimizer. This created a false impression that practical inverse mapping should have been solved long ago and be available in open-source projects.

What Was Found

Formant/Web Audio Demos

There are many examples where a sawtooth/impulse source passes through several band-pass filters to produce a sound resembling a vowel.

This only proves direct generation from pre-known parameters. Such examples do not provide the reverse decoding we need.

Pink Trombone / Modular Pink Trombone

This is the most interesting generator found because:

• sound is created procedurally;
• the core does not have to operate on letters;
• vocal tract state can be changed continuously;
• the implementation runs in the browser;
• intermediate sounds outside a fixed vocabulary can be obtained.

However, it lacks the main thing for our task:

• no found mechanism for recording → tract trajectory;
• no automatic recovery of parameter trajectories from a real A sound or any other sound;
• the quality of the synthesis itself remains noticeably synthetic and does not demonstrate natural speech even with manual control.

In other words, Pink Trombone provides a generator, but not a decoder.

Klatt and Klatt-like Implementations

These are not an architectural reference point for our task.

Reasons:

• many implementations already build an interface around phonemes, ARPABET, presets, and language rules;
• language and letters become part of the core architecture;
• parameters are often handcrafted;
• quality is close to old synthesizers;
• no universal inverse mapping from arbitrary sound to parameters;
• no demonstration of automatic recovery of high-quality real speech.

For us, this is too narrow a layer: we are looking for universal sound decoding, not a phoneme table.

LPC / Source-Filter Analysis

They allow estimating individual characteristics: spectral envelope, resonances, excitation source, etc.

However, no end-to-end case was found where these characteristics are automatically turned into a compact control program of a sufficiently universal generator, after which a sound close to the original is reproduced.

Thus, these are separate parts of a solution rather than a ready-made system.

Analysis-by-Synthesis

Papers were found where synthesizer parameters are fitted to natural speech via optimization.

This is useful not as a ready-made solution, but as a confirmation of engineering complexity: inverse mapping is non-linear, ambiguous, and requires searching in a large parameter space.

At the same time, no ready-made universal browser-friendly solution of the required quality was found.

Neural TTS / Codec Models

High quality is present, but this is a different class of solution:

• internal representation is hidden inside a trained model;
• no explicit universal controllable VoiceState;
• impossible to use the system as a transparent physical/procedural generator;
• this does not answer the research goal of decoding sound into a compact, understandable control program.

What Could Not Be Found

Not a single convincing open-source technical example was found that simultaneously:

1. Takes a real recording of a specific sound or short syllable.
2. Does not require knowing the letter/phoneme/language in advance.
3. Automatically reconstructs a compact temporal trajectory of parameters.
4. Uses a universal procedural generator rather than a neural TTS/codec decoder.
5. Reverse-synthesizes sound with quality sufficient at least for normal recognizable speech and noticeably better than demonstration physical synthesizers.
6. Allows viewing and studying the decoding implementation.
7. Can be realistically ported to the browser stack.

This is currently the main negative result of the research.

Why the Task Turned Out to Be Engineering-Wise Complex

The Inverse Problem is Ambiguous

A similar output waveform can be obtained by different internal states of the generator. One cannot simply uniquely reconstruct a single "correct" set of parameters from a recording.

A Perceptual Metric is Needed, Not Just Waveform Error

Two waveforms can have a large sample-by-sample difference while sounding almost identical. Conversely, a small error in a critical temporal or spectral region can significantly alter perception.

This means standard MSE is not a sufficient objective function.

The Search Target is a Trajectory, Not a Vector

Sound is a dynamic process. The excitation source, resonances, noise, attack, closures and openings, nasal tract, and other parameters change.

Therefore, the actual search variable is a temporal program of parameters, not a single static point.

The Generator Itself Might Not Be Able to Reproduce the Target Sound

If the chosen model is not expressive enough, no optimizer will reconstruct the original sound. The more expressive the generator, the larger the parameter space and the more difficult the inverse search.

The Voice is More Complex Than a Few Formants

For high-quality speech, the glottal source, spectral tilt, aspiration, turbulence, antiresonances, nasal coupling, jitter/shimmer, vocal tract dynamics, source-filter interaction, and other mechanisms can be essential.

Adding each mechanism improves expressiveness but simultaneously complicates decoding.

Manual Tuning Does Not Scale

Tuning the parameters of a single A manually is possible. Universally restoring an arbitrary sound by this method is impossible.

This is akin to manually selecting an exact color via CMYK without converting from the target color to parameters: you can achieve individual examples, but it does not solve the general problem.

Refined Statement of the Core Problem

The main task is now formulated neither as "making a synthesizer of Russian letters" nor as "picking good presets."

It is more universal:

To learn how to automatically find a compact temporal parameter program for a sufficiently universal generator given a real target sound.

Letters, phonemes, and languages should only be possible categories built on top of the discovered sound space, not part of the core.

Meta-Conclusion on Working with LLMs

The case revealed a specific risk: LLMs easily mistake clear architectural decomposition for low engineering complexity.

Several models initially described the task as practically simple because all components are familiar individually: Web Audio, filters, interpolation, optimization.

However, the presence of understandable components does not prove the existence of a working system built from them.

Only an attempt to find a specific reproducible end-to-end case showed that the critical inverse mapping among the found solutions is practically absent.

For further work, technical feasibility must be verified through minimal proof-of-feasibility experiments rather than LLM confidence.

Clarification on the Search for Existing Implementations

The search for external projects was primarily needed to answer one question:

to what extent the space of real human speech lies within the space of sounds that a compact procedural generator can reproduce.

Direct confirmation of this has not yet been found.

What is Known for Certain

AudioContext is capable of high-quality reproduction of any pre-recorded digital sound if given the full array of samples. This means the problem is not the browser as a playback environment.

The problem is representation.

A full waveform is universal, but expensive: even a heavily compressed second of speech remains a sequence of thousands of values. Our model replaces this sequence with a compact script of meaningful parameters and commands over time.

Therefore, the boundary under investigation looks like this:

full waveform ← more data / higher universality ... structural VoiceScenario → less data / higher controllability

We need to understand where on this axis the quality sufficient for natural speech appears.

Why the Found Projects Did Not Provide an Answer

Pink Trombone and similar physical models show that it is possible to procedurally generate a portion of the voice-like space, but:

• they are not universal;
• they mostly demonstrate vowel-like states and limited articulation;
• the sound remains computer-generated;
• there is no full demonstration of syllables and high-quality connected speech;
• there is no inverse decoder recording → compact script.

Formant demos are even narrower: they show individual static or near-static sounds, but do not answer the question of covering real speech.

Klatt-like systems are not a benchmark for our architecture because they already rely on phonemic/language presets and do not solve the universal decoding of arbitrary sound.

Neural TTS/codec systems demonstrate high quality, but hide the internal representation and do not provide a transparent, controllable script that can be explored and edited.

New Conclusion

The absence of a found example does not mean that a compact script-based generator is incapable of approaching human speech. It only means that we do not yet have external proof of the required quality level.

Therefore, the next practical path is not to continue an endless search for analogues, but to measure the limits of our own model:

1. improve the direct generator;
2. build a visual script editor to speed up manual tuning;
3. create an inverse decoder and first test it on our own synthetic data with known ground truth;
4. then transfer it to human recordings;
5. gradually increase the expressiveness of the representation and measure the quality gain provided by each additional mechanism.

The main object of research now is the tradeoff between the compactness of the structural description and perceptual quality.

The fundamental physical limit for high-quality speech reproduction and benchmarks for the Russian language have been investigated.

Key takeaways:

• A single audio channel (mono) is sufficient for reproducing the human voice: at any single point in space, air pressure is described by a single scalar time function. Stereo is needed for spatial features, not for the timbre or phonemic intelligibility of the voice itself.
• ITU wideband speech: approximately 50–7000 Hz. According to the Nyquist theorem, preserving this band requires a sample rate >14 kHz; the practical standard is 16 kHz. This yields raw PCM 16-bit mono = 16,000 * 16 = 256 kbps.
• ITU super-wideband: 50–14,000 Hz; practical sample rate >=32 kHz, raw PCM 16-bit mono = 512 kbps. 48 kHz/16-bit mono = 768 kbps, which covers the fullband of human hearing much wider than typically necessary for speech.
• For compressed speech, Opus RFC 6716 specifies sweet spots: 8–12 kbps for narrowband speech, 16–20 kbps for wideband speech, 28–40 kbps for fullband speech.
• G.722 defines high-quality wideband speech at 50–7000 Hz at 64 kbps (an older SB-ADPCM codec); modern Opus achieves comparable or better subjective efficiency at a noticeably lower bitrate.
• The Russian OpenSTT (~20 thousand hours) uses mono, 16 kHz, int16 as its main practical format; their former MP3 profile was 16 kHz mono 32 kbps. This is a useful Russian empirical baseline, although not a fundamental minimum.

It is important to distinguish between storage/transmission bitrate and synthesizer complexity: a single output audio channel can be derived from many internal sources/filters (glottal source, aspiration, frication, resonances, nasal branch, etc.). The number of internal DSP components does not equal the number of output audio channels.

Web Speech API / SpeechSynthesis as a Separate Solution Class

As part of the search for existing implementations, the built-in browser SpeechSynthesis was tested.

Minimal example:

const utterance = new SpeechSynthesisUtterance('Hello, this is speech synthesis!');
utterance.lang = 'en-US';
const voices = speechSynthesis.getVoices();
utterance.voice = voices.find(v => v.lang === 'en-US');
speechSynthesis.speak(utterance);

This code demonstrates an important fact: a modern browser is already capable of reproducing quite normal human speech from text without our own DSP implementation.

However, architecturally this is a completely different class of system, and it is not suitable for our primary task.

What the API Actually Does

SpeechSynthesisUtterance takes text and settings like language/voice, and then passes them to the built-in speech engine of the browser/operating system.

Conditional diagram:

text
→ SpeechSynthesisUtterance
→ language / voice selection
→ hidden TTS engine
→ audio

The result is available to the user, but not the internal sound generation program.

Critical Limitation 1. No Direct Control Over the Sound Itself

The API accepts text, not an articulatory or acoustic trajectory.

For example:

mama

is usually pronounced normally.

However, trying to write:

mmmmmmaaamama

does not mean the following to the engine:

hold /m/
→ smoothly transition to /a/
→ continue the syllable

The engine interprets the string as text and may start reading it roughly as a sequence of letter names or syllables:

em-em-em-em... ah... ma-ma...

Similarly:

shhhhhhhh

may turn into something like:

sha-sha-sha-sha...

That is, through this API, you cannot reliably set:

• the duration of a specific consonant;
• the holding of a phoneme;
• a smooth transition between sounds;
• the shape of the articulatory transition;
• a separate time trajectory for noise, voicing, formants, and other parameters.

Therefore, the API is practically unsuitable for experiments of the type:

M ─────────→ A

where the transition itself is the object of study.

Critical Limitation 2. Singing and Arbitrary Temporal Structure

Since the input is text rather than a sound scenario, you cannot freely specify:

• stretching of individual sounds;
• arbitrary syllable durations;
• the melodic trajectory of each sound;
• vocal transitions;
• non-standard rhythmics;
• normal "singing" through direct phoneme control.

Changing the general rate or pitch does not solve this problem: it changes the behavior of the entire utterance, rather than providing control over internal sound events.

Critical Limitation 3. Language Binding

SpeechSynthesisUtterance relies on lang and a specific voice.

This means that the engine expects text within the framework of a certain language system.

Problematic areas include:

• mixing multiple languages within a single phrase;
• arbitrary cross-language sounds;
• artificial words;
• non-standard letter sequences;
• sounds that are not words of the language at all.

Even if the engine tries to pronounce something, it does so through its own text interpretation rules, rather than as a universal generator of arbitrary sound.

Critical Limitation 4. Pronunciation Language Model Constraint

It is more accurate to speak not only of a "dictionary" in the literal sense, but of a broader limitation:

the engine knows how to pronounce what it is able to interpret within the framework of its pronunciation language model.

That is, it may try to read an unknown string, but it will still be an interpretation as text, rather than a direct reproduction of a given acoustic object.

For our task, this is fundamentally important.

We need a mechanism at the level of:

sound scenario
→ exact parameter trajectories
→ sound

whereas here the following is used:

text
→ hidden linguistic interpretation
→ hidden TTS model
→ sound

Critical Limitation 5. Black Box

Even if the engine pronounces a word with high quality, it does not output a structural description of how exactly the sound was constructed.

There is no accessible representation of the type:

initial state
+ timeline
+ source parameters
+ noise parameters
+ resonances
+ transitions

Therefore, SpeechSynthesis does not help solve the central research problem:

to obtain a compact controllable description of sound and be able to transition between sound and this description.

It only demonstrates the system's ability to reproduce good speech from text.

Why It Is Still Useful to Document This

Despite its architectural unsuitability for our task, SpeechSynthesis is useful to consider as a separate ready-made option for other projects.

If the task sounds simply like:

text → normal speech

then the browser API can completely fulfill it without its own synthesizer.

It is especially useful where:

• controlling individual phonemes is not required;
• non-standard sounds are not needed;
• singing is not needed;
• decryption or access to the internal representation is not required;
• standard TTS in a supported language is sufficient.

Conclusion for Our Project

SpeechSynthesis shows that high-quality speech in the browser as a final result is already available today.

However, it does not suit us architecturally because:

1. the input is text, not a sound scenario;
2. there is no precise control over the internal temporal structure of the sound;
3. there is no proper control over the duration of individual phonemes;
4. it is poorly suited for singing and arbitrary vocal trajectories;
5. it depends on the language and a specific voice engine;
6. it is not a universal generator of arbitrary sounds;
7. it does not expose the internal representation;
8. it does not solve the inverse problem of sound → scenario.

Therefore, for our research task, this is not a competitor to VoiceScenario, but a separate ready-made TTS tool for a much narrower class of tasks.