Worklog for task "Research qualitative programmatic human voice synthesis in the browser"

Research Progress

We tested the hypothesis that a practically applicable example of the following form should exist for our task:

real audio/syllable recording → automatic analysis → compact parametric trajectory of a universal generator → inverse synthesis of recognizable sound

Why Such an Example Seemed to Exist

1. Procedural synthesis exists on its own. Web Audio provides oscillators, noise, filters, and AudioWorklet; Pink Trombone shows a browser-native physical vocal tract; formant demos allow obtaining individual vowel-like sounds from a small number of parameters.

2. Regular waveform representation is very large. About 1 second of microphone recording can take around 200 KB in a Float32Array. Even after aggressive μ-law @ 8 kHz, it leaves about 11 KB and over 10,000 values per second, although speech is still perceived normally. This creates a strong intuition that the perceived sound structure should have a substantially more compact representation.

3. There are many speech analysis methods in literature. LPC, source-filter analysis, formant tracking, spectral-envelope estimation, vocoders, analysis-by-synthesis, etc. By their names and descriptions, they look close to the required mechanism.

4. LLMs initially evaluated the task as overly optimistic. The conceptual scheme looks simple: generator + parameters + target sound + optimizer. This created a false impression that practical inverse mapping should have been solved long ago and be available in open-source projects.

What Was Found

Formant/Web Audio Demos

There are many examples where a sawtooth/impulse source passes through several band-pass filters to produce a sound resembling a vowel.

This only proves direct generation from pre-known parameters. Such examples do not provide the reverse decoding we need.

Pink Trombone / Modular Pink Trombone

This is the most interesting generator found because:

• sound is created procedurally;
• the core does not have to operate on letters;
• vocal tract state can be changed continuously;
• the implementation runs in the browser;
• intermediate sounds outside a fixed vocabulary can be obtained.

However, it lacks the main thing for our task:

• no found mechanism for recording → tract trajectory;
• no automatic recovery of parameter trajectories from a real A sound or any other sound;
• the quality of the synthesis itself remains noticeably synthetic and does not demonstrate natural speech even with manual control.

In other words, Pink Trombone provides a generator, but not a decoder.

Klatt and Klatt-like Implementations

These are not an architectural reference point for our task.

Reasons:

• many implementations already build an interface around phonemes, ARPABET, presets, and language rules;
• language and letters become part of the core architecture;
• parameters are often handcrafted;
• quality is close to old synthesizers;
• no universal inverse mapping from arbitrary sound to parameters;
• no demonstration of automatic recovery of high-quality real speech.

For us, this is too narrow a layer: we are looking for universal sound decoding, not a phoneme table.

LPC / Source-Filter Analysis

They allow estimating individual characteristics: spectral envelope, resonances, excitation source, etc.

However, no end-to-end case was found where these characteristics are automatically turned into a compact control program of a sufficiently universal generator, after which a sound close to the original is reproduced.

Thus, these are separate parts of a solution rather than a ready-made system.

Analysis-by-Synthesis

Papers were found where synthesizer parameters are fitted to natural speech via optimization.

This is useful not as a ready-made solution, but as a confirmation of engineering complexity: inverse mapping is non-linear, ambiguous, and requires searching in a large parameter space.

At the same time, no ready-made universal browser-friendly solution of the required quality was found.

Neural TTS / Codec Models

High quality is present, but this is a different class of solution:

• internal representation is hidden inside a trained model;
• no explicit universal controllable VoiceState;
• impossible to use the system as a transparent physical/procedural generator;
• this does not answer the research goal of decoding sound into a compact, understandable control program.

What Could Not Be Found

Not a single convincing open-source technical example was found that simultaneously:

1. Takes a real recording of a specific sound or short syllable.
2. Does not require knowing the letter/phoneme/language in advance.
3. Automatically reconstructs a compact temporal trajectory of parameters.
4. Uses a universal procedural generator rather than a neural TTS/codec decoder.
5. Reverse-synthesizes sound with quality sufficient at least for normal recognizable speech and noticeably better than demonstration physical synthesizers.
6. Allows viewing and studying the decoding implementation.
7. Can be realistically ported to the browser stack.

This is currently the main negative result of the research.

Why the Task Turned Out to Be Engineering-Wise Complex

The Inverse Problem is Ambiguous

A similar output waveform can be obtained by different internal states of the generator. One cannot simply uniquely reconstruct a single "correct" set of parameters from a recording.

A Perceptual Metric is Needed, Not Just Waveform Error

Two waveforms can have a large sample-by-sample difference while sounding almost identical. Conversely, a small error in a critical temporal or spectral region can significantly alter perception.

This means standard MSE is not a sufficient objective function.

The Search Target is a Trajectory, Not a Vector

Sound is a dynamic process. The excitation source, resonances, noise, attack, closures and openings, nasal tract, and other parameters change.

Therefore, the actual search variable is a temporal program of parameters, not a single static point.

The Generator Itself Might Not Be Able to Reproduce the Target Sound

If the chosen model is not expressive enough, no optimizer will reconstruct the original sound. The more expressive the generator, the larger the parameter space and the more difficult the inverse search.

The Voice is More Complex Than a Few Formants

For high-quality speech, the glottal source, spectral tilt, aspiration, turbulence, antiresonances, nasal coupling, jitter/shimmer, vocal tract dynamics, source-filter interaction, and other mechanisms can be essential.

Adding each mechanism improves expressiveness but simultaneously complicates decoding.

Manual Tuning Does Not Scale

Tuning the parameters of a single A manually is possible. Universally restoring an arbitrary sound by this method is impossible.

This is akin to manually selecting an exact color via CMYK without converting from the target color to parameters: you can achieve individual examples, but it does not solve the general problem.

Refined Statement of the Core Problem

The main task is now formulated neither as "making a synthesizer of Russian letters" nor as "picking good presets."

It is more universal:

To learn how to automatically find a compact temporal parameter program for a sufficiently universal generator given a real target sound.

Letters, phonemes, and languages should only be possible categories built on top of the discovered sound space, not part of the core.

Meta-Conclusion on Working with LLMs

The case revealed a specific risk: LLMs easily mistake clear architectural decomposition for low engineering complexity.

Several models initially described the task as practically simple because all components are familiar individually: Web Audio, filters, interpolation, optimization.

However, the presence of understandable components does not prove the existence of a working system built from them.

Only an attempt to find a specific reproducible end-to-end case showed that the critical inverse mapping among the found solutions is practically absent.

For further work, technical feasibility must be verified through minimal proof-of-feasibility experiments rather than LLM confidence.