Task: Research parametric synthesis of Russian sounds and transitions between them

Research Progress

The core architectural approach has been established and experimentally verified: human sound is described as a continuous scenario of modifying a universal DSP state over time, rather than as a special letter/phoneme renderer.

Implemented

• VoiceScenario with initial and a shared timeline.
• set and animate commands, including cubic bezier.
• Continuous state with parameters for excitation, periodicity, noise, F0, source shape, three universal resonance channels, and final level.
• A new command for a single parameter starts moving from its current value and replaces any previous future automation.
• The synthesizer accepts both a single VoiceScenario and VoiceScenario[].
• The array of scenarios unfolds into a single timeline without resetting the source phase or DSP state between elements.
• This made it possible to assemble Privet as [PRI_SCENARIO, VET_SCENARIO] rather than creating a separate word scenario.
• The noise source was migrated to a seeded PRNG for reproducible PCM.
• Synthetic sound passes through the same VoiceLogger as live recordings.

Experimental Cycle

The following scheme is used:

live recording → log → analysis → scenario → synthesis → same logger → comparison → adjustment

The current logger measures RMS, peak, ZCR, spectral centroid, spectral flatness, dominant frequency, and several energy bands. It is still rough: analysis at 128 samples yields a frequency step of about 375 Hz, so it needs further improvement.

Results for Individual Sounds

A

The sustained A has already turned out relatively natural. This confirmed that timeline + periodic source + universal resonances are generally suitable for some stable voice states.

S

After several iterations, it turned out that subjective "hissing" cannot be reduced to an excess of high frequencies. A live, strong S simultaneously has a high ZCR, a high spectral centroid, and intense high-frequency turbulence. Boosting only the overall amplitude does not solve the source quality problem.

Sh

Sh is treated as a continuous noise state close to S, but with a lower and broader spectral energy distribution.

T

T confirmed the presence of short-term events that cannot be stretched as a stable state. A live T is characterized by a very short high-frequency burst and rapid decay; in several recordings, the main peak often turned out to be around 15–16 kHz. This was successfully expressed using the same timeline model without a special T type.

Composite Scenarios

The scenarios Pri and Vet were created, and their composition via an array of scenarios was tested. They are already recognizable, but the sound quality remains distinctly computer-generated.

After comparison with a live Privet, it was revealed that:

• the initial scenarios were greatly stretched in time;
• manual deep oscillations of sourceLevel for R sound mechanical;
• the final T in one of the variants reached peak = 1, i.e., hit a hard clamp;
• the scenarios were shortened to more conversational durations and the final T was attenuated.

Current DSP Limitations

The main risk now lies not in the scenario architecture, but in the quality of the sound engine itself. Possible missing universal mechanisms:

• a more realistic glottal source;
• spectral tilt;
• aspiration / breathiness;
• broadband/direct noise path;
• a more physical turbulence model;
• antiresonances;
• nasal tract;
• source-filter interaction;
• a richer vocal tract model;
• possibly jitter/shimmer and other small variations;
• clipping control instead of a hard clamp that hides the causes.

Architectural Conclusions

• letters, phonemes, and syllables should remain metadata, not DSP ontology;
• stable, transitional, and plosive phenomena can currently be expressed using a single timeline model;
• scenario composition seems to be the right direction;
• generative TTS is not suitable as a research subject, since the task requires explicit programming of new sounds, including previously unseen ones;
• Pink Trombone and similar physical models are useful as references, but their quality shows that a single continuous tract model is not enough.

Next Step

A separate related research task cmtzxmokp085eqw0qtb7vg7c2 has been created regarding high-quality programmatic human voice synthesis in the browser. It aims to find browser-native/browser-capable projects and DSP approaches that qualitatively reproduce at least a few human sounds or syllables without generative TTS, and to understand which universal mechanisms should be ported to the current engine.

New Progress: Structural Sound Representation and the Human ↔ AI Loop

Several encouraging signs of the current prototype have been recorded.

1. We Are No Longer Working with Waveforms, but with a Compact Program

The current synthesizer operates on individual initial state variables and separate scenarios on a shared timeline. This is fundamentally different from directly storing thousands or tens of thousands of samples.

Despite the model's coarseness, a relatively small number of parameters already yields sounds that a human sometimes recognizes as letters, phonemic fragments, or syllables. This does not yet prove the model's sufficiency for natural speech, but it shows that the parametric space is not completely chaotic.

2. The Human Effectively Acts as a Perceptual Validator

The AI does not have a reliable standard for how natural the synthesis sounds. Therefore, the current working loop looks like this:

expected sound → scenario → synthesis → human listens → evaluates quality → AI suggests parameter changes → new synthesis.

Here, the human provides the final perceptual feedback, while the AI helps analyze structural differences and navigate the parameter space.

3. Using a Real Recording as a Local Reference

When searching for the sound Kha, a recording of the word/phrase KuKhaRe was used, where the desired segment is actually present. The AI received:

• a log of the synthetic variant;
• a log of the real recording;
• information on where the target fragment is located within the recording.

Next, it compares structural features and suggests which scenario parameters can be changed to get closer to the desired section.

This does not provide an instantaneous result, but it looks like a working human-AI loop for approximating the target sound.

4. Perhaps the Current Logs Are Too Heavily Compressed

There is a suspicion that the quality of the analytical logs is insufficient for precise comparison. If a human hears differences that the AI cannot infer from the log, the bottleneck may lie not only in the generator, but also in the analysis resolution.

It is necessary to separately check how the quality of the AI's suggestions changes when log compression is reduced and the number of temporal/spectral features is increased.

5. Accidental Real Sounds Found Are Also Knowledge

During the search for Kha, one of the variants was perceived as Va.

This is useful for two reasons:

1. the found sound itself may be needed later;
2. the transition from the previous "hissing" to the perceived Va shows the direction in the parametric space.

Such accidental finds must be saved along with parameter changes, because they help map the space of possible sounds.

6. The Space Still Remains in the Domain of Speech-Like Sounds

Despite the potential universality of AudioContext, when changing the parameters of the current voice-oriented generator, we still mainly encounter sounds that are perceived as vocal/phonetic or close to them, rather than random real sounds like barking, bird singing, or hoof beats.

This does not prove universality, nor does it prove the completeness of human speech coverage. A more cautious conclusion:

the current generator architecture already contains a strong inductive bias towards the voice-like sound space.

This can be a useful property because the search is performed not across the entire possible audio space, but within a narrower region.

7. Refined Practical Goal

Now the main question is not only "what parameters are needed for speech synthesis," but:

how compactly can the required region of human speech be described without reverting to thousands of samples.

The extreme points are already clear:

• waveform / μ-law — high quality and universality, but thousands of values per second;
• VoiceScenario — compactness and controllability, but currently unknown quality limits.

Further work should explore the gap between them and the dependency:

quality ↑ ↔ complexity/size of representation ↑.

A possible path is to gradually add only those mechanisms that provide a noticeable perceptual gain while preserving the model's structural nature.

New Findings on Synthesizer Architecture and Reversibility

An analysis of the full VoiceScenario -> PCM pipeline has been conducted.

1. Synthesis is Entirely in Our Code

It has been clarified that there is no external hidden audio engine generating the PCM. The final audio track array is entirely computed inside synthesizeScenario:

VoiceScenario -> VoiceControlState(t) -> excitation -> resonators -> value -> Float32Array.

At each sample, the code independently calculates the final value and writes it to output[sampleIndex]. The browser's AudioBufferSourceNode then simply plays back the ready-to-use PCM.

This means the entire DSP can be instrumented and analyzed at every step.

2. Control Parameters and Measured Acoustic Metrics are Different Levels

After fixing the scenario representation, it became clear that even nearly constant control parameters produce high-frequency changes in the measured audio. This is expected.

Reasons:

• a constant f0Hz determines the rate of change of voicePhase, not a constant waveform value;
• a constant noiseLevel controls the amplitude of a random source that changes every sample;
• resonators have an internal stateful state (low, band) and continue to change dynamically even with constant parameters;
• RMS, spectralFlatness, and dominantFrequency depend on multiple control parameters and the internal DSP state simultaneously.

Therefore, control curve != measured acoustic curve.

3. An Additional Layer Hidden from the Scenario Has Been Found: Internal DSP State

Between VoiceControlState and PCM, there are internal states:

• voicePhase;
• resonator1/2/3.low;
• resonator1/2/3.band;
• the state of the deterministic PRNG;
• active automations.

It is precisely this layer that explains why nearly horizontal control lines can lead to a complex high-frequency waveform and uneven acoustic metrics.

4. Next Research Direction

We need to move from the question "which log metric corresponds to a parameter" to the system identification task:

how does a change in each VoiceControlState parameter affect the waveform and measurable acoustic features?

Proposed experiment: a parameter sweep for each control parameter while keeping the others fixed, collecting the waveform + spectral/temporal characteristics. This will allow building a control -> acoustic features map and understanding which parameters are actually observable from the audio, which depend on combinations of parameters, and which are completely invisible to the current logger.

The basic theory of PCM was also discussed: a constant unsigned 8-bit PCM level (e.g., 255,255,255,...) does not produce a stable tone because there is no periodic change; transitions are primarily what is heard. This emphasizes that an individual PCM value is an instantaneous amplitude, while frequency/timbre/loudness are properties of the sequence's structure over time.