The theoretical foundation of the project is compiled in the main concept ["uchitsya-legko.rf — learning to read as decoding a written code"](/concepts/uchitsya-legko-rf-obuchenie-chteniyu-kak-deshifrovke-pismennogo-koda). It is the central point of the project: it describes the general model, current state, and connects specific concepts.

The project is currently at the **Idea** stage. The basic theory of reading as information reconstruction from a written code has been formed. The information entities on which learning is built have been recorded: object and situation, concept, image, spoken word, sound, written word, letters, and larger structural parts of a word.

Key components of the theory:

- [Reading is word recognition and meaning reconstruction, not sequential letter sounding](/concepts/chtenie-eto-raspoznavanie-slova-i-vosstanovlenie-smysla).
- [A written word connects several representations of the same content](/concepts/pismennoe-slovo-svyazyvaet-neskolko-predstavleniy).
- [Learning to read should go from known supports to the unknown code](/concepts/obuchenie-chteniyu-ot-izvestnyh-opor-k-neizvestnomu-kodu).
- [The oral vocabulary is the recognition base when learning to read](/concepts/ustnyy-slovar-baza-raspoznavaniya-pri-obuchenii-chteniyu).
- [A long word must be revealed through familiar structural parts](/concepts/dlinnoe-slovo-cherez-znakomye-strukturnye-chasti).
- [Hints must gradually disappear as connections are mastered](/concepts/podskazki-dolzhny-postepenno-ischezat).

The practical direction is an interactive system where the child starts with recognizable objects and words, confirms their sound, gets acquainted with the spelling, discovers repeating elements of the written code, and then applies the discovered correspondences to decode new words.

Critical principle: the system must not simultaneously require the mastery of several unknown levels. If the child does not know the concept, the spoken word, its sound, and the letters, such a word is poorly suited as starting material. The next task must rely on the already confirmed knowledge of the specific child.

The next stage of the project is to turn the theory into minimal interactive mechanics and test the main criterion: is the child able to transfer mastered correspondences to previously unseen words, rather than just memorizing shown examples.

An interactive project for teaching reading as the gradual decoding of a written code through meanings, images, and spoken words already familiar to the child.

uchitsya-legko.rf

## Connection to the Main Task

The parent task is [researching universal parametric synthesis of Russian sounds and transitions](/tasks/cmtzfmwqs07rwqw0q6l7d5izg).

The scenario architecture and the direction of the main task look promising. The current main risk lies at a lower level: **the DSP itself still reproduces the human voice in a way that sounds too computerized**.

This task should specifically research the capabilities and limitations of programmatically generating a plausible human sound in the browser.

## Main Question

> Which existing methods and browser implementations are capable of programmatically, deterministically, and without runtime-generative TTS reproducing at least a few human vocal sounds or syllables closely enough to a real voice?

We are primarily interested in the quality of the acoustic engine, rather than the text-to-speech pipeline.

## What to Look For

### 1. Real Browser Demos

Find projects that run entirely or almost entirely in the browser via:

* Web Audio API;
* AudioWorklet;
* WebAssembly;
* JavaScript / TypeScript;
* browser DSP in C/C++/Rust via WASM.

It is desirable that a live demo exists where the result can be heard without installing a large TTS stack.

It is minimally sufficient if the project qualitatively reproduces a few vowels, consonants, syllables, or short human sounds.

### 2. Source Code

Priority is given to open-source projects, especially on GitHub/GitLab.

For each useful project, find out:

* what DSP is used;
* how the voice source is modeled;
* how turbulence is modeled;
* how the vocal tract is modeled;
* whether resonances, antiresonances, waveguides, tube models, LF models, glottal flow, aspiration, spectral tilt, etc., are used;
* which parameters are controlled over time;
* how deterministic the model is;
* whether a previously unseen parameter trajectory can be specified.

### 3. Do Not Limit to Speech TTS Projects

Also search for:

* physical vocal synthesis;
* singing voice synthesis;
* vocal tract simulation;
* glottal source synthesis;
* vocal effects / vocal modeling;
* voice instruments;
* research demos;
* browser audio experiments;
* speech production simulators.

The project may not work with letters or language at all — this is even preferable if it generates good human sound quality.

## What to Consider an Anti-Pattern for This Task

### Systems Where the Core Is Built Around Letters/Phonemes/Syllables

For example, a Klatt-like project is uninteresting on its own if its main API is a dictionary of known phonemes, ARPABET, syllables, or ready-made synthesis rules for a specific language.

You can study its DSP components, but do not accept such an ontology as an architectural guideline.

### Generative TTS

Do not consider as a solution:

* VITS;
* diffusion TTS;
* transformer/LLM TTS;
* voice cloning;
* large speech models;
* other models that generate waveforms or codec tokens from a trained distribution.

They can only be used as an external quality benchmark.

Reason: the system under study must be able to construct a **new sound it has never encountered before** through an explicitly specified program of parameters.

### Pink Trombone as a Final Answer

Pink Trombone is useful architecturally, but its sound is too obviously synthetic. We need to understand what mechanisms are missing between such a physical model and a convincing human voice.

## Research Questions

1. Which fully programmatic non-generative methods yield the most human-like voice today?
2. Which of them can run in real-time in the browser?
3. How far can one go with source-filter models?
4. What do waveguide/tube vocal-tract models offer compared to a set of resonators?
5. How important is an accurate glottal flow model?
6. Which universal mechanisms particularly affect the sensation of a "live voice":
   * spectral tilt;
   * aspiration;
   * breathiness;
   * jitter/shimmer;
   * source-filter interaction;
   * antiresonances;
   * nasal tract;
   * broadband turbulence;
   * nonlinearities;
   * radiation model;
   * dynamic vocal tract?
7. Which mechanisms can be implemented as universal `VoiceState` coordinates rather than special phonemic modes?
8. What browser limitations are significant: sample rate, AudioWorklet block size, WASM latency, filter stability, CPU?
9. Are there published demos where several syllables or vocal gestures sound noticeably more natural than Pink Trombone without a neural network?
10. What from the findings can be practically ported to the current VoiceSynthesizer?

## Working Method

For each candidate found, record:

* link to live demo;
* link to source code;
* model type;
* whether it works in the browser;
* subjective quality;
* what sounds it actually can produce;
* whether there is free parameter control;
* whether a new trajectory can be set;
* which mechanisms are potentially useful for our engine;
* what limitations were discovered.

After the review, select 2–5 most promising DSP approaches and perform a separate technical comparison with the current architecture.

## Expected Result

* review of existing browser-native or browser-capable projects;
* list of the best live demos;
* map of utilized DSP mechanisms;
* explanation of why some models sound computerized while others sound more natural;
* list of universal mechanisms worth adding to the current `VoiceControlState`/DSP;
* experimental engine improvement plan;
* conclusion on the practical ceiling of fully programmatic browser-based synthesis without generative TTS.

Find and study browser-based projects and DSP approaches that programmatically reproduce at least a few human syllables or vocal sounds with high quality, without generative TTS and without tying the core to letters/syllables.

Research qualitative programmatic human voice synthesis in the browser

## Goal

Create a visual editor for synthesizer scenarios to stop editing code for every iteration and shift the work to the `VoiceScenario` level itself.

## What the editor should be able to do

- load an existing scenario JSON;
- create a scenario from scratch;
- edit the initial state;
- edit common timeline commands (`set`, `animate`, and other supported operations);
- change durations, levels, frequencies, noise parameters, resonances, and other available parameters;
- play the result immediately after modification;
- save and export the modified JSON;
- quickly switch between variants.

## Human as Perceptual Validator

The quality of the result is still most reliably evaluated by ear. Therefore, the editor should minimize the cycle:

`change parameters → play → hear result → change again`.

The goal is to remove mandatory AI source code editing from this cycle.

## Automation via AI Agent

The next layer of the editor:

1. a human uploads/records a reference sound;
2. the system receives its analytical log;
3. the current `VoiceScenario` is played and its log is built;
4. upon clicking a button, both logs along with the current JSON are sent to the AI agent;
5. the agent compares them and returns the modified scenario JSON;
6. the editor applies the variant;
7. the human checks it by ear.

The AI is not the final judge of quality here — it helps search for changes in the structured parameter space, while the human remains the perceptual quality validator.

## Separately Check Log Quality

There is a hypothesis that current analytical logs are compressed too much and lose features that a human can hear. It is necessary to be able to increase the analysis resolution and compare whether this improves the AI's ability to suggest useful changes.

## Important Research Mode

Do not discard accidentally discovered useful sounds.

If searching for `Ha` results in a recognizable `Va` or another desired voice fragment:

- save the scenario as a separate result;
- save both original and modified variants;
- record which parameter changes led to the transition;
- use this as knowledge about the structure of the parametric space.

## Success Criterion

The editor should make sound iterations significantly cheaper: a human is able to independently change and test scenarios, and the AI works with JSON/logs instead of constantly modifying code.

Speed up the manual search for quality sounds: edit scenarios visually, play them back instantly, and be able to send reference and synthetic logs to the AI agent to return modified JSON.

Create VoiceScenario Visual Editor and Human ↔ AI Loop

## Goal

Test the hypothesis of the practical reversibility of our parametric model:

`VoiceScenario → generator → audio`

and vice versa:

`audio → decoder → VoiceScenario`.

## Key verification principle

Start not with human speech, but with our own scenarios where the correct answer is known in advance.

For a scenario `S`:

`S → G(S) = X`

then:

`X → D(X) = S'`

The first rigorous criterion:

> if the decoder is unable to reliably reconstruct the original scenarios from the audio created by our own generator, speaking of decoding real speech is premature.

Verification must be performed on the entire available set of different scenarios, rather than on a single successful case.

## Stage 1. Synthetic ground truth

Gather a set of existing and new scenarios:

- sustained vocal sounds;
- noise;
- plosives;
- transitions;
- syllables/fragments;
- short words.

For each, store:

- original JSON;
- exact generated audio;
- detailed analytical log;
- decoder result;
- differences between the original and reconstructed scenario;
- result of re-synthesis from the reconstructed scenario.

We need to understand which parameters are restored reliably, which are ambiguous, and what log resolution is necessary.

## Stage 2. Re-synthesis verification

It is not enough to get a similar JSON. The reconstructed scenario must be played back again:

`S → X → D(X)=S' → G(S')=X'`

And compare:

- scenario structure;
- analytical features of `X` and `X'`;
- human auditory perception.

## Stage 3. Transfer to human speech

Only after stable performance on our own synthetic data should we move on to recording a human.

The goal here is no longer necessarily to reconstruct some "true" physical human scenario, which we do not have. Practical criterion:

> the decoder creates a `VoiceScenario` which, when played back, produces a recognizable and sufficiently close human sound.

## Main open question

To what extent the space of human speech is covered by the current audio space of our generator.

`AudioContext` is technically capable of reproducing any digital waveform if given a full array of samples. But our system intentionally replaces thousands of samples with a compact structural description.

Therefore, we are investigating not the possibility of reproduction in general, but the boundary:

> **what is the minimum complexity of the structural representation sufficient for an acceptable quality of human speech?**

## Potential model evolution

If the current `VoiceScenario` is not enough, do not consider this an automatic refutation of the approach. Explore a gradual increase in expressiveness:

- richer source model;
- additional resonances/anti-resonances;
- turbulence/noise;
- local spectral details;
- denser temporal control points;
- short residual components if necessary.

Ultimately, the representation can approach a waveform. The practical goal is to find the point where the quality is already sufficient, while the description is still significantly more compact and structural than an array of samples.

## Connection with the editor

Even an imperfect decoder can be useful if it produces a good draft scenario:

`recording → draft VoiceScenario → visual editor/AI → human refines by ear`.

## Success criteria

1. The decoder reliably restores its own scenarios at least on a limited set of sound classes.
2. Re-synthesis from the restored scenario is perceptually close to the original synthetic sound.
3. It is clear what data is lost during the current analysis and whether log compression needs to be reduced.
4. On real recordings, at least useful starting scenarios are obtained.
5. It is possible to estimate the dependency `quality ↔ complexity/representation size` and approach the practical boundary of the system.

Verify whether generator scenarios can be automatically reconstructed from recorded audio, first using our own synthetic examples with a known ground truth, and then on human speech.

Investigate the inverse decoder: audio → VoiceScenario

## Objective

Investigate not the instantaneous correspondence of the form `parameter[t] -> sample[t]`, but the temporal causality in the chain:

`VoiceControlState(t) -> internal DSP state -> PCM -> acoustic features in a time window`.

We need to understand **which control parameter actually affects what**, after how long this influence appears, how long it persists, and what observable features allow it to be recovered.

This is necessary for correct inverse decoding of `Sound -> VoiceScenario`.

---

## Context and Problem

The current `VoiceScenario` specifies control parameters, but the output PCM is generated by a dynamic system with internal state:

- `voicePhase`;
- `resonator1.low`, `resonator1.band`;
- `resonator2.low`, `resonator2.band`;
- `resonator3.low`, `resonator3.band`;
- PRNG state;
- active automation.

Therefore, even constant control parameters produce a complex, high-frequency waveform.

Examples:

- constant `f0Hz` does not mean a constant waveform — it sets the rotation speed of `voicePhase`;
- constant `noiseLevel` controls the amplitude of a random source that changes every sample;
- constant `resonance*` controls stateful filters whose internal state changes every sample;
- observable features (`RMS`, `spectralFlatness`, `dominantFrequency`, etc.) depend simultaneously on several control parameters and DSP history.

Consequently, direct heuristics like:

`RMS -> sourceLevel`

`dominantFrequency -> f0Hz`

are not a valid inverse transformation model.

---

## Main Hypothesis

For speech and this synthesizer, we need to analyze **not a single sample and not a single log frame**, but a time window.

Instead of:

`parameter[t] -> sample[t]`

investigate:

`parameter trajectory[t0:t1] -> audio response[t0:t1+Δ]`

where `Δ` is the possible reaction lag and/or decay time of the internal DSP state.

Working window scales to test:

- 5 ms;
- 10 ms;
- 20 ms;
- 30 ms;
- 50 ms;
- 100 ms if necessary.

For each parameter, it is necessary to determine at which time scale its influence is observed most stably.

---

## What to Investigate for Each VoiceControlState Parameter

At a minimum:

- `sourceLevel`
- `periodicity`
- `noiseLevel`
- `f0Hz`
- `glottalOpenPhase`
- `glottalReturnPhase`
- `resonance1Freq`
- `resonance1Bandwidth`
- `resonance1Gain`
- `resonance2Freq`
- `resonance2Bandwidth`
- `resonance2Gain`
- `resonance3Freq`
- `resonance3Bandwidth`
- `resonance3Gain`
- `outputLevel`

For each parameter, determine:

1. **Sensitivity** — how strongly its change affects PCM and features.
2. **Lag** — how long after a control change this influence becomes observable.
3. **Decay time** — how long the effect persists after the parameter changes.
4. **Locality** — whether the parameter affects mainly a specific spectrum range / temporal characteristic or globally.
5. **Uniqueness** — whether the observed effect can distinguish a change in this parameter from a change in another.
6. **Interactions** — which pairs of parameters produce a non-additive effect.

---

## Experiment 1: Single Parameter Sweep

For each parameter:

1. Fix all other parameters in a single baseline state.
2. Change only the parameter under study across a grid of values.
3. Synthesize PCM of equal duration for each value.
4. Simultaneously collect:
   - full `VoiceControlState(t)`;
   - internal DSP trace;
   - PCM;
   - acoustic features by windows.
5. Compare the result with the baseline.

Example for `f0Hz`:

- 80 Hz
- 100 Hz
- 120 Hz
- 150 Hz
- 180 Hz
- 220 Hz
- 300 Hz

Example for `resonance1Freq`:

- fix the source and other resonators;
- move only `resonance1Freq`;
- see which spectral peak actually moves and how.

---

## Experiment 2: Step Change Over Time

Test the dynamic response of the system.

Example:

`t < 100 ms: resonance1Freq = 600`

`t >= 100 ms: resonance1Freq = 800`

Analyze windows:

- 60–80 ms
- 80–100 ms
- 100–120 ms
- 120–140 ms
- 140–160 ms
- 160–200 ms

Objective:

- determine the start time of the observed change;
- measure the lag;
- understand the shape of the transient process;
- measure the stabilization time.

Repeat for other parameters.

---

## Experiment 3: Impulse / Short-Term Perturbation

For stateful parameters, it is especially important to measure system memory.

Scheme:

`baseline -> parameter + δ for a short interval -> baseline`

For example:

- 100 ms baseline;
- 10–20 ms change;
- return to baseline;
- observation for another 100–200 ms.

Measure:

- how fast the effect appears;
- whether it persists after the parameter returns;
- how long it takes for the DSP to return to its initial state.

For resonators, this should show the real memory duration of `low/band` states.

---

## Experiment 4: Parameter Pairs

After single sweeps, test interactions for at least the most coupled pairs:

- `sourceLevel × outputLevel`
- `periodicity × noiseLevel`
- `f0Hz × glottalOpenPhase`
- `f0Hz × glottalReturnPhase`
- `resonanceFreq × resonanceGain`
- `resonanceFreq × resonanceBandwidth`
- `sourceLevel × resonanceGain`

Objective — to understand whether parameter effects can be considered independent or if they need to be recovered jointly.

---

## What Data to Log

### Control Level

- all `VoiceControlState(t)` values;
- active automation;
- timestamp of each `set` / `animate` command.

### Internal DSP Trace

At a minimum:

- `voicePhase`;
- `periodicSource`;
- `noiseSource`;
- `excitation`;
- `resonator1.low` / `resonator1.band`;
- `resonator2.low` / `resonator2.band`;
- `resonator3.low` / `resonator3.band`;
- separate output of each resonator before summation;
- `value` before clamp;
- `value` after clamp.

### Acoustic Features by Windows

At a minimum:

- RMS / energy;
- zero crossing rate;
- autocorrelation;
- pitch estimate;
- periodicity estimate;
- spectral centroid;
- spectral flatness;
- spectral peaks;
- energy by bands;
- formant-like peaks / spectral envelope;
- LPC if necessary.

Important: treat `dominantFrequency` only as one of the spectral features, and not as a direct `f0Hz`.

---

## Time Windows and Logging Frequency

It is necessary to separately test the impact of throttling.

The current log may hide significant temporal structure. Therefore, for the study, it is desirable to have the most frequent internal trace possible, and then programmatically aggregate it into windows.

Recommended approach:

- DSP trace — as close to sample-level as possible or with controlled decimation;
- acoustic features — computed over 5/10/20/30/50 ms windows;
- window overlap: 50–75% for smooth temporal analysis;
- do not mix data collection frequency and analytic window size.

---

## Expected Results

Build an experimental map:

`VoiceControl parameter -> observable consequences`

For each parameter, record:

- which features react;
- direction of reaction;
- scale of reaction;
- lag;
- memory duration;
- dependence on other parameters;
- how recoverable the parameter is from audio.

It is advisable to format this into a table like:

| Control | Observable Features | Lag | Window | Uniqueness | Comment |
|---|---|---:|---:|---|---|
| f0Hz | autocorrelation peak, harmonic spacing | ... | ... | high/medium/... | ... |
| resonance1Freq | spectral envelope / peak shift | ... | ... | ... | ... |
| ... | ... | ... | ... | ... | ... |

---

## Result Criterion

After the study, it should become clear:

1. Which `VoiceControlState` parameters can be recovered directly from audio.
2. Which ones can only be recovered jointly with other parameters.
3. Which parameters are not observable at all with the current feature set.
4. What additional features need to be added to the logger.
5. What temporal context the decoder needs for each parameter class.
6. Whether the decoder can be built as an analytical/heuristic transformation or if an optimization/learned approach is required.

The main result is not just a new decoder, but a **physically and experimentally confirmed correspondence model between the synthesizer control space and the observable sound over time**.

Build an experimental map of how control parameter changes over time manifest in PCM and acoustic features, taking into account DSP lags and memory.

Investigate temporal causality of VoiceControlState → DSP → acoustic features

## Goal

Create a new visual interface not for cosmetic editing of an existing `VoiceScenario`, but as an **experimental speech synthesis laboratory**.

The main objective of the interface is to allow manual exploration of:

- which synthesizer parameters;
- in what sequence;
- at what moments in time;
- with what values and laws of change

produce a particular audible result.

We do NOT assume in advance that we know the correct scenario for a specific sound. The interface is specifically designed for exploration: the user moves time points and parameters, listens to the result, and can then save the discovered interesting sounds and export them along with logs/scenarios for subsequent analysis.

---

## General Interface Model

The interface should be organized as a multitrack timeline, logically similar at the same time to:

- automation lanes in a DAW;
- a keyframe editor;
- a Gantt chart, but only in terms of a shared timeline and event synchronization.

This is NOT a classic Gantt chart nor a list of `set/animate` commands.

### Coordinates

For each track:

- horizontal axis = time;
- vertical axis = parameter value;
- a parameter is defined by a set of editable points `(time, value)`;
- a point can be dragged horizontally to change the moment of the event;
- a point can be dragged vertically to change the value;
- the transition law between points is defined.

The UI may normalize the vertical scale to `0..1`, but the physical parameter value must retain its own range and units (`Hz`, gain, etc.).

---

## Separate Overall Sound Duration

An independent **overall scenario duration** parameter is required.

This is crucial: control parameters can be set or changed only at the beginning, but the resulting state must continue to sound for a long time.

Examples of the semantic model:

- quickly set a state and hold a long `AAAAA`;
- early events form `MA`, and then the state holds `AAAAAAA` for a long time;
- after the last keyframe point, the value of each control continues to be held until the total duration of the scenario.

The duration should not be automatically truncated at the last control point.

---

## Cyclic Playback

A **"Loop"** toggle must be provided.

In loop mode:

- the scenario plays again immediately after finishing;
- the user does not need to press Play every time;
- parameter/point changes must take effect in the next playback cycle;
- the user can move a point or value and hear several variant results over several consecutive repetitions.

This is a key mode specifically for manual sound discovery.

---

## Parameter Tracks

At the first stage, the editor should work with all existing `VoiceControlState` parameters.

For each track, you need to see:

- control name;
- its physical value;
- normalized position on the vertical scale;
- keyframe points;
- value holding between events;
- transition shape between points.

The editor should show the actual temporal trajectory of the parameter, not just active `animate` segments.

---

## Types of Transitions and Modulators

The architecture should be designed from the start so that more than just a single `animate` type exists between points.

An extensible model of a **time interval converter/modulator** is needed.

At a minimum, provide the ability to have different types in the future:

- hold / constant value;
- linear transition;
- bezier transition;
- periodic modulator;
- sinusoidal modulation by formula;
- impulse / burst;
- repeating beat;
- noise or other stochastic modulation;
- other time functions.

That is, in the future, not only a keyframe should be placed on the timeline, but also a **converter** that forms a complex parameter trajectory over a given interval.

Example meaning: a parameter can have a stable base, but on top of it, periodic modulation works over a time interval.

---

## Connection to Speech Physiology

The editor architecture cannot be designed as if each control is completely physically independent.

A human has a single speech apparatus:

- one tongue;
- one lip configuration;
- one mouth/oral cavity;
- one vocal tract;
- one vocal cord and airflow system.

You cannot simultaneously set mutually exclusive articulatory states as if two independent languages or two independent mouth shapes exist.

While the current DSP operates in the acoustic space (`f0`, periodicity, noise, resonances, gains, etc.), the editor must be designed so that in the future it is possible to introduce **coupled physiological/articulatory converters** that control multiple DSP controls coherently.

Examples of semantic converters that may appear later:

- mouth opening/closing;
- change in tongue position;
- pressure accumulation and release;
- short plosive burst;
- steady airflow;
- articulator tremor/beat for `R`;
- transition between articulatory states.

Important: this is not a requirement to immediately implement a full physiological model. The requirement is **not to block such a possibility with the architecture**.

---

## Special Meaning of Temporal Structure

The editor must proceed from the premise that sound only makes sense in time.

For example, a plosive consonant is not a single parameter value, but a sequence of phases:

- preparation/closure;
- accumulation;
- release;
- burst;
- noise or vocal tail;
- transition to the next sound.

And a sustained sound can be the result of a stable state that is held for a long time after a short input transition.

Therefore, the user must see and edit the **composition of phases over time**, rather than just a list of numbers.

---

## What to Save

A discovered experiment must be reproducible.

You need to be able to save:

- total duration;
- all tracks and keyframes;
- transition/converter types;
- their parameters;
- the final `VoiceScenario` or compilable representation;
- experiment metadata if necessary.

The existing `VoiceScenario` can be used as a runtime/execution format, but the UI should not be limited by its current command-centric form. A separate experimental/keyframe format that compiles into a runtime representation is acceptable.

---

## Connection to Existing Runtime

The editor must use the same shared `VoiceScenario` runtime used by the synthesizer and visualizer, after appropriate refactoring.

You cannot have separate parameter execution semantics only inside the UI.

The actual sound and actually displayed trajectories must come from a single state machine.

---

## What is NOT the Goal of the First Stage

Currently, it is not required to:

- automatically guess the correct sound;
- automatically construct a phoneme;
- add new acoustic metrics just for visualization;
- immediately build a full physiological model of the speech apparatus;
- replace the decoder.

The primary applied goal is **to maximize the simplification of manual experimentation**:

1. quickly place time points;
2. change values;
3. hear the result in a loop;
4. understand the sequence of one's own actions;
5. save successful options;
6. export the found scenario and its logs for further analysis.

---

## Result Criterion

The user can open the laboratory, set the total duration, enable `Loop`, place and move keyframe points of several parameters along a shared timeline, and continuously hear how changes affect the sound.

The interface should help specifically in **searching for pre-unknown sounds**, rather than requiring prior understanding of what JSON needs to be written.

An interactive multitrack editor for manual experimentation with synthesizer parameters over time, cyclic listening, and discovering interesting sounds.

Create VoiceSynthesizer Temporal Scenario Visual Laboratory

## Context

Right now we are not building a new synthesizer, nor are we trying to immediately recognize letters or phonemes. The goal of this stage is to understand fundamentally **what exact features we can reliably measure from a microphone recording**, using what methods, and with what temporal and frequency resolution.

Working hypothesis: first, we need to build a good decoder/analyzer as a measuring instrument, find stable features of known sounds, and only then go in the reverse direction—building generators that reproduce these features.

---

## Basic Signal Model

At `sampleRate = 48 kHz`, one second of mono PCM is 48,000 consecutive samples.

Each sample is a single final instantaneous amplitude value. Once internal sources are mixed down, the information about **which specific components combined to create an individual sample** is lost.

For example, the same final result can be produced by different internal sums. Therefore, reverse recovery of individual causes from a single sample is impossible.

Useful information appears only when analyzing the **trajectory over time**.

It is important to distinguish:

- `x[n]` — current waveform deviation;
- `Δx[n] = x[n] - x[n-1]` — direction and rate of change;
- `Δ²x[n]` — change in speed / steepness of the transient;
- long-term temporal structure;
- frequency structure.

A single `0` does not mean silence: inside an oscillation, it can be a high-speed zero-crossing. Rest is a prolonged sequence near zero without significant change.

---

## What Can Be Measured Directly in the Time Domain

Even before FFT, features can be reliably computed over short windows:

- RMS / total energy;
- peak amplitude;
- mean absolute amplitude;
- zero crossing rate;
- `mean(|Δx|)` — average rate of waveform change;
- `max(|Δx|)` — steepness of the strongest transient;
- second difference / local "acceleration";
- variance;
- crest factor;
- autocorrelation and its maxima;
- duration of silence/low-energy segments;
- onset / sudden energy appearances;
- decay / drop-off after an event.

These metrics potentially already separate different types of mechanics:

- sustained voiced sound;
- sustained noise/fricative segment;
- short burst/plosive;
- silence/closure;
- periodic tremor.

---

## Frequency Analysis

Frequency is not stored in PCM as a separate field. It is calculated from waveform repeatability.

For a window of length `N` samples, one can use:

- Fourier/FFT — which periodic components are present;
- autocorrelation — at what lag the waveform resembles itself;
- filter bank — energy in predefined bands.

For FFT, the trade-off is important:

- long window → good frequency resolution, poor temporal localization;
- short window → good temporal localization, coarser frequency resolution.

Examples:

- 1 s → approx. 1 Hz;
- 100 ms → approx. 10 Hz;
- 20 ms → approx. 50 Hz;
- 10 ms → approx. 100 Hz.

For speech, multi-scale analysis will likely be required: fast bursts visible at 5–10 ms, pitch/formants at 20–50 ms, long stable structures at 100+ ms.

---

## Frequency Tracks / Auditory Filter Bank

Instead of thousands of FFT bins, it makes sense to build a limited set of perceptual bands, for example, 24–40 bands on a logarithmic/ERB-like scale.

Basic idea:

`PCM -> short windows -> FFT/filter bank -> energy of each band over time`.

We get a fixed set of tracks:

`Band 1 energy(t)`
`Band 2 energy(t)`
`...`

The number of tracks is fixed, while activity within them fluctuates over time.

The first visual layer can be a heatmap:

- X = time;
- Y = frequency band;
- brightness = energy.

There is no need to encode many metrics at once via the thickness or color of a single curve. The basic vertical value of a track can simply be normalized energy `0..1`.

---

## Important Distinction: Total Energy vs. Its Distribution

`Uint8 0..255` is the format of an instantaneous sample, not an "energy budget" between bands.

For analysis, it is more useful to separate:

- window `totalEnergy`;
- relative energy distribution across bands.

For example:

`Frame { totalEnergy, bands[] }`

where upon normalization we can have:

`sum(bands) ≈ 1`.

This allows us to see separately:

- how loud the signal is overall;
- where the energy went across the spectrum.

---

## Finding Spectral Groups

One should not rigidly take the `top N` strongest bins.

A more sensible scheme:

1. compute all bands;
2. remove values below the adaptive noise floor/threshold;
3. find local maxima;
4. combine neighboring active bands into spectral groups;
5. link similar groups across adjacent time windows.

This turns fixed sensory tracks into floating spectral objects:

- center frequency;
- bandwidth;
- energy;
- duration;
- stability;
- onset/decay;
- periodic/noise-like character.

This is closer to how the auditory system groups components into objects than simply picking the ten strongest bins.

---

## Hypothesis on Speech Mechanisms

Instead of dozens of scalar controls for a future synthesizer, it is proposed to think in terms of separate physiologically/acoustically meaningful generators or mechanisms.

Examples:

- frication/constriction — prolonged noise mechanism;
- plosive release — very short burst after closure/pressure build-up;
- voicing — stable periodic source;
- trill — recurring quasi-periodic interruptions/modulations;
- nasal coupling — characteristic spectral structure;
- guttural mechanism — distinct excitation/resonance type.

Key idea: mechanisms should differ **not only in amplitude**, but in observable features:

- characteristic duration;
- frequency range;
- periodicity;
- onset/decay;
- spectral width;
- stability;
- temporal repeatability.

Then the decoder will potentially be able to extract them from the general PCM via dynamic signatures.

---

## Experimental Plan

Initially, work only with a microphone and known uttered material.

Order:

1. Record short fragments of 0.5–1.5 s.
2. Utter known sounds/syllables/words.
3. Build temporal and frequency metrics.
4. Visually search for stable signatures.
5. Verify the same signatures in other words and neighboring contexts.
6. Check the distinguishability of close classes (`S/Sh`, `P/T/K`, voiced/unvoiced, etc.).
7. Do not call a pattern a "letter" until it transfers across different contexts.
8. Achieve high distinguishability first for a single person, a single microphone, and controlled conditions.
9. Only after this, design the reverse generator.

The decoder must become a measuring instrument and then an automated generator test:

`human audio -> decoder -> mechanism signature`

`synthetic audio -> decoder -> mechanism signature`

Comparison is performed in the space of measurable features, not just subjectively "sounds similar / does not sound similar".

---

## Main Criterion for the Current Stage

Not speech recognition, and not speech synthesis.

We need to understand:

> what features we can truly stably and reproducibly measure from PCM, at what time scales, and which of them allow us to distinguish stable speech mechanisms.

At this stage, confirming measurability and distinguishability is more important than adding new entities to the generator.

Theoretically investigate what properties of PCM and the speech signal we can reliably measure in time and frequency before designing a new generator.

Tasks

Investigate measurable features of the speech signal before building a new synthesizer

Create VoiceSynthesizer Temporal Scenario Visual Laboratory

Investigate temporal causality of VoiceControlState → DSP → acoustic features

Investigate the inverse decoder: audio → VoiceScenario

Create VoiceScenario Visual Editor and Human ↔ AI Loop

Research qualitative programmatic human voice synthesis in the browser