Task: Create VoiceSynthesizer Temporal Scenario Visual Laboratory

Detailed Progress on the Temporal Scenario Visual Laboratory

What Has Been Implemented

A separate, new experimental synthesizer and visual editor has been created, intentionally disconnected from the former VoiceSynthesizer and its accumulated complex DSP model.

This important architectural decision was made deliberately: the goal of the current stage is not to maintain compatibility with old scenarios or reproduce past experiments, but to obtain the cleanest possible laboratory bench for manual sound exploration.

The current structure includes:

• a multi-track editor for temporal parameters;
• keyframe points on a common timeline;
• drag-and-drop by time and by value;
• Play / Pause / Stop;
• Loop mode;
• independent total scenario duration;
• timeline scrubbing (seek);
• display of current parameter values;
• transition types hold, linear, bezier, sine, burst, noise;
• a separate new audio synthesis based on the Web Audio API;
• a minimal acoustic model based on sawtooth + noise + formant filters + gain.

The UI now presents the main experimental tracks:

• F0 / Pitch;
• Periodicity;
• Noise;
• Formant 1;
• Formant 2;
• Formant 3;
• Gain.

Visually, the interface turned out well as a laboratory tool: the timeline is readable, events are visible, parameters can be quickly changed, and the total duration is not tied to the last point. This allows for building long sustained sounds and state sequences.

Important Clarification of Objectives

The new editor should NOT currently serve as an interface to the old DSP.

It is conceived as an independent clean experiment:

simple sources + simple filters + temporal trajectories -> audible result.

The main research question:

Is it possible at all, manually through controlled trial and listening, to find at least some recognizable speech-like sounds?

That is, the current task is not the reproducibility of old VoiceScenarios, but finding the most minimal working acoustic model.

Why a Separate New Synthesizer Was Chosen

The previous synthesis branch accumulated many interdependent experiments and parameters. It is now difficult to understand where exactly the error lies, and changing one mechanism can easily break another.

Therefore, a clean bench is needed as a controlled environment where each new mechanism is added consciously and its effect can be heard in isolation.

The principle of further work:

minimal engine -> manual search -> observation -> discovery of missing mechanism -> add only that mechanism -> repeat experiment.

What the First Practical Check Showed

According to the developer agent, the set of features looks complete, but an actual verification of the result showed that the report was overly optimistic.

At the current stage:

• most of the prepared sounds/presets are practically inaudible or do not yield the expected result;
• the agent's report in itself cannot be considered proof of functionality;
• among simple recognizable effects, only a hissing sound close to S is actually audible so far;
• full-fledged A, M, R, T and other sounds have not yet been obtained manually or confirmed by ear.

This is not considered a failure of the experiment. On the contrary, the bench now makes it possible to figure out which exact mechanisms are minimally necessary for the emergence of specific sound classes.

Current Practical Value of the Laboratory

The practical value of the editor has not yet been definitively proven.

A more basic thing is currently being tested:

1. Is there an audible causal link between a specific parameter and the result.
2. Is it possible, by moving one or more points manually, to obtain a predictably changing sound.
3. Is it possible to find stable acoustic classes:
   • vowel-like;
   • noise/fricative;
   • plosive;
   • nasal;
   • periodic/trembling;
   • other speech-like classes.
4. Is it possible to save the found configuration as a reproducible recipe.

If these points are confirmed, the laboratory is already useful as a research tool, even if the sound quality is still far from human speech.

Important Conceptual Conclusion

There is no need to build a complex physiological speech model in advance.

It is more correct to first search for minimal acoustic mechanisms manually.

For example:

• if R is impossible to obtain without a periodic modulator, this will become the experimental basis to add such a mechanism;
• if T requires a sequence of silence -> burst -> noise tail, this should also stem from manual experiment;
• if a certain vowel type requires a stable F0 + F1/F2/F3, this must be found and confirmed by ear.

That is, new features are added not by assumption, but by the observed deficit of the current model.

Current Limitations and Questions

1. Each existing control needs to be tested individually and in minimal combinations.
2. It is necessary to understand which of the current transition types are actually useful and which are still decorative.
3. burst, sine, repeating oscillations, and noise modulations conceptually may turn out to be not just interpolations, but a separate class of temporal modulators. Perhaps later it is worth separating:
   • envelope / keyframes;
   • modulators on top of the base trajectory.
4. It is necessary to check how correctly the physical ranges of parameters and their zero states are organized.
5. Convenient saving of successful sounds and scenarios needs to be added when truly interesting results appear.
6. Spectrograms and other methods of result analysis may be needed later, but the priority now is auditory manual search and causal clarity of the interface.

Criteria for Immediate Success

Not "the synthesizer already knows how to talk" and not "presets match letters."

The immediate success criterion is much simpler:

The user can turn on Loop, move a keyframe point or parameter, hear several options in a row, and consistently find at least some recognizable speech-like sounds, understanding what change led to them.

After finding interesting sounds, the scenario and logic will be used for subsequent analysis and for making decisions about which new mechanisms are truly worth adding.

Brief progress: the current implementation needs to be rewritten from scratch

Actual testing showed that the current component cannot be considered a viable foundation for further sound experiments.

Main issues:

• The audio architecture is fundamentally unsuitable for high-quality synthesis. Parameters are updated in chunks via Web Audio nodes during playback, whereas for controlled experimental synthesis, it is more logical to pre-calculate PCM/an array of sample values based on the entire timing logic and pass the ready buffer to the AudioContext for playback.
• The current DSP itself is poorly structured: a sawtooth passes through three serially connected narrow bandpass-formant filters, which almost destroys the periodic part; meanwhile, the noise goes through a separate direct path.
• The UI timeline and audio timeline effectively have different clocks. Loop, seek, pause/play, and real audio time do not form a single state machine.
• Sound control via requestAnimationFrame (~60 Hz) is unsuitable for short acoustic events, bursts, and fast transitions; additional smoothing blurs the events even further.
• The lifecycle of audio sources contains race conditions and inconsistent states: repeated Play works unreliably, Stop can lead to unexpected sound resumption, and old callbacks/source refs can outlive state changes.
• There are far too many independent refs/states/hooks in the UI/UX responsible for parts of a single playback state. Instead, a single useReducer / finite state machine with explicit stopped/playing/paused states and deterministic transitions is needed.
• The implementation report turned out to be significantly more optimistic than the actual behavior: the interface visually looks like a finished laboratory, but basic user scenarios like Play/Stop/Replay and the sound results themselves are unreliable.

Conclusion: piecemeal fixing of the current implementation is impractical. The component needs to be rewritten from scratch, keeping only the general idea of a visual laboratory: timeline, parameter tracks, keyframes, loop, and manual sound discovery. The new version should be built around a single state machine and the pre-calculation of the audio buffer from the scenario.