Task: Investigate temporal causality of VoiceControlState → DSP → acoustic features

Objective

Investigate not the instantaneous correspondence of the form parameter[t] -> sample[t], but the temporal causality in the chain:

VoiceControlState(t) -> internal DSP state -> PCM -> acoustic features in a time window.

We need to understand which control parameter actually affects what, after how long this influence appears, how long it persists, and what observable features allow it to be recovered.

This is necessary for correct inverse decoding of Sound -> VoiceScenario.

---

Context and Problem

The current VoiceScenario specifies control parameters, but the output PCM is generated by a dynamic system with internal state:

• voicePhase;
• resonator1.low, resonator1.band;
• resonator2.low, resonator2.band;
• resonator3.low, resonator3.band;
• PRNG state;
• active automation.

Therefore, even constant control parameters produce a complex, high-frequency waveform.

Examples:

• constant f0Hz does not mean a constant waveform — it sets the rotation speed of voicePhase;
• constant noiseLevel controls the amplitude of a random source that changes every sample;
• constant resonance* controls stateful filters whose internal state changes every sample;
• observable features (RMS, spectralFlatness, dominantFrequency, etc.) depend simultaneously on several control parameters and DSP history.

Consequently, direct heuristics like:

RMS -> sourceLevel

dominantFrequency -> f0Hz

are not a valid inverse transformation model.

---

Main Hypothesis

For speech and this synthesizer, we need to analyze not a single sample and not a single log frame, but a time window.

Instead of:

parameter[t] -> sample[t]

investigate:

parameter trajectory[t0:t1] -> audio response[t0:t1+Δ]

where Δ is the possible reaction lag and/or decay time of the internal DSP state.

Working window scales to test:

• 5 ms;
• 10 ms;
• 20 ms;
• 30 ms;
• 50 ms;
• 100 ms if necessary.

For each parameter, it is necessary to determine at which time scale its influence is observed most stably.

---

What to Investigate for Each VoiceControlState Parameter

At a minimum:

• sourceLevel
• periodicity
• noiseLevel
• f0Hz
• glottalOpenPhase
• glottalReturnPhase
• resonance1Freq
• resonance1Bandwidth
• resonance1Gain
• resonance2Freq
• resonance2Bandwidth
• resonance2Gain
• resonance3Freq
• resonance3Bandwidth
• resonance3Gain
• outputLevel

For each parameter, determine:

1. Sensitivity — how strongly its change affects PCM and features.
2. Lag — how long after a control change this influence becomes observable.
3. Decay time — how long the effect persists after the parameter changes.
4. Locality — whether the parameter affects mainly a specific spectrum range / temporal characteristic or globally.
5. Uniqueness — whether the observed effect can distinguish a change in this parameter from a change in another.
6. Interactions — which pairs of parameters produce a non-additive effect.

---

Experiment 1: Single Parameter Sweep

For each parameter:

1. Fix all other parameters in a single baseline state.
2. Change only the parameter under study across a grid of values.
3. Synthesize PCM of equal duration for each value.
4. Simultaneously collect:
   • full VoiceControlState(t);
   • internal DSP trace;
   • PCM;
   • acoustic features by windows.
5. Compare the result with the baseline.

Example for f0Hz:

• 80 Hz
• 100 Hz
• 120 Hz
• 150 Hz
• 180 Hz
• 220 Hz
• 300 Hz

Example for resonance1Freq:

• fix the source and other resonators;
• move only resonance1Freq;
• see which spectral peak actually moves and how.

---

Experiment 2: Step Change Over Time

Test the dynamic response of the system.

Example:

t < 100 ms: resonance1Freq = 600

t >= 100 ms: resonance1Freq = 800

Analyze windows:

• 60–80 ms
• 80–100 ms
• 100–120 ms
• 120–140 ms
• 140–160 ms
• 160–200 ms

Objective:

• determine the start time of the observed change;
• measure the lag;
• understand the shape of the transient process;
• measure the stabilization time.

Repeat for other parameters.

---

Experiment 3: Impulse / Short-Term Perturbation

For stateful parameters, it is especially important to measure system memory.

Scheme:

baseline -> parameter + δ for a short interval -> baseline

For example:

• 100 ms baseline;
• 10–20 ms change;
• return to baseline;
• observation for another 100–200 ms.

Measure:

• how fast the effect appears;
• whether it persists after the parameter returns;
• how long it takes for the DSP to return to its initial state.

For resonators, this should show the real memory duration of low/band states.

---

Experiment 4: Parameter Pairs

After single sweeps, test interactions for at least the most coupled pairs:

• sourceLevel × outputLevel
• periodicity × noiseLevel
• f0Hz × glottalOpenPhase
• f0Hz × glottalReturnPhase
• resonanceFreq × resonanceGain
• resonanceFreq × resonanceBandwidth
• sourceLevel × resonanceGain

Objective — to understand whether parameter effects can be considered independent or if they need to be recovered jointly.

---

What Data to Log

Control Level

• all VoiceControlState(t) values;
• active automation;
• timestamp of each set / animate command.

Internal DSP Trace

At a minimum:

• voicePhase;
• periodicSource;
• noiseSource;
• excitation;
• resonator1.low / resonator1.band;
• resonator2.low / resonator2.band;
• resonator3.low / resonator3.band;
• separate output of each resonator before summation;
• value before clamp;
• value after clamp.

Acoustic Features by Windows

At a minimum:

• RMS / energy;
• zero crossing rate;
• autocorrelation;
• pitch estimate;
• periodicity estimate;
• spectral centroid;
• spectral flatness;
• spectral peaks;
• energy by bands;
• formant-like peaks / spectral envelope;
• LPC if necessary.

Important: treat dominantFrequency only as one of the spectral features, and not as a direct f0Hz.

---

Time Windows and Logging Frequency

It is necessary to separately test the impact of throttling.

The current log may hide significant temporal structure. Therefore, for the study, it is desirable to have the most frequent internal trace possible, and then programmatically aggregate it into windows.

Recommended approach:

• DSP trace — as close to sample-level as possible or with controlled decimation;
• acoustic features — computed over 5/10/20/30/50 ms windows;
• window overlap: 50–75% for smooth temporal analysis;
• do not mix data collection frequency and analytic window size.

---

Expected Results

Build an experimental map:

VoiceControl parameter -> observable consequences

For each parameter, record:

• which features react;
• direction of reaction;
• scale of reaction;
• lag;
• memory duration;
• dependence on other parameters;
• how recoverable the parameter is from audio.

It is advisable to format this into a table like:

Control	Observable Features	Lag	Window	Uniqueness	Comment
f0Hz	autocorrelation peak, harmonic spacing	...	...	high/medium/...	...
resonance1Freq	spectral envelope / peak shift	...	...	...	...
...	...	...	...	...	...

---

Result Criterion

After the study, it should become clear:

1. Which VoiceControlState parameters can be recovered directly from audio.
2. Which ones can only be recovered jointly with other parameters.
3. Which parameters are not observable at all with the current feature set.
4. What additional features need to be added to the logger.
5. What temporal context the decoder needs for each parameter class.
6. Whether the decoder can be built as an analytical/heuristic transformation or if an optimization/learned approach is required.

The main result is not just a new decoder, but a physically and experimentally confirmed correspondence model between the synthesizer control space and the observable sound over time.