Worklog for task "Investigate temporal causality of VoiceControlState → DSP → acoustic features"

Interim Conclusions on Temporal Causality and Parameter Observability

Single experiments were conducted for VoiceControlState -> DSP -> acoustic features across all major control parameters.

What the Experiments Showed

During a single sweep, when only one control changes while the others remain fixed, many parameters show a strong correlation with the observed acoustic features.

The most pronounced results:

• sourceLevel is strongly correlated with peak, rms, and spectral peak magnitude;
• periodicity is also strongly reflected in peak/rms, but these features overlap with amplitude controls;
• f0Hz manifests well through pitch-related features, ZCR, and high-frequency energy distribution;
• resonance1Freq manifests well through the position of the first spectral peak;
• resonance2Freq is reflected through rolloff/formant-related features;
• resonance*Gain are clearly observed through amplitude features;
• outputLevel is reflected almost perfectly in peak/rms.

At the same time, noiseLevel, glottal parameters, bandwidth parameters, and resonance3Freq are observed significantly less effectively with the current feature set.

Important Interpretation Correction

High correlation in a single sweep does NOT imply direct parameter recoverability.

A distinction must be made between:

1. Sensitivity — whether the observed feature changes when we modify the control.
2. Identifiability — whether it can be determined from the observed sound that this specific control changed rather than another one.

For example:

• sourceLevel, outputLevel, and resonanceGain all significantly alter rms/peak;
• periodicity in the current experiment also strongly affects the same amplitude features.

Consequently, these controls are sensitive and observable, but it has not yet been proven that they are distinguishable from one another in a real-world scenario where multiple parameters change simultaneously.

Current Categorization of Controls by Observability Nature

1. Frequency coordinates — most promising for direct reconstruction:

• f0Hz;
• resonance1Freq;
• resonance2Freq;
• potentially resonance3Freq after improving spectral analysis.

They alter not only the total energy but also the spectral structure.

2. Amplitude/scaling controls — observable, but overlapping:

• sourceLevel;
• outputLevel;
• resonance1Gain;
• resonance2Gain;
• resonance3Gain;
• partially periodicity.

They require joint reconstruction or additional features because similar effects can be achieved through different combinations of parameters.

3. Shaping controls — poorly captured by the current log:

• glottalOpenPhase;
• glottalReturnPhase;
• resonance*Bandwidth;
• noiseLevel;
• partially resonance3Freq.

These likely require period shape features, spectral envelope, harmonic/noise structure, LPC, or other more specialized measurements.

Temporal Conclusions

In all single experiments, the reaction was detected almost immediately (~0.2 ms), and a 20 ms window proved sufficient for the current feature set. However, the identical lag across all parameters likely reflects the resolution of the measurement methodology and window time-stamping rather than a true DSP physical constant.

Separately, a long resonator memory was detected (~250 ms in the current experiment). This value cannot yet be considered universal: the decay time must depend on frequency/bandwidth/Q and requires a separate sweep over the resonator parameters.

What Became Clear About the Decoder Task Itself

The scenario and the acoustic log exist at different levels of representation:

VoiceControlState -> internal DSP state -> waveform -> spectrum/features.

Control parameters may remain nearly constant over time while generating a complex high-frequency waveform. Therefore, the control graph should not necessarily resemble the acoustic feature graph.

To reconstruct parameters, one should investigate not the correspondence of a single control to a single feature, but the mapping:

vector(features over a time window) -> vector(controls).

The next crucial experiment must test simultaneous changes in multiple controls. This will reveal true identifiability, not just sensitivity.

Practical Conclusion

Single sweeps have already answered the question: "Does the sound react to a specific control, and through which features is this visible?"

The next question is: "Can we distinguish the change of one control from another when they act simultaneously?"

Furthermore, studies indicate that future work requires a convenient interactive time editor: not for editing already generated JSON scenarios, but as a laboratory for manually searching for sounds and causal relationships between parameter temporal trajectories and the outcome.