Worklog for task "Investigate measurable features of the speech signal before building a new synthesizer"

Interim Summary: Current Experiment Failed

At the current stage, none of the implemented/proposed sensors have yielded any useful, visually stable, or interpretable information for distinguishing speech sounds.

It is important to document this as a negative result rather than trying to "force" the experiment with optimistic heuristics.

What Was Tried

Time-domain sensors were discussed and/or proposed:

• RMS / average energy;
• peak amplitude;
• zero crossing rate;
• mean |Δx|;
• max |Δx|;
• autocorrelation / periodicity;
• pitch estimate.

Then an attempt was made to build a rough sibilanceSensor for SNR via:

• high-frequency energy;
• spectral centroid;
• energy separation below/above an arbitrary threshold;
• spectral concentration.

After that, a rough frequency-band sensor with 12 logarithmic bands ranging approximately from 60 Hz to 14 kHz was proposed.

Practical result: none of these approaches has yet shown even an approximately useful picture that would allow us to confidently say the measurements reflect differences between known pronounced sounds.

---

Main Reasons for Failure

1. The AI Began "Designing a Solution" Too Early Instead of Conducting a Measurement Experiment

The main mistake was the premature transition from the question "what are we actually seeing in the data?" to the question "what sensor should detect the SNR?".

As a result, heuristics appeared, such as an arbitrary threshold of 5.5 kHz, sibilance-presence, a set of pre-selected frequency bands, and expected charts before even a single confirming series of real measurements was obtained.

This repeats a previously observed problem: the code looks plausible, but the foundation beneath it is not proven by experiment.

2. Features Are Measurable on Their Own, but Their Diagnostic Value Is Not Proven

RMS, ZCR, |Δx|, autocorrelation, and spectral metrics are indeed calculated correctly as waveform properties. However, this does not imply they are useful for distinguishing phonemes or speech mechanisms in a specific recording.

For example, two different sounds may have very similar RMS/ZCR, while the exact same sound can drastically change these values depending on loudness, distance to the microphone, and context.

3. Normalization Can Destroy the Exact Information We Are Trying to See

Attempting to normalize band energies in each frame so that their sum equals about 1 makes spectral distribution visualization convenient, but simultaneously removes absolute intensity information.

When comparing sibilants and non-sibilants, the absolute noise energy itself may be an important feature.

We cannot decide in advance which signal components are "interfering scale" until we have tested them experimentally.

4. Time Windows Are Too Coarse

A fixed window of 20–40 ms is technically convenient, but different speech mechanisms operate on different scales:

• a burst may take just a few milliseconds;
• pitch requires several periods;
• formant structures require a different trade-off;
• sustained fricative noise can last hundreds of milliseconds.

A single window scale inevitably hides some events.

5. Frequency Bands Are Too Coarse

12 bands like 2.5–4 kHz, 4–6.3 kHz, 6.3–10 kHz can be so wide that real local spectral differences within them simply disappear.

At the same time, jumping straight to dozens of bands without proven utility will only create more data, not more understanding.

6. Lack of a Standard Experimental Procedure

There is currently no strict set of comparative recordings:

• identical microphone;
• identical distance;
• identical loudness;
• multiple repetitions of the same sound;
• several different sounds;
• saved raw PCM data;
• identical processing;
• ability to overlay/compare metrics across repetitions.

Without this, it is impossible to distinguish a stable phoneme feature from random recording variation.

7. Analyzing "a Single Recording" Rather Than Repetition Statistics So Far

A single chart proves almost nothing.

We need at least a few repetitions of each class and a comparison of feature distributions:

• intra-class variability;
• inter-class distance;
• stability over time;
• sensitivity to loudness and microphone.

If the scatter of the same sound is greater than the difference between two sounds, the sensor is useless for our task.

8. Premature Binding to Letters

С, Ш, Ф are useful test classes, but a sensor should not start with an assumption about a letter.

It is more correct to first find objective differences between sets of recordings, and only then look at what they correspond to articulation-wise and phonetically.

9. Potential Issues with the Original audioData Itself

Before drawing further conclusions, it is necessary to separately verify:

• what exactly audioData contains;
• whether it is truly PCM from the microphone;
• data type (Uint8, Float32, other);
• correct center/DC offset;
• actual sampleRate;
• whether there is any browser/microphone processing prior to analysis: AGC, noise suppression, echo cancellation;
• whether audioData is already the result of an AnalyserNode with limited/altered semantics.

If the input signal is already automatically normalized or filtered by the browser, this can completely alter the observed features.

10. Visualization Can Also Hide Differences

If each sensor independently chooses a normalized 0..1 range, visually any small differences might look significant, and large absolute differences might look identical.

For a research tool, raw values with units of measurement must be primary, while visual normalization should be a separate, transparent layer.

---

What We Consider the Correct Next Step

Do not write another "smart sensor."

First, build a minimal experimental infrastructure:

1. save several short raw recordings with a known label;
2. guaranteed knowledge of the PCM format and sampleRate;
3. ability to render the waveform itself without transformations;
4. ability to build a standard detailed spectrogram/STFT without attempting classification;
5. compare several repetitions of the same sound side by side;
6. only after visually discovering a stable difference, extract it into a separate sensor-helper;
7. evaluate the sensor based on repetition statistics rather than a single successful chart.

Main conclusion of the current stage:

there is currently no basis to claim that the proposed sensors provide useful information for sound recognition. The experiment needs to be returned to a lower level: first, honestly observe the waveform and spectrum of real recordings, and then extract only those metrics whose usefulness is confirmed by repeatable data.