Task: Investigate measurable features of the speech signal before building a new synthesizer

Context

Right now we are not building a new synthesizer, nor are we trying to immediately recognize letters or phonemes. The goal of this stage is to understand fundamentally what exact features we can reliably measure from a microphone recording, using what methods, and with what temporal and frequency resolution.

Working hypothesis: first, we need to build a good decoder/analyzer as a measuring instrument, find stable features of known sounds, and only then go in the reverse direction—building generators that reproduce these features.

---

Basic Signal Model

At sampleRate = 48 kHz, one second of mono PCM is 48,000 consecutive samples.

Each sample is a single final instantaneous amplitude value. Once internal sources are mixed down, the information about which specific components combined to create an individual sample is lost.

For example, the same final result can be produced by different internal sums. Therefore, reverse recovery of individual causes from a single sample is impossible.

Useful information appears only when analyzing the trajectory over time.

It is important to distinguish:

• x[n] — current waveform deviation;
• Δx[n] = x[n] - x[n-1] — direction and rate of change;
• Δ²x[n] — change in speed / steepness of the transient;
• long-term temporal structure;
• frequency structure.

A single 0 does not mean silence: inside an oscillation, it can be a high-speed zero-crossing. Rest is a prolonged sequence near zero without significant change.

---

What Can Be Measured Directly in the Time Domain

Even before FFT, features can be reliably computed over short windows:

• RMS / total energy;
• peak amplitude;
• mean absolute amplitude;
• zero crossing rate;
• mean(|Δx|) — average rate of waveform change;
• max(|Δx|) — steepness of the strongest transient;
• second difference / local "acceleration";
• variance;
• crest factor;
• autocorrelation and its maxima;
• duration of silence/low-energy segments;
• onset / sudden energy appearances;
• decay / drop-off after an event.

These metrics potentially already separate different types of mechanics:

• sustained voiced sound;
• sustained noise/fricative segment;
• short burst/plosive;
• silence/closure;
• periodic tremor.

---

Frequency Analysis

Frequency is not stored in PCM as a separate field. It is calculated from waveform repeatability.

For a window of length N samples, one can use:

• Fourier/FFT — which periodic components are present;
• autocorrelation — at what lag the waveform resembles itself;
• filter bank — energy in predefined bands.

For FFT, the trade-off is important:

• long window → good frequency resolution, poor temporal localization;
• short window → good temporal localization, coarser frequency resolution.

Examples:

• 1 s → approx. 1 Hz;
• 100 ms → approx. 10 Hz;
• 20 ms → approx. 50 Hz;
• 10 ms → approx. 100 Hz.

For speech, multi-scale analysis will likely be required: fast bursts visible at 5–10 ms, pitch/formants at 20–50 ms, long stable structures at 100+ ms.

---

Frequency Tracks / Auditory Filter Bank

Instead of thousands of FFT bins, it makes sense to build a limited set of perceptual bands, for example, 24–40 bands on a logarithmic/ERB-like scale.

Basic idea:

PCM -> short windows -> FFT/filter bank -> energy of each band over time.

We get a fixed set of tracks:

Band 1 energy(t) Band 2 energy(t) ...

The number of tracks is fixed, while activity within them fluctuates over time.

The first visual layer can be a heatmap:

• X = time;
• Y = frequency band;
• brightness = energy.

There is no need to encode many metrics at once via the thickness or color of a single curve. The basic vertical value of a track can simply be normalized energy 0..1.

---

Important Distinction: Total Energy vs. Its Distribution

Uint8 0..255 is the format of an instantaneous sample, not an "energy budget" between bands.

For analysis, it is more useful to separate:

• window totalEnergy;
• relative energy distribution across bands.

For example:

Frame { totalEnergy, bands[] }

where upon normalization we can have:

sum(bands) ≈ 1.

This allows us to see separately:

• how loud the signal is overall;
• where the energy went across the spectrum.

---

Finding Spectral Groups

One should not rigidly take the top N strongest bins.

A more sensible scheme:

1. compute all bands;
2. remove values below the adaptive noise floor/threshold;
3. find local maxima;
4. combine neighboring active bands into spectral groups;
5. link similar groups across adjacent time windows.

This turns fixed sensory tracks into floating spectral objects:

• center frequency;
• bandwidth;
• energy;
• duration;
• stability;
• onset/decay;
• periodic/noise-like character.

This is closer to how the auditory system groups components into objects than simply picking the ten strongest bins.

---

Hypothesis on Speech Mechanisms

Instead of dozens of scalar controls for a future synthesizer, it is proposed to think in terms of separate physiologically/acoustically meaningful generators or mechanisms.

Examples:

• frication/constriction — prolonged noise mechanism;
• plosive release — very short burst after closure/pressure build-up;
• voicing — stable periodic source;
• trill — recurring quasi-periodic interruptions/modulations;
• nasal coupling — characteristic spectral structure;
• guttural mechanism — distinct excitation/resonance type.

Key idea: mechanisms should differ not only in amplitude, but in observable features:

• characteristic duration;
• frequency range;
• periodicity;
• onset/decay;
• spectral width;
• stability;
• temporal repeatability.

Then the decoder will potentially be able to extract them from the general PCM via dynamic signatures.

---

Experimental Plan

Initially, work only with a microphone and known uttered material.

Order:

1. Record short fragments of 0.5–1.5 s.
2. Utter known sounds/syllables/words.
3. Build temporal and frequency metrics.
4. Visually search for stable signatures.
5. Verify the same signatures in other words and neighboring contexts.
6. Check the distinguishability of close classes (S/Sh, P/T/K, voiced/unvoiced, etc.).
7. Do not call a pattern a "letter" until it transfers across different contexts.
8. Achieve high distinguishability first for a single person, a single microphone, and controlled conditions.
9. Only after this, design the reverse generator.

The decoder must become a measuring instrument and then an automated generator test:

human audio -> decoder -> mechanism signature

synthetic audio -> decoder -> mechanism signature

Comparison is performed in the space of measurable features, not just subjectively "sounds similar / does not sound similar".

---

Main Criterion for the Current Stage

Not speech recognition, and not speech synthesis.

We need to understand:

what features we can truly stably and reproducibly measure from PCM, at what time scales, and which of them allow us to distinguish stable speech mechanisms.

At this stage, confirming measurability and distinguishability is more important than adding new entities to the generator.