Spectrograms, Mel Scale & Audio Features

The Problem

Take a 10-second 16 kHz clip. That is 160,000 floats, all in [-1, 1], almost perfectly uncorrelated with the label "dog barking" or "the word cat". The raw waveform has the information but in a form the model cannot easily extract. Two identical phonemes spoken 100 ms apart have completely different raw samples.

A spectrogram fixes this. It collapses the temporal detail where human perception ignores it (microsecond jitter) and preserves the structure where perception attends (which frequencies are energetic, over time windows of ~10–25 ms).

Mel spectrograms push further. Humans perceive pitch logarithmically: 100 Hz vs 200 Hz sounds "the same distance apart" as 1000 Hz vs 2000 Hz. The mel scale warps the frequency axis to match. A mel-scaled spectrogram is the single most important feature in speech ML from 2010 through 2026.

The Concept

STFT (Short-Time Fourier Transform). Slice the waveform into overlapping frames (typical: 25 ms window, 10 ms hop = 400 samples / 160 samples at 16 kHz). Multiply each frame by a window function (Hann is the default; Hamming slightly different tradeoff). FFT each frame. Stack the magnitude spectra into a matrix of shape (n_frames, n_freq_bins). That is your spectrogram.

Log-magnitude. Raw magnitudes span 5-6 orders of magnitude. Take log(|X| + 1e-6) or 20 * log10(|X|) to compress dynamic range. Every production pipeline uses log-magnitude, not raw magnitude.

Mel scale. Frequency f in Hz maps to mel m by m = 2595 \cdot \log_{10}(1 + f / 700). The mapping is roughly linear below 1 kHz and roughly logarithmic above. 80 mel bins covering 0–8 kHz is the standard ASR input.

Mel filterbank. A set of triangular filters spaced equally on the mel scale. Each filter is a weighted sum of adjacent FFT bins. Multiplying the STFT magnitude by the filterbank matrix gives the mel spectrogram in one matmul.

Log-mel spectrogram. log(mel_spec + 1e-10). Whisper's input. Parakeet's input. SeamlessM4T's input. The universal 2026 audio frontend.

MFCCs. Take the log-mel spectrogram, apply a DCT (type II), keep the first 13 coefficients. Decorrelates the features and compresses further. Dominant feature until about 2015 when CNNs/Transformers on raw log-mels caught up. Still used in speaker recognition (x-vectors, ECAPA).

Resolution trade. Larger FFT = better frequency resolution but worse time resolution. 25 ms / 10 ms is the audio-ML default; 50 ms / 12.5 ms for music; 5 ms / 2 ms for transient detection (drum hits, plosives).

Build It

Step 1: frame the waveform

pythonCopydef frame(signal, frame_len, hop):
    n = 1 + (len(signal) - frame_len) // hop
    return [signal[i * hop : i * hop + frame_len] for i in range(n)]

A 10-second 16 kHz clip with frame_len=400, hop=160 yields 998 frames.

Step 2: Hann window

pythonCopyimport math

def hann(N):
    return [0.5 * (1 - math.cos(2 * math.pi * n / (N - 1))) for n in range(N)]

Multiply element-wise before the FFT. Removes spectral leakage caused by truncating at non-zero endpoints.

Step 3: STFT magnitude

pythonCopydef stft_magnitude(signal, frame_len=400, hop=160):
    win = hann(frame_len)
    frames = frame(signal, frame_len, hop)
    return [magnitudes(dft([w * s for w, s in zip(win, f)])) for f in frames]

Production uses torch.stft or librosa.stft (FFT-backed, vectorized). The loop here is pedagogical; it runs on short clips in code/main.py.

Step 4: mel filterbank

pythonCopydef hz_to_mel(f):
    return 2595.0 * math.log10(1.0 + f / 700.0)

def mel_to_hz(m):
    return 700.0 * (10 ** (m / 2595.0) - 1)

def mel_filterbank(n_mels, n_fft, sr, fmin=0, fmax=None):
    fmax = fmax or sr / 2
    mels = [hz_to_mel(fmin) + (hz_to_mel(fmax) - hz_to_mel(fmin)) * i / (n_mels + 1)
            for i in range(n_mels + 2)]
    hzs = [mel_to_hz(m) for m in mels]
    bins = [int(h * n_fft / sr) for h in hzs]
    fb = [[0.0] * (n_fft // 2 + 1) for _ in range(n_mels)]
    for m in range(n_mels):
        for k in range(bins[m], bins[m + 1]):
            fb[m][k] = (k - bins[m]) / max(1, bins[m + 1] - bins[m])
        for k in range(bins[m + 1], bins[m + 2]):
            fb[m][k] = (bins[m + 2] - k) / max(1, bins[m + 2] - bins[m + 1])
    return fb

80 mels covering 0–8 kHz with n_fft=400 gives an (80, 201) matrix. Multiply the (n_frames, 201) STFT magnitude by the transpose to get (n_frames, 80) mel spectrogram.

Step 5: log-mel

pythonCopydef log_mel(mel_spec, eps=1e-10):
    return [[math.log(max(v, eps)) for v in frame] for frame in mel_spec]

Common alternatives: librosa.power_to_db (reference-normalized dB), 10 * log10(power + eps). Whisper uses a more involved clip + normalize routine (see Whisper's log_mel_spectrogram).

Step 6: MFCCs

pythonCopydef dct_ii(x, n_coeffs):
    N = len(x)
    return [
        sum(x[n] * math.cos(math.pi * k * (2 * n + 1) / (2 * N)) for n in range(N))
        for k in range(n_coeffs)
    ]

Apply DCT to each log-mel frame, keep the first 13 coefficients. That is your MFCC matrix. The first coefficient is usually dropped (it encodes overall energy).

Use It

The 2026 stack:

Rule of thumb: if you are not working on music, start with 80 log-mels. The burden of proof is on any deviation.

Pitfalls that still ship in 2026

• Mel count mismatch. Training with 80 mels, inference with 128 mels. Silent failure. Log the feature shape at both ends.
• Sample-rate mismatch upstream. Mels computed at 22.05 kHz look different from 16 kHz. Fix SR before featurization.
• dB vs log. Whisper expects log-mel, not dB-mel. Some HF pipelines autodetect; your custom code will not.
• Normalization drift. Per-utterance normalization during training, global normalization during inference. Production bug that doubles WER.
• Leakage from padding. Zero-padding the end of a clip produces a flat spectrum in the trailing frames. Pad symmetrically or replicate.

Ship It

Save as outputs/skill-feature-extractor.md. The skill picks feature type, mel count, frame/hop, and normalization for a given model target.

Exercises

1. Easy. Run code/main.py. It synthesizes a chirp (frequency swept 200 → 4000 Hz) and prints the argmax mel bin per frame. Plot (optional) and confirm it matches the sweep.
2. Medium. Re-run with n_mels in {40, 80, 128} and frame_len in {200, 400, 800}. Measure sharp-peak bandwidth across the time axis. Which combo resolves the chirp the best?
3. Hard. Implement power_to_db and compare ASR accuracy of a tiny CNN classifier on AudioMNIST using (a) raw log-mel, (b) dB-mel with ref=max, (c) MFCC-13 + delta + delta-delta. Report top-1 accuracy.

Key Terms

Further Reading

• Davis, Mermelstein (1980). Comparison of parametric representations for monosyllabic word recognition — the MFCC paper.
• Stevens, Volkmann, Newman (1937). A Scale for the Measurement of the Psychological Magnitude Pitch — the original mel scale.
• OpenAI — Whisper source, log_mel_spectrogram — read the reference implementation.
• librosa feature extraction docs — reference for mfcc, melspectrogram, and hop/window.
• NVIDIA NeMo — audio preprocessing — production-scale pipeline for Parakeet + Canary models.

Diagrams1 figure