Skip to main content
Acoustic Neural Codec Engine

Audio Tokenizer

Explore audio tokenizer acoustics, frame rates, and multi-codebook Residual Vector Quantization (RVQ) across Meta AudioCraft, EnCodec, Descript Audio Codec, Wav2Vec, GPT tokenizer, and Whisper.

Audio Tokens 0
Duration 0:00.0
Sample Rate 24.0 kHz
Estimated Cost ~$0.0004

Upload or drag & drop an audio file

MP3, WAV, OGG, FLAC, AAC, or WebM. 100% processed locally in your browser.

speech-sample.wav 24kHz Mono
0:00.0 / 0:04.2
Rate: 50 tokens/s

Model Tokenization Specifications

Architecture: Encoder-Decoder Transformer (128-Mel Bins, 50Hz)
Frame Stride / Hop: 20ms (50 acoustic frames/sec)
Target Sample Rate: 16,000 Hz
Chunking Strategy: 30-second fixed context windows

Acoustic Token Segments

Loading audio tokens…
Numeric Token ID Sequence
Numeric token IDs…
Acoustic Neural Codecs Deep Dive

Audio Tokenization & Neural Codecs Breakdown

A step by step mathematical and signal processing derivation of how modern speech and music models compress high dimensional acoustic waveforms into discrete tokens via Short Time Fourier Transforms, Mel filterbanks, and Residual Vector Quantization.

Audio Signal & Tokenization Cheat Sheet

Acoustic Codec Setup
Raw Audio Rate
44,100 Hz
Continuous Samples/s
Whisper STFT
100 Hz
Frames/s (10ms Hop)
EnCodec Frame
75 Hz
Tokens/s per Codebook
Mel Channels
80 / 128
Filterbank Frequency Bins
RVQ Codebooks
8 - 32
Residual Quantizers
Target Bitrate
1.5 - 6 kbps
Pristine Speech Quality
01

Overcoming the High Dimensionality of Raw Audio Signals

Waveform: x(t) ∈ ℝ

Standard CD quality audio produces 44,100 samples per second per channel, generating 88,200 continuous scalar values every second. Feeding raw audio samples directly into self attention layers causes instantaneous GPU memory exhaustion due to quadratic O(N^2) scaling.

To make audio computable within Large Language Models, engineers use two foundational tokenization strategies:

1. Spectral Features

STFT transforms time series into 80 channel log Mel spectrograms at 100 frames/s.

2. Latent Encoders

Strided 1D convolutional networks downsample raw audio by factors of 320 to 640.

3. Discrete RVQ

Multi codebook vector quantization maps continuous latents to integer token IDs.

Compression Factor Principle Step 1
Ratio = Sample_Rate / Token_Rate
Raw Samples / Sec: 24,000 samples
Tokens / Sec (EnCodec): 75 tokens
Compression Factor: 320× reduction in tokens
Enables models to ingest minutes of speech in a standard 128k context window!
02

Mathematical Foundations of Mel Scale and RVQ

RVQ: q_k = argmin ||r_k - c||^2

Human ears perceive pitch logarithmically, differentiating subtle differences at low frequencies with high resolution while grouping higher frequencies into broader spectral bins:

1. Mel Frequency Conversion:

m = 2595 · log10(1 + f / 700) warps linear Hertz into perceptual Mel pitch bands.

2. Short Time Fourier Transform (STFT):

Windowed Fourier transforms convert 1D waveforms into 2D time frequency spectrograms.

3. Residual Vector Quantization (RVQ):

Stacked codebooks recursively quantize approximation residuals r_k = z - ∑ q_i, capturing acoustic nuances with minimal tokens.

RVQ Step by Step Step 2
q_1 = argmin_c ||z - c_1||^2
r_1 = z - q_1 → q_2 = argmin_c ||r_1 - c_2||^2
Codebook Entries: 1,024 vectors (10 bits)
Typical Stages K: 8 to 32 codebooks
Bitrate at 75 Hz: 6 kbps (75 × 8 × 10 bps)
03

Time Frequency Windowing & STFT Mechanics

STFT: Window = 25ms, Hop = 10ms

In systems like Whisper and speech transformers, raw continuous audio is windowed through Hann tapering functions to eliminate edge spectral leakage before Fourier transformations:

Window Length (25ms):

At a 16 kHz sample rate, a 25ms window extracts 400 audio samples, providing adequate frequency resolution for human vocal tracts.

Hop Size (10ms):

Shifting by 10ms (160 samples) produces 100 spectral frames per second with 60% overlap between adjacent frames.

Spectrogram Sizing Rule Step 3
Frames = floor((N_samples - N_fft) / Hop_length) + 1
30 Second Whisper Audio: 480,000 raw samples
Spectrogram Shape: 80 channels × 3,000 frames
Whisper Encoder Reduction: Downsampled 2× to 1,500 tokens
04

Neural Audio Codec Architectural Comparison

Codec Benchmark
Codec Architecture Input Sample Rate Frame Rate RVQ Codebooks Target Bitrate Primary Application
Meta EnCodec 24 kHz / 48 kHz 75 Hz 8 to 32 codebooks 1.5, 3, 6, 12 kbps AudioCraft & Voice Generation
Descript DAC 44.1 kHz 86 Hz 9 codebooks 8 kbps Pristine Hi Fi Music & Vocals
Google SoundStream 24 kHz 50 Hz 3 to 8 codebooks 3 - 6 kbps Real time Voice Communications
OpenAI Whisper 16 kHz 100 Hz (STFT) None (Continuous Mel) 80 channels Speech Recognition & Translation
05

Production Implementation Code: Torchaudio Mel Spectrogram

Runtime: Python 3.10+
audio_spectrogram.py torchaudio transforms
import torch
import torchaudio.transforms as T

def generate_whisper_mel_spectrogram(waveform: torch.Tensor, sample_rate: int = 16000):
    mel_transform = T.MelSpectrogram(
        sample_rate=sample_rate,
        n_fft=400,
        win_length=400,
        hop_length=160,
        n_mels=80,
        power=2.0
    )
    mel_spec = mel_transform(waveform)
    log_mel_spec = torch.log(torch.clamp(mel_spec, min=1e-5))
    print(f"Log-Mel Spectrogram Shape: {log_mel_spec.shape}")
    return log_mel_spec

# Synthesize 3 seconds of dummy audio at 16 kHz
dummy_audio = torch.randn(1, 16000 * 3)
generate_whisper_mel_spectrogram(dummy_audio)
06

Engineering Frequently Asked Questions

Neural Audio

Why does RVQ use multiple codebooks instead of one giant codebook?

A single codebook capable of representing the combinatorial diversity of audio would require 2^80 entries, which is computationally intractable. Stacking eight 10-bit codebooks achieves the exact same 80-bit expressive capacity with just 8,192 total stored vectors.

What is the difference between semantic and acoustic audio tokens?

Semantic tokens (from models like w2v-BERT or HuBERT) capture what words were spoken, discarding timbre and background noise. Acoustic tokens (from EnCodec or DAC) capture how it was spoken, preserving vocal timbre, pitch inflections, and acoustics.

Why is 16 kHz typically chosen for speech recognition models?

By the Nyquist-Shannon sampling theorem, 16 kHz captures all audio frequencies up to 8 kHz. Since human speech intelligibility and formant frequencies live almost entirely below 4 kHz, 16 kHz provides full phoneme clarity while eliminating unnecessary high frequency compute.

How do omni models generate audio in real time?

Frontier omni models predict discrete RVQ codebook indices autoregressively. These integer codes are passed through a lightweight convolutional neural vocoder (like HiFi-GAN) running in milliseconds on consumer GPUs.

Key Architectural Takeaway

Neural audio tokenization conquers the extreme data density of sound waves through strided convolutional encoders and hierarchical Residual Vector Quantization. By condensing 44,100 continuous audio samples per second into fewer than 100 discrete codebook tokens, modern transformers process and synthesize human speech with natural prosody and low latency.

matrix