Overcoming the High Dimensionality of Raw Audio Signals
Standard CD quality audio produces 44,100 samples per second per channel, generating 88,200 continuous scalar values every second. Feeding raw audio samples directly into self attention layers causes instantaneous GPU memory exhaustion due to quadratic O(N^2) scaling.
To make audio computable within Large Language Models, engineers use two foundational tokenization strategies:
STFT transforms time series into 80 channel log Mel spectrograms at 100 frames/s.
Strided 1D convolutional networks downsample raw audio by factors of 320 to 640.
Multi codebook vector quantization maps continuous latents to integer token IDs.