Multimodal Tokenization: Vision Patch Embeddings and Acoustic Audio Codecs

The Multimodal Unification Challenge

Early deep learning systems treated language, vision, and audio as separate disciplines requiring distinct neural architectures: transformers for text, convolutional networks for images, and recurrent or spectrogram pipelines for audio.

Modern frontier models (including GPT 4o, Claude 3.5, Gemini 1.5, and Llama 3.2 Vision) abandon these siloed architectures. Instead, they unify all modalities into a single shared sequence of discrete token vectors. Once images and audio waveforms are transformed into vectors that share the same hidden dimension $d_{\text{model}}$ as text embeddings, the standard transformer attention backbone can process text, images, and audio indiscriminately.

The critical engineering challenge lies in tokenization: how do we discretize continuous two dimensional visual pixels and continuous one dimensional audio sound waves into discrete tokens without discarding crucial details?

Vision Tokenization: Vision Transformers (ViT)

The breakthrough paper An Image is Worth 16x16 Words (Dosovitskiy et al., 2020) demonstrated that self attention could replace convolutional layers entirely in image processing.

Patch Extraction and Linear Projection

Rather than computing attention over every individual pixel (which would require millions of pairwise calculations for a single image), a Vision Transformer divides an input image of shape $(H, W, C)$ into a grid of non overlapping square patches of size $P \times P$ (typically $14 \times 14$ or $16 \times 16$ pixels).

The total number of visual tokens produced is:

$$N = \frac{H \times W}{P^2}$$

For a standard $224 \times 224$ image with $P = 16$:

$$N = \frac{224 \times 224}{16 \times 16} = 14 \times 14 = 196 \text{ visual tokens}$$

Each individual patch is flattened into a one dimensional vector $x_p \in \mathbb{R}^{P^2 \cdot C} = \mathbb{R}^{16 \times 16 \times 3} = \mathbb{R}^{768}$. This patch vector is then multiplied by a learned linear projection matrix $E \in \mathbb{R}^{768 \times d_{\text{model}}}$:

$$z_0 = [x_{\text{class}}; x_p^1 E; x_p^2 E; \dots; x_p^N E] + E_{\text{pos}}$$

Because self attention operations are permutation invariant (meaning they do not inherently know which patch came from the top left corner versus the bottom right), 1D or 2D learned positional embeddings $E_{\text{pos}}$ are added to each projected patch vector.

High Resolution Image Decomposition: OpenAI Vision Grid System

While standard ViT models require downsampling an image to a fixed low resolution like $224 \times 224$ or $336 \times 336$, real world user images contain fine text, complex diagrams, and small details that would blur during aggressive downscaling.

To resolve this, models like GPT 4 Vision and GPT 4o employ a multi scale tile decomposition system:

1. Aspect Ratio Preservation: The image is scaled so that its shortest side fits within a bounded bounding box (maximum $2048 \times 2048$ resolution).
2. 512x512 High Resolution Tile Decomposition: The image is divided into a regular grid of $512 \times 512$ pixel tiles. An image measuring $1024 \times 1024$ decomposes into four tiles.
3. Low Resolution Context Overview: In addition to the detailed tiles, a downscaled overview of the entire scene (scaled to $512 \times 512$) is generated.
4. Token Budget Calculation: Each high resolution $512 \times 512$ tile costs a fixed budget of 170 tokens, plus an initial base budget of 85 tokens for the whole image overview:

$$\text{Total Visual Tokens} = 85 + (\text{Number of Tiles} \times 170)$$

A $1024 \times 1024$ image yields 4 tiles, consuming $85 + (4 \times 170) = 765$ tokens in the prompt context.

Audio Tokenization: From Continuous Waveforms to Discrete Codes

Audio presents a distinct challenge because raw sound is recorded as continuous pressure waves sampled 16,000 to 48,000 times per second (16 kHz to 48 kHz). Passing 48,000 numbers per second directly into an attention mechanism would saturate context windows within seconds.

Audio tokenization utilizes two main paradigms: continuous spectrogram embeddings and discrete neural codecs.

Paradigm 1: Mel Spectrogram Projections (Whisper)

In speech recognition systems like OpenAI Whisper, audio waveforms are transformed using the Short Time Fourier Transform (STFT) into log Mel filter bank spectrograms:

• The audio is framed into 25 millisecond windows with a 10 millisecond stride.
• Fast Fourier Transforms decompose each window into frequency intensities.
• Frequencies are mapped onto the Mel scale, which mimics the human ear’s non linear pitch perception (higher sensitivity to low frequencies, lower sensitivity to high frequencies).
• An 80 channel Mel spectrogram is projected through two 1D convolution layers with stride 2, compressing 30 seconds of audio into 1,500 continuous token representations.

Paradigm 2: Residual Vector Quantization (RVQ) Neural Audio Codecs

To allow a language model to synthesize natural sounding speech autoregressively, the continuous audio signal must be compressed into discrete integer codes, similar to text vocabulary IDs.

Traditional Vector Quantization (VQ VAE) maps an encoder vector $z$ to the single closest codebook vector $e_k$:

$$z_q = \arg\min_{e_k \in \mathcal{C}} | z - e_k |_2$$

However, using a single codebook to represent rich audio (voice tone, pitch, acoustic reverb, and speech phonemes) requires codebooks with billions of vectors, which cannot be trained effectively.

Residual Vector Quantization (RVQ), used in EnCodec, SNAC, and SoundStream, solves this through hierarchical quantization stages:

1. Stage 1: The first quantizer finds the closest codebook vector $q_1(z)$ for the latent vector $z$.
2. Residual Calculation: The residual error $r_1 = z - q_1(z)$ is computed.
3. Stage 2: A second independent codebook quantizes the residual $r_1$, finding $q_2(r_1)$.
4. Iterative Compression: This cascade continues across $K$ quantizers (typically 4 to 8 levels).

The final audio representation is the sum of all quantizers:

$$\hat{z} = \sum_{k=1}^K q_k(r_{k-1})$$

A 1 second audio segment can now be represented as a compact grid of integers: 50 frames per second, with 8 codebook indices per frame. High levels capture semantic word pronunciation, while lower levels capture emotional timbre, background ambiance, and speaker voice characteristics.

Unified Multimodal Sequences

By converting vision patches into projected vectors and audio frames into discrete RVQ codes, frontier architectures process multiple modalities in a single interleaved sequence:

[TEXT: "Look at this blueprint and describe the motor:"]
[VISION_START] [TILE_1_VECTORS...] [TILE_2_VECTORS...] [VISION_END]
[AUDIO_START] [RVQ_LEVEL_1_CODES...] [AUDIO_END]

Every modality shares the same causal self attention mechanism, allowing the model to attend from an acoustic pitch anomaly directly back to a visual schematic patch and formulate a textual diagnostic report.

Summary and Practical Takeaway

Multimodal intelligence relies on sophisticated discretization bridges that transform continuous sensory inputs into mathematical token streams. Vision models utilize non overlapping patch projections and multi tile grid calculators, while audio models rely on Mel spectrogram transforms and multi codebook Residual Vector Quantization.

You can experiment with both Vision tile decompositions and Mel spectrogram acoustic tokenizers directly in our interactive multimodal laboratory.