Interactive 3D LLM Engine

3D LLM Visualization

Interactive visual LLM and 3D transformer visualization. Explore how GPT Transformer models process tokens, calculate self-attention, project query/key/value vectors, and compute next-token probability distributions step-by-step.

Mathematical Deep Dive

3D Transformer Self-Attention Breakdown

A step-by-step linear algebra derivation of how Large Language Models transform discrete token IDs into contextualized vector representations via scaled dot-product attention and multi-head projections.

Tensor Dimension Cheat Sheet

Standard GPT Architecture

Sequence Length

Number of Tokens

Model Dimension

d_model

e.g., 768 or 4096

Attention Heads

e.g., 12 or 32 heads

Head Dimension

d_k = d_v

d_model / h

Attention Matrix

n × n

Token affinities

FFN Hidden Dim

d_ff ≈ 4d_model

MLP Expansion

Token Embedding & Positional Encoding

Input Tensor: X ∈ ℝ^{n × d_model}

When text enters a Transformer, each token string is mapped to an integer token ID from a vocabulary matrix V of size |V| (e.g., 50,257 for GPT-2 or 100,277 for GPT-4o). An embedding lookup matrix W_E ∈ ℝ^{|V| × d_model} translates each discrete token ID into a continuous dense embedding vector E ∈ ℝ^{n × d_model}.

Because self-attention operations are inherently permutation-invariant (order-agnostic), Transformers inject sequential position information via a positional encoding tensor P ∈ ℝ^{n × d_model}. In traditional architectures, this is computed via sinusoidal harmonic frequencies or learned absolute tables:

Sinusoidal Evens: PE_{(pos, 2i)} = sin(pos / 10000^{2i / d_model})
Sinusoidal Odds: PE_{(pos, 2i+1)} = cos(pos / 10000^{2i / d_model})
Modern LLMs: RoPE (Rotary Position Embeddings) apply a 2D rotation matrix directly to Q and K vectors.

Formula Step 1

X = E + P

E (Embeddings): [n × d_model]

P (Positional): [n × d_model]

X (Input Tensor): [n × d_model]

Query, Key, and Value Projections (QKV)

Projections: Q, K ∈ ℝ^{n × d_k}, V ∈ ℝ^{n × d_v}

At each attention layer, the input tensor X is linearly projected into three distinct semantic vector spaces using learned weight matrices:

Queries (Q)

Represents what the current token is seeking or asking about surrounding context.

Keys (K)

Represents what characteristics or semantic content this token offers to others.

Values (V)

The actual feature information extracted and passed forward if matched.

The projections are computed via matrix multiplication: Q = X · W_Q, K = X · W_K, and V = X · W_V, where W_Q, W_K ∈ ℝ^{d_model × d_k} and W_V ∈ ℝ^{d_model × d_v}.

Projection Equations Step 2

Q = X · W_Q → [n × d_k]

K = X · W_K → [n × d_k]

V = X · W_V → [n × d_v]

In standard multi-head setups: d_k = d_v = d_model / h

Scaled Dot-Product Attention Mechanism

Core Equation: Attention(Q, K, V)

The mathematical heart of the Transformer is the Scaled Dot-Product Attention operator. It determines how much attention token i pays to token j across the context window:

1. Raw Affinity Scores (Q · K^T):

Multiplying the Query matrix by the transposed Key matrix yields an n × n matrix where each cell S_ij = q_i · k_j measures directional alignment in embedding space.

2. Variance Normalization (√d_k):

Under the assumption that components of Q and K are independent random variables with mean 0 and variance 1, their dot product has variance d_k. For large d_k (e.g. 64 or 128), dot products grow extremely large in magnitude, pushing the Softmax function into regions with near-zero gradients. Dividing by √d_k stabilizes variance back to 1.0.

3. Softmax Probability & Value Weighting:

Applying row-wise Softmax transforms raw logits into a strictly normalized probability distribution summing to 1.0 per row. Finally, multiplying by Value matrix V produces the attention-weighted context vector representation.

Canonical Equation Attention Engine

Attention(Q, K, V) = softmax(Q · K^T / √d_k) · V

Q · K^T Shape: [n × d_k] × [d_k × n] = [n × n]

Softmax(A) Shape: [n × n] (Normalized weights)

Final Output Tensor: [n × n] × [n × d_v] = [n × d_v]

Multi-Head Attention & Output Projection

h Heads → W_O ∈ ℝ^{(h · d_v) × d_model}

Rather than calculating a single attention distribution, Multi-Head Attention linearly projects the queries, keys, and values h times with different learned parameter matrices.

This enables the neural network to jointly attend to information from different representation subspaces at different positions. For instance, one head might specialize in tracking syntactic verb-object agreements, while another head attends to long-range coreference resolution (pronoun antecedent matching).

Each head produces an output tensor of shape n × d_v. The h heads are concatenated along the channel dimension into a unified tensor of size n × (h · d_v) = n × d_model, then multiplied by output weight matrix W_O:

Multi-Head Formula Step 4

head_i = Attention(Q · W_Q,i, K · W_K,i, V · W_V,i)

MultiHead(Q, K, V) = Concat(head₁, ..., head_h) · W_O

Each Head Tensor: [n × (d_model / h)]

Concatenated: [n × d_model]

After Output Projection W_O: [n × d_model]

Feed-Forward Network (FFN) & Residual LayerNorm

Residual: LayerNorm(x + Sublayer(x))

Following multi-head self-attention, each token vector passes identically and independently through a Position-wise Feed-Forward Network (FFN). The FFN consists of two linear transformations with a non-linear activation function (such as GELU or SwiGLU in contemporary LLMs):

FFN(x) = GELU(x · W₁ + b₁) · W₂ + b₂

Here, the hidden inner dimension expands to d_ff ≈ 4 × d_model before projecting back to d_model. This expansion provides the parameter capacity where factual knowledge and associative memories are stored.

Every sublayer is wrapped in a residual skip connection followed by Layer Normalization (Pre-LN in modern decoders or Post-LN in original Transformers), which prevents vanishing or exploding gradients across deep stacks:

x_norm = LayerNorm(x) = γ · ((x - μ) / √(σ² + ε)) + β

Layer Pipeline Step 5

1. Self-Attention Output:

x₁ = LayerNorm(x + MultiHead(x))

2. MLP Expansion & Contraction:

x₂ = LayerNorm(x₁ + FFN(x₁))

The resulting tensor x₂ ∈ ℝ^{n × d_model} feeds into subsequent transformer layers or output vocabulary unembedding (Softmax over logits).

Key Architectural Takeaway

Self-attention performs pairwise token communication across the sequence length n, while the Feed-Forward Network operates independently per position. Together, repeated across dozens of stacked transformer layers, these mathematical operations give Large Language Models the ability to grasp syntax, reason over multi-step instructions, and generate coherent text.