Understanding Byte Pair Encoding in Modern Large Language Models

Introduction to Subword Tokenization

Large language models do not read human text as words, sentences, or characters. Instead, every sentence you send to a model is converted into a sequential array of numbers called token identifiers. These tokens are looked up in a matrix of vectors called an embedding table, providing continuous numerical vectors for mathematical operations in transformer layers.

The method used to divide continuous text into discrete pieces is known as tokenization. Early natural language systems relied either on word level splitting or character level processing. Both approaches suffered from severe engineering limitations:

• Word level vocabularies: Storing every distinct word creates enormous vocabularies with millions of entries. Furthermore, any word not seen during training becomes an unknown token (often represented as <unk>), completely blinding the model to rare names, typos, or specialized technical terms.
• Character level vocabularies: Processing text letter by letter keeps the vocabulary tiny (a few hundred symbols). However, text sequences become four to six times longer, causing attention mechanisms with quadratic complexity to consume unsustainable amounts of compute and memory.

Byte Pair Encoding (abbreviated as BPE) solves this fundamental dilemma. It begins with individual bytes or characters, iteratively identifies the most frequent adjacent pairs across a massive training corpus, and merges them into new subword units.

The Algorithmic Mechanics of BPE

Byte Pair Encoding was originally invented in 1994 by Philip Gage as a general data compression technique. Modern natural language processing adapted this algorithm into a deterministic, greedy vocabulary builder.

Phase 1: Corpus Initialization

To build a BPE tokenizer, we begin with a training corpus of text. We pre tokenize the text into word boundaries (often using regular expressions) and split every word into individual character symbols followed by a special end of word marker (such as </w>).

Consider a small sample corpus containing four words with their occurrence frequencies:

• low: 5 occurrences
• lower: 2 occurrences
• newest: 6 occurrences
• widest: 3 occurrences

We represent each word as a sequence of space separated characters:

l o w </w> : 5
l o w e r </w> : 2
n e w e s t </w> : 6
w i d e s t </w> : 3

Our initial base vocabulary consists only of the unique symbols: d, e, i, l, n, o, r, s, t, w, and </w>.

Phase 2: Counting Adjacent Pairs

Next, the algorithm scans the entire corpus and counts how frequently every adjacent pair of symbols appears across all words:

• Pair (e, s): appears 6 times in newest and 3 times in widest = 9 total counts.
• Pair (s, t): appears 6 times in newest and 3 times in widest = 9 total counts.
• Pair (t, </w>): appears 6 times in newest and 3 times in widest = 9 total counts.
• Pair (l, o): appears 5 times in low and 2 times in lower = 7 total counts.
• Pair (o, w): appears 5 times in low and 2 times in lower = 7 total counts.

Phase 3: Merging the Most Frequent Pair

The algorithm selects the pair with the highest frequency. In our sample, (e, s) appears 9 times. We merge this pair into a single new symbol es and record the merge rule e + s -> es in our merge table:

l o w </w> : 5
l o w e r </w> : 2
n es w e s t </w> -> n es w es t </w> : 6
w i d es t </w> : 3

Now es becomes a permanent unit in our vocabulary. In the next iteration, the pair (es, t) appears 9 times (6 from newest and 3 from widest). We merge it into est and append the rule es + t -> est to our merge table.

This iterative loop continues until the vocabulary reaches a predetermined target size (such as 32,000 for Llama 1, 100,256 for GPT 4 cl100k, or 200,000 for GPT 4o o200k).

Byte Level Fallbacks and UTF 8 Encoding

Traditional character BPE had one major weakness: if a user entered an unusual Unicode character, emoji, or mathematical symbol not seen in the training data, the tokenizer had to emit an unknown token.

Radford and the OpenAI team solved this in GPT 2 by introducing byte level Byte Pair Encoding. Instead of initializing the vocabulary with Unicode characters, the base alphabet starts with all 256 individual bytes of the UTF 8 encoding specification.

Because any valid or invalid text, emoji, code snippet, or binary string can be represented as a sequence of bytes, byte level BPE guarantees that:

1. The unknown token <unk> is never required. Every input can be tokenized.
2. Common English words compress into single tokens of 4 to 8 characters.
3. Complex Unicode characters (such as multi byte Japanese kanji, Arabic script, or multi code point emojis) simply decompose into their constituent UTF 8 byte values if no merged token exists yet.

Vocabulary Size Tradeoffs in Frontier Models

The size of a model vocabulary represents an engineering tradeoff between computational efficiency, sequence length, and model memory:

When the vocabulary is larger, each token represents a longer fragment of text. For instance, increasing the vocabulary from 100,000 to 200,000 tokens allows GPT 4o to compress non English languages (such as Hindi, Arabic, and Japanese) into roughly half as many tokens as GPT 4.

However, a larger vocabulary expands the size of the final linear layer (the language model head) where the hidden dimension is projected across all vocabulary items to calculate next token logits. A vocabulary of 200,000 items with a hidden dimension of 4,096 requires over 800 million parameters just for the embedding and prediction matrices.

Why Tokenization Causes Arithmetic and Coding Failures

Understanding token boundaries explains many surprising behaviors observed in large language models:

• Arithmetic mistakes: Numbers are often broken into irregular chunks depending on whitespace and digits. For example, 12345 might tokenize as 123 and 45, whereas 123456 might tokenize as 12 and 3456. Because the model never sees individual arithmetic columns aligned predictably, calculating multi digit sums requires high dimensional attention routing rather than straightforward column math.
• Spelling and anagrams: When a model is asked how many times the letter r appears in strawberry, it does not see the letters s, t, r, a, w, b, e, r, r, y. It sees two integers representing straw and berry. To count letters, the model must retrieve spelling knowledge stored in its feed forward weights.
• Code indentation: In languages like Python where leading indentation conveys program structure, tokenizers employ specialized rules to represent groups of 2, 4, and 8 spaces as single tokens, preventing excessive sequence bloat.

Summary and Practical Takeaway

Byte Pair Encoding remains the foundational bridge between unstructured human language and the tensor operations of modern artificial intelligence. By iteratively building a subword vocabulary from byte level statistics, BPE balances compression ratio against vocabulary memory.

You can inspect exactly how GPT 4o, Llama 3, DeepSeek, and other leading models break down arbitrary text, code, and symbols using our interactive tool suite.