Module 3.1
Byte-Pair Encoding doesn't know English. It just counts which pair of symbols shows up next to each other most often, and glues that pair together — over and over, on real text, until common substrings become single tokens.
The math
No linguistics anywhere in this rule. It's pure frequency — which is exactly why tokenizers trained on code split differently from ones trained on prose.
The code
from collections import Counter
def get_pair_counts(words): # words: {tuple_of_symbols: freq}
counts = Counter()
for symbols, freq in words.items():
for a, b in zip(symbols, symbols[1:]):
counts[(a, b)] += freq
return counts
def merge_pair(pair, words):
new_words = {}
for symbols, freq in words.items():
merged, i = [], 0
while i < len(symbols):
if symbols[i:i+2] == pair:
merged.append(pair[0] + pair[1]); i += 2
else:
merged.append(symbols[i]); i += 1
new_words[tuple(merged)] = freq
return new_words
for step in range(num_merges):
pair_counts = get_pair_counts(words)
best = max(pair_counts, key=pair_counts.get)
words = merge_pair(best, words)
vocab.append(best[0] + best[1]) # the new tokenThis is the whole algorithm — real tokenizer training just runs it on gigabytes of text instead of 10 words, for tens of thousands of merges instead of 7.
Go deeper (optional)
Another course-length build (2h 13m), not a quick clip — Andrej Karpathy builds a complete, GPT-4-style tokenizer from scratch, including the regex pre-splitting rules and byte-level fallback this toy example skips entirely.
Andrej Karpathy — "Let's build the GPT Tokenizer." Supplementary and optional.
The Nimbus thread
If "Nimbus" appears often enough in a model's training data, BPE merges it into one or two tokens, same as "export" did here. If it's rare — a small company's product name almost certainly is, in a tokenizer trained mostly on the general web — it stays fragmented into odd pieces like Nim + bus. That fragmentation is invisible in a chat UI, but it's exactly why brand names, made-up words, and rare technical terms cost more tokens (and sometimes get subtly misunderstood) than common English words.
So what
A tokenizer's vocabulary is discovered from data, not designed — whatever repeats most in training becomes a token; everything else gets stuck spelling itself out in pieces.