BPE: from bytes to subwords

DIRECT ANSWER · VERIFIED SOURCES · 2026-07-22

Why can BPE represent arbitrary text with a finite vocabulary?

Byte-level BPE begins with UTF-8 bytes and repeatedly merges the most frequent adjacent token pair in its training corpus. Encoding replays the learned merge rules. Because the base vocabulary covers all 256 byte values, an unseen character, language, or code fragment can always fall back to bytes instead of becoming an unknown token.

Key takeaways

• The merge count sets vocabulary size: larger vocabularies often shorten sequences but add parameters and rare tokens.
• BPE learns compression rules, not semantic word boundaries; one token does not necessarily equal one word.
• Encode and decode must round-trip exactly, while special tokens need separate collision and permission rules.

Boundary and caveat

Results also depend on Unicode normalization, pre-tokenization, regex rules, and merge ordering. Compare tokenizers using vocabulary, corpus, compression ratio, and special-token policy together.

Primary sources

• Sennrich et al. · Neural Machine Translation of Rare Words with Subword Units
• Andrej Karpathy · minbpe
• OpenAI · tiktoken

UTF-8

UTF-8 is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.

pair merges

pair merges is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

vocabulary

vocabulary is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

encode

encode is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

and decode

and decode is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.