LESSON 2.8 · Lab · 90 min
The hidden cost of tokenization
The hidden cost of tokenization. Learn numbers, whitespace, multilingual text, special tokens, and compression and complete: Compare tokenization pathologies across Chinese, Englis
Learning objectives
- Explain what problem “The hidden cost of tokenization” solves without hiding behind terminology.
- Trace the variables and causal links across numbers, whitespace, multilingual text.
- Complete “Compare tokenization pathologies across Chinese, English, and Japanese” and judge the result with evidence rather than intuition.
Core concepts
numbers
numbers is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
whitespace
whitespace is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
multilingual text
multilingual text is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
special tokens
special tokens is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and compression
and compression is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Compare tokenization pathologies across Chinese, English, and Japanese
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.