LESSON 2.1 · Code · 100 min
Bigram: the smallest language model
Bigram: the smallest language model. Learn counts, conditional distributions, sampling, and smoothing and complete: Train and sample a name generator. Part of the “Language Becomes
Learning objectives
- Explain what problem “Bigram: the smallest language model” solves without hiding behind terminology.
- Trace the variables and causal links across counts, conditional distributions, sampling.
- Complete “Train and sample a name generator” and judge the result with evidence rather than intuition.
Core concepts
counts
counts is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
conditional distributions
conditional distributions is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
sampling
sampling is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and smoothing
and smoothing is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Train and sample a name generator
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.