LESSON 3.1 · Intuition · 100 min
Context aggregation: from averaging to attention
Context aggregation: from averaging to attention. Learn weighted aggregation and the motivation for queries, keys, and values and complete: Replace a bigram model’s context mechani
Learning objectives
- Explain what problem “Context aggregation: from averaging to attention” solves without hiding behind terminology.
- Trace the variables and causal links across weighted aggregation and the motivation for queries, keys, and values.
- Complete “Replace a bigram model’s context mechanism step by step” and judge the result with evidence rather than intuition.
Core concepts
weighted aggregation and the motivation for queries
weighted aggregation and the motivation for queries is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
keys
keys is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and values
and values is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Replace a bigram model’s context mechanism step by step
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.