LESSON 3.3 · Code · 100 min
Multi-head attention: read different relations in parallel
Multi-head attention: read different relations in parallel. Learn head splitting, concatenation, projection, and capacity and complete: Implement and visualize multi-head attention
Learning objectives
- Explain what problem “Multi-head attention: read different relations in parallel” solves without hiding behind terminology.
- Trace the variables and causal links across head splitting, concatenation, projection.
- Complete “Implement and visualize multi-head attention” and judge the result with evidence rather than intuition.
Core concepts
head splitting
head splitting is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
concatenation
concatenation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
projection
projection is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and capacity
and capacity is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Implement and visualize multi-head attention
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.