LESSON 3.9 · Engineering · 180 min
Reproduce 124M: from correct to fast
Reproduce 124M: from correct to fast. Learn mixed precision, compile, FlashAttention, and gradient accumulation and complete: Optimize throughput step by step with Karpathy. Part o
Learning objectives
- Explain what problem “Reproduce 124M: from correct to fast” solves without hiding behind terminology.
- Trace the variables and causal links across mixed precision, compile, FlashAttention.
- Complete “Optimize throughput step by step with Karpathy” and judge the result with evidence rather than intuition.
Core concepts
mixed precision
mixed precision is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
compile
compile is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
FlashAttention
FlashAttention is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and gradient accumulation
and gradient accumulation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Optimize throughput step by step with Karpathy
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.