LESSON 4.3 · Lab · 120 min
Every control in the training loop
Every control in the training loop. Learn AdamW, warmup, cosine decay, clipping, and checkpoints and complete: Ablate optimizers and learning rates. Part of the “Turn Training into
Learning objectives
- Explain what problem “Every control in the training loop” solves without hiding behind terminology.
- Trace the variables and causal links across AdamW, warmup, cosine decay.
- Complete “Ablate optimizers and learning rates” and judge the result with evidence rather than intuition.
Core concepts
AdamW
AdamW is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
warmup
warmup is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
cosine decay
cosine decay is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
clipping
clipping is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and checkpoints
and checkpoints is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Ablate optimizers and learning rates
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.