LESSON 2.5 · Diagnosis · 120 min
Training deep nets: activations, gradients, BatchNorm
Training deep nets: activations, gradients, BatchNorm. Learn initialization scale, saturation, and BatchNorm train/eval behavior and complete: Build an activation and gradient dash
Learning objectives
- Explain what problem “Training deep nets: activations, gradients, BatchNorm” solves without hiding behind terminology.
- Trace the variables and causal links across initialization scale, saturation, and BatchNorm train/eval behavior.
- Complete “Build an activation and gradient dashboard” and judge the result with evidence rather than intuition.
Core concepts
initialization scale
initialization scale is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
saturation
saturation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and BatchNorm train/eval behavior
and BatchNorm train/eval behavior is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Build an activation and gradient dashboard
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.