LESSON 2.6 · Derivation · 150 min
Manual backprop: become a Backprop Ninja
Manual backprop: become a Backprop Ninja. Learn tensor-level backprop through cross-entropy, tanh, and BatchNorm and complete: Backpropagate through the whole network without calli
Learning objectives
- Explain what problem “Manual backprop: become a Backprop Ninja” solves without hiding behind terminology.
- Trace the variables and causal links across tensor-level backprop through cross-entropy, tanh, and BatchNorm.
- Complete “Backpropagate through the whole network without calling backward” and judge the result with evidence rather than intuition.
Core concepts
tensor-level backprop through cross-entropy
tensor-level backprop through cross-entropy is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
tanh
tanh is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and BatchNorm
and BatchNorm is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Backpropagate through the whole network without calling backward
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.