LESSON 1.3 · Code · 120 min
Make gradients flow backward through a graph
Backpropagation starts from a scalar loss and evaluates vector–Jacobian products backward through the computation graph. A node must receive every downstream contribution before it
DIRECT ANSWER · VERIFIED SOURCES ·
Why does backpropagation run in reverse topological order and accumulate gradients with +=?
Backpropagation starts from a scalar loss and evaluates vector–Jacobian products backward through the computation graph. A node must receive every downstream contribution before it can pass its complete upstream gradient to its parents, so nodes run in reverse topological order. When one value affects the loss through several paths, the total derivative is the sum of those path contributions; gradients must accumulate rather than overwrite one another.
Key takeaways
- The forward pass records values and dependencies; the backward pass applies one local derivative rule per operation.
- The output gradient starts at 1 because the loss derivative with respect to itself is 1.
- Shared parameters, branches, and repeated use create multiple gradient paths that must be summed.
Boundary and caveat
Topological order fixes dependency timing, not an incorrect derivative rule. Verify operators with central differences and a trusted autodiff system; both very large and very small finite-difference steps can mislead.
Primary sources
Learning objectives
- Explain why reverse mode propagates from a scalar output in reverse topological order.
- Implement local backward rules for add, multiply, and tanh.
- Explain why gradients at shared nodes must accumulate.
- Validate the engine with finite differences and a shared-node regression test.
Core concepts
Dynamic DAG
Every operation creates a Value and records its parents plus a local backward closure. The graph follows the actual execution path, including branches and repeated use.
Local backward rules
Add copies the upstream gradient to both inputs; multiply scales by the other forward value; tanh scales by 1-t². Each rule owns one local edge.
Topological dependency
A node can propagate only after all downstream paths have contributed. Build parent-first topology with DFS, then execute _backward in reverse.
Gradient accumulation
A node may influence the loss through several paths. The chain rule sums those contributions, so assignment silently discards earlier paths.
Scalar seed
For final scalar y, dy/dy=1 seeds the pass. A non-scalar output requires an explicit upstream vector for a vector–Jacobian product.
Build and verify
Implement a tested Value class, operators, and backward
- Implement Value(data, parents, op), with add, multiply, and tanh returning new nodes.
- Write local _backward closures and update every parent with +=.
- Build topology with DFS, seed the output with grad=1, and execute in reverse.
- Compare y=x*x+x against hand math, centered differences, and a scalar PyTorch result.