LESSON 1.7 · Diagnosis · 100 min
Why neural networks fail to train
Why neural networks fail to train. Learn dead ReLUs, vanishing/exploding gradients, initialization, and activation statistics and complete: Diagnose four broken networks. Part of t
Learning objectives
- Explain what problem “Why neural networks fail to train” solves without hiding behind terminology.
- Trace the variables and causal links across dead ReLUs, vanishing/exploding gradients, initialization.
- Complete “Diagnose four broken networks” and judge the result with evidence rather than intuition.
Core concepts
dead ReLUs
dead ReLUs is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
vanishing/exploding gradients
vanishing/exploding gradients is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
initialization
initialization is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and activation statistics
and activation statistics is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Diagnose four broken networks
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.