LESSON 4.7 · Systems · 170 min
Distributed training: DP, TP, PP, FSDP
Distributed training: DP, TP, PP, FSDP. Learn communication topology, all-reduce, sharding, and pipeline bubbles and complete: Choose a parallel strategy for different clusters. Pa
Learning objectives
- Explain what problem “Distributed training: DP, TP, PP, FSDP” solves without hiding behind terminology.
- Trace the variables and causal links across communication topology, all-reduce, sharding.
- Complete “Choose a parallel strategy for different clusters” and judge the result with evidence rather than intuition.
Core concepts
communication topology
communication topology is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
all-reduce
all-reduce is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
sharding
sharding is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and pipeline bubbles
and pipeline bubbles is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Choose a parallel strategy for different clusters
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.