LESSON 8.1 · Code Study · 150 min
MoE, MLA, and multi-token prediction
MoE, MLA, and multi-token prediction. Learn Mixture-of-Experts, routing, load balancing, MLA, and multi-token prediction and complete: Dissect DeepSeek-V3 activation, KV compressio
Learning objectives
- Explain what problem “MoE, MLA, and multi-token prediction” solves without hiding behind terminology.
- Trace the variables and causal links across Mixture-of-Experts, routing, load balancing.
- Complete “Dissect DeepSeek-V3 activation, KV compression, and objectives” and judge the result with evidence rather than intuition.
Core concepts
Mixture-of-Experts
Mixture-of-Experts is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
routing
routing is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
load balancing
load balancing is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
MLA
MLA is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and multi-token prediction
and multi-token prediction is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Dissect DeepSeek-V3 activation, KV compression, and objectives
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.