LESSON 3.4 · Theory · 110 min
Position, residual streams, and normalization
Position, residual streams, and normalization. Learn positional embeddings, pre-norm, and residual streams and complete: Compare pre-norm and post-norm gradients. Part of the “From
Learning objectives
- Explain what problem “Position, residual streams, and normalization” solves without hiding behind terminology.
- Trace the variables and causal links across positional embeddings, pre-norm, and residual streams.
- Complete “Compare pre-norm and post-norm gradients” and judge the result with evidence rather than intuition.
Core concepts
positional embeddings
positional embeddings is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
pre-norm
pre-norm is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and residual streams
and residual streams is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Compare pre-norm and post-norm gradients
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.