Implement a Transformer block line by line

DIRECT ANSWER · VERIFIED SOURCES · 2026-07-22

How does a modern GPT Transformer block update the residual stream?

A common pre-norm GPT block applies x = x + Attention(LayerNorm(x)), followed by x = x + MLP(LayerNorm(x)). Causal self-attention lets each position read only itself and earlier tokens, while the MLP transforms the channels at each position independently. Both branches add their updates back to the residual stream, helping information and gradients travel through many layers.

Lecture alignment

Lecture alignment: trace LayerNorm → causal multi-head attention → residual add → LayerNorm → GELU/MLP → residual add, checking the code and tensor shapes at every step.

Key takeaways

• Attention mixes information across positions; the MLP mixes feature channels within each position.
• Pre-norm places LayerNorm before each sublayer; the original Transformer paper used a different post-norm order.
• A residual add requires matching shapes, so concatenated heads must be projected back to the model dimension.

Boundary and caveat

A “Transformer block” is not one immutable recipe. Models vary normalization order, activation, positional encoding, attention type, and biases; follow the target architecture’s configuration. Dropout is normally active only during training.

Primary sources

• Raschka · LLMs from Scratch, Chapter 4
• Vaswani et al. · Attention Is All You Need
• PyTorch · TransformerEncoderLayer

attention

attention is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.

MLP

MLP is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

dropout

dropout is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

and LayerNorm composition

and LayerNorm composition is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.