LESSON 3.2 · Derivation · 120 min
Scaled dot-product attention
Q, K, and V are learned linear projections of the same token representations. A Query describes what the current position is trying to retrieve, a Key describes how each position c
DIRECT ANSWER · VERIFIED SOURCES ·
What do Q, K, V, and scaled dot-product attention actually compute?
Q, K, and V are learned linear projections of the same token representations. A Query describes what the current position is trying to retrieve, a Key describes how each position can be matched, and a Value carries the information to aggregate. Attention computes QKᵀ/√dₖ, adds a mask, applies softmax, and uses the resulting weights to form a weighted sum of V.
Key takeaways
- QKᵀ performs content addressing, softmax turns each row into weights summing to 1, and multiplication by V aggregates information.
- Division by √dₖ controls dot-product variance so larger head dimensions do not prematurely saturate softmax and shrink gradients.
- A causal mask blocks future tokens by assigning invalid positions negative infinity before softmax.
Boundary and caveat
“What to retrieve” and “what to provide” are useful mental models, not fixed semantics built into Q, K, or V. Their meaning emerges from training, and attention weights alone are not a causal explanation.
Primary sources
Learning objectives
- Write scaled dot-product attention with every tensor shape.
- Derive the √d_k divisor from the growth of dot-product variance.
- Explain why a causal mask is applied to logits before softmax.
- Prove row normalization, zero future weights, and invariance of past outputs with executable assertions.
Core concepts
The jobs of Q, K, and V
One token can be projected into distinct roles. Q describes what a position requests, K describes how each position can be matched, and V carries the content retrieved after matching. Address and content are separate.
Shape ledger
Let Q,K∈Rⁿˣᵈᵏ and V∈Rⁿˣᵈᵛ. QKᵀ produces n×n logits; row-wise softmax remains n×n; multiplying by V produces n×d_v outputs. Write shapes before operations.
Why divide by √d_k
If q_i and k_i are independent with variance near one, the variance of Σq_ik_i grows approximately with d_k. Dividing by √d_k returns the standard deviation to a constant scale and avoids premature softmax saturation.
Where the mask belongs
For causal attention, logit j>i becomes −∞ before softmax. Its exponential is exactly zero while valid positions renormalize to one. Clearing weights after softmax breaks normalization and invites leakage bugs.
Stable softmax and value mixing
Subtract each row maximum before exponentiating to reduce overflow risk, then assert the row sums to one. The output is a weighted sum of visible values, not the queries, keys, or weights themselves.
Build and verify
Implement and verify four-token causal single-head attention
- Write the Q, K, V, logits, weights, and output shapes for four 2-D tokens.
- Implement attention_row: scale dot products, set future logits to −∞, then use stable softmax.
- Assert every row sums to one, future weights are zero, and output dimensions are correct.
- Change only token four and prove outputs one through three are unchanged; compare softmax entropy with and without scaling.