GPT: autoregressive objectives and weight tying

DIRECT ANSWER · VERIFIED SOURCES · 2026-07-22

Why do both GPT training and generation reduce to next-token prediction?

During training, GPT processes a token sequence in parallel under a causal mask and uses the same sequence shifted by one position as labels, computing next-token cross-entropy at every position. During generation there are no future labels, so the model takes logits at the final position, selects or samples one token, appends it to the context, and repeats. Both estimate the same conditional distribution but execute it differently.

Lecture alignment

Lecture alignment: in Karpathy's build-GPT walkthrough, trace the one-token x/y shift, triangular mask, logits reshape, cross-entropy, and token-by-token generate loop.

Key takeaways

• The causal mask prevents position t from seeing later labels and leaking training information.
• Training supervises many sequence positions in parallel; generation feeds each new token back autoregressively.
• Temperature and top-k change how a token is selected from logits, not the next-token learning objective.

Boundary and caveat

Training receives ground-truth prefixes while generation increasingly consumes its own outputs, so their input distributions are not identical. Tying input and output weights is common but not required, and context length remains bounded by the model configuration.

Primary sources

• Andrej Karpathy · Let's build GPT
• Andrej Karpathy · nanoGPT
• Radford et al. · GPT-2 Technical Report

causal language modeling

causal language modeling is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.

embedding tying

embedding tying is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

logits

logits is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

and generation

and generation is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.