LESSON 5.3 · Theory · 120 min
Preference data and reward models
Preference data and reward models. Learn pairwise preferences, Bradley–Terry, and reward hacking and complete: Train a tiny reward model. Part of the “Make the Model an Assistant”
Learning objectives
- Explain what problem “Preference data and reward models” solves without hiding behind terminology.
- Trace the variables and causal links across pairwise preferences, Bradley–Terry, and reward hacking.
- Complete “Train a tiny reward model” and judge the result with evidence rather than intuition.
Core concepts
pairwise preferences
pairwise preferences is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.
Bradley–Terry
Bradley–Terry is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
and reward hacking
and reward hacking is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.
Build and verify
Train a tiny reward model
- Predict: write the expected output, trend, or failure before running code.
- Build: implement only the minimum components needed to answer the question.
- Verify: compare with a baseline or trusted implementation; save seeds, parameters, and raw outputs.
- Transfer: change one shape, dataset, scale, or workload condition and explain whether the conclusion still holds.