DPO: preference optimization without a reward model

DIRECT ANSWER · VERIFIED SOURCES · 2026-07-22

How does DPO learn preferences without a separate reward model and online RL loop?

DPO takes chosen/rejected responses to the same prompt, compares their log-probability gap under the current policy, subtracts the corresponding gap under a fixed reference policy, and applies a β-scaled logistic loss that increases the chosen response’s relative advantage. This rewrites KL-constrained preference optimization as a classification-style objective on the language model, so the standard procedure needs neither a separately fitted reward model nor online rollouts.

Lecture alignment

Lecture alignment: track four log-probabilities—the policy and reference scores for both chosen and rejected sequences—and how they form a margin, get scaled by β, and enter log-sigmoid.

Key takeaways

• The reference policy supplies the baseline for limiting drift and is normally frozen while the current policy updates.
• β controls preference strength relative to the reference; it is not an ordinary learning rate and depends on data and implementation conventions.
• The loss scores conditional response-token log-probabilities; prompt tokens are normally excluded from chosen/rejected scoring.

Boundary and caveat

DPO simplifies the classic RLHF pipeline but does not remove preference-data bias, label noise, length effects, or out-of-distribution degradation. Pairwise relative preference optimization does not automatically guarantee factuality, safety, or agreement across groups.

Primary sources

• Rafailov et al. · Direct Preference Optimization
• Hugging Face TRL · DPO Trainer
• Stanford CS336 · Language Modeling from Scratch

reference policies

reference policies is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment before moving to an optimized implementation.

KL control

KL control is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.

and the DPO objective

and the DPO objective is part of the lesson’s causal model. State its inputs, outputs, invariants, and failure mode; then verify it with a hand-check or a minimal experiment.