The DPO Family and Preferences

Module FT13 · Course 3 — LLM Fine-Tuning Masterclass

75 minutes · Pillar 3 opens — preference alignment: RLHF → DPO → SimPO/ORPO, and where GRPO takes over.

Prereq: FT12 (SFT: The Baseline). Next: FT14 (GRPO and Verifiable Rewards).

Pillar 3 — Alignment & Preferences

The trajectory — how alignment evolved

RLHF / PPO (2022) · SFT → reward model → PPO · 3 models, RL instability

↓ collapses to

DPO (2023) · reparameterizes RLHF · no reward model · no RL · one supervised loss

↓ specializes into

DPO Variants · IPO (overfitting) · KTO (unpaired) · ORPO (one-stage) · SimPO (ref-free)

↓ converges on

Modern defaults · SimPO & ORPO — reference-free, length-normalized

PPO retired for general alignment — but survives for verifiable rewards (math, code, tools). That is FT14. The line between the two is this module's central judgment.

RLHF/PPO — what DPO replaced

The original alignment pipeline (InstructGPT, ChatGPT). Three stages, three models in memory.

1. SFT · demonstration data → instruct model

2. Reward model · trained on preference pairs to predict human preference

3. PPO · RL loop: generate → reward model scores → policy update + KL penalty

Why it was retired: expensive (3 models in VRAM), unstable (PPO is famously finicky), hard to debug (reward model has its own failure modes — reward hacking, misgeneralization). A hyperparameter mistake produces silent degradation, not a clean error.

The DPO mechanism

Rafailov et al. 2023 (arXiv:2305.18290). Your language model is secretly a reward model.

The insight: the optimal RLHF policy can be reparameterized — the reward function is expressible in terms of the policy itself. So you cut out the reward model and the RL loop.

L_DPO = -log σ( β · [ (log π(chosen)   - log π_ref(chosen))
                    - (log π(rejected) - log π_ref(rejected)) ] )

π = trainable policy · π_ref = frozen reference (your SFT model) · β = drift temperature · σ = sigmoid

No reward model. The model is the reward model — log-probs relative to the reference.

No RL. Standard supervised loss. Backprop, optimizer step — same mechanics as SFT.

β and the reference model

β — the drift temperature

β	Effect
too low (0.01)	weak anchor → drift, over-optimization, reward hacking
0.1–0.5	typical band · 0.1 is the common start
too high (0.5+)	strong anchor → barely moves, weak effect

π_ref — the reference (load-bearing)

The frozen SFT model you are improving
The contrast is always against π_ref
Keeps the policy from drifting arbitrarily
DPO on a base model = garbage — no coherent reference distribution

Rule: DPO needs the SFT starting point. The whole pipeline assumes a coherent response distribution to refine.

The variant decision tree

Six methods. Pick by data shape, reference budget, and whether you combine SFT.

Method	Ref?	Data	Reach for it when
DPO	yes	pairs	The baseline default. Start here.
IPO	yes	pairs	DPO overfits your dataset (eval diverges).
KTO	yes	unpaired	You only have thumbs-up/down, not pairs.
ORPO	no	pairs+SFT	One training stage, no reference. Saves compute.
SimPO	no	pairs	Modern default. Ref-free, length-normalized. Beats DPO.
R-DPO	yes	pairs	Regularized for stability. Niche.

DPO or SimPO ~90% of real work. IPO/KTO are specialists. ORPO when you want to skip the SFT stage.

Building a preference dataset

The format

{ "prompt":   "Explain recursion to a junior dev.",
  "chosen":   "Recursion is when a function calls itself...",
  "rejected": "Recursion is a programming concept. Read a textbook." }

Three sources

Human annotation — expensive, slow, high quality (OpenAssistant, Chatbot Arena)
AI feedback — model-as-judge with a rubric (UltraFeedback, RLAIF). Cheap, scalable.

Synthetic — chosen = good response, rejected = base/degraded output. Controllable. This lab's method.

The signal test. Sample 20 pairs. Can you articulate why chosen is better? If not, your model can't learn from it. DPO fits noise — and degrades.

DPO family vs GRPO — the line

The central judgment of Pillar 3. Misplace it and you pick the wrong tool.

DPO family (FT13)
Subjective preference · "which is better?"
Offline — fixed pairs, no exploration
Helpfulness, tone, refusals, style

GRPO (FT14)
Verifiable reward · math correct? tests pass?
On-policy — generates, explores, discovers
Math, code, tool use, agents

The rule: if the reward can be computed by a verifier — the answer is checkable — use GRPO. If it's a human/aesthetic judgment, use the DPO family. Pure offline methods cannot do on-policy exploration.

The standard pipeline — SFT then DPO

1. Base model · Layer 1 (FT03)

↓

2. SFT · demonstration data → format + instruction-following (FT12)

↓ this becomes the reference π_ref

3. DPO (or SimPO/ORPO) · preference pairs → shift preferences

↓

Preference-aligned model · more helpful · better refusals · on-brand

Anti-patterns: DPO on a base model (not SFT'd — no coherent reference) · β too low (over-optimization) · preference data with no real signal (fits noise) · treating DPO as a knowledge tool (it steers preference, doesn't teach facts).

The lab — SFT then DPO

Take your SFT'd model from FT12 (or a provided one)
Build a 500-pair synthetic preference dataset (chosen = good response, rejected = degraded)
Run DPO via TRL DPOTrainer on top of the SFT model
Measure win-rate improvement on a held-out set (model judge)

The payoff: a measurable shift in the model's preferences — see the win-rate move. That is alignment, felt. Consumer-GPU runnable (RTX 4090 / 24GB or Colab).

Runnable Python in 07-lab-spec.md. ~45–60 min.

What you can now do

Trace the trajectory: RLHF → DPO → SimPO/ORPO, and state why PPO retired for general alignment.
Explain the DPO mechanism — a reparameterized logistic loss on preference pairs, no reward model, no RL.
Apply the variant decision tree — DPO, IPO, KTO, ORPO, SimPO — by data shape and reference budget.
Build a {prompt, chosen, rejected} dataset from human, AI-judge, or synthetic sources.
Run the SFT-then-DPO pipeline and recognize the anti-patterns (base-not-SFT'd, β too low, no-signal data).

Next: FT14 — GRPO and Verifiable Rewards