L_DPO = -log σ( β · [ (log π(chosen) - log π_ref(chosen))
                    - (log π(rejected) - log π_ref(rejected)) ] )

{
  "prompt": "Explain recursion to a junior developer.",
  "chosen": "Recursion is when a function calls itself to solve a smaller version of the same problem. Think of it like Russian nesting dolls...",
  "rejected": "Recursion is a programming concept. It is used in computer science. You can read more about it in a textbook."
}

# Module FT13 — The DPO Family and Preferences

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT13 — The DPO Family and Preferences
**Duration**: 75 minutes
**Level**: Senior Engineer and above
**Prerequisites**: FT12 (SFT: The Baseline)

---

## Learning Objectives

After completing this module, you will be able to:

1. Trace the trajectory of preference alignment — RLHF/PPO → DPO → DPO variants (IPO, KTO) → reference-free methods (SimPO, ORPO) — and state *why* PPO largely retired for general alignment while surviving for verifiable rewards (FT14).
2. Explain the DPO mechanism (Rafailov et al., 2023): a reparameterization of the RLHF objective into a logistic classification loss on preference pairs, with no reward model and no RL, optimized against a reference policy (your SFT'd model).
3. Apply the DPO variant decision tree: when to use DPO, IPO (overfitting), KTO (unpaired binary feedback), ORPO (one-step SFT+preference), SimPO (reference-free, length-normalized modern default), and R-DPO.
4. Build a preference dataset `{prompt, chosen, rejected}` from human annotation, AI feedback (model-as-judge), or synthetic construction, and recognize the anti-pattern of preference data with no real signal.
5. Run the standard SFT-then-DPO pipeline, and explain why DPO on top of a base (non-SFT'd) model is a cardinal error.

---

# 13.1 — The Trajectory: From RLHF to the DPO Family

*Where preference alignment came from, and why almost everyone reading this will never run PPO.*

## The starting point: RLHF/PPO

The technique that aligned the first generation of instruct models — InstructGPT, the original ChatGPT — was **RLHF: Reinforcement Learning from Human Feedback**. The pipeline had three stages:

1. **SFT** a base model on demonstration data (Module FT12).
2. **Train a reward model** on human preference pairs — `{prompt, chosen, rejected}` — to predict which response a human would prefer.
3. **Run PPO** (Proximal Policy Optimization): the language model generates responses, the reward model scores them, and the policy is updated with RL to maximize reward, with a KL-divergence penalty keeping it close to the SFT model.

RLHF works. It is also expensive, unstable, and hard to debug. Three models live in memory simultaneously (the policy being trained, a frozen reference copy for the KL penalty, and the reward model). The reward model is a separate trained artifact with its own failure modes (reward hacking, misgeneralization). PPO is famously finicky — a hyperparameter mistake produces silent degradation, not a clean error. For years this was the only game in town. It no longer is.

## The pivot: DPO (2023)

In May 2023, Rafailov et al. published *Direct Preference Optimization* (arXiv:2305.18290). The insight was mathematical: the optimal RLHF policy can be *reparameterized* — the reward function can be expressed directly in terms of the policy itself. This eliminates the reward model. It eliminates the RL loop. What remains is a simple classification loss on preference pairs, computed directly on the language model. You can train it with the same optimizer stack as SFT.

DPO was a watershed. It collapsed a three-model, multi-stage RL pipeline into a single supervised loss you could run on one GPU. Almost overnight, "RLHF" in most teams came to mean "DPO or one of its variants." The original PPO pipeline is now a specialty tool, not a default.

> **The current state of play.** For *general* preference alignment — making the model helpful, harmless, honest, on-style, refusing appropriately — the DPO family is the default. PPO-style RL has largely retired from this role. It survives, and dominates, where rewards are *verifiable* — math correctness, code passing tests, tool calls succeeding. That is the subject of Module FT14 (GRPO). The line between the two is the key judgment of this module.

## The variant explosion

DPO spawned a family. Each variant fixes a specific failure mode of the original:

- **IPO** (Interior Point Optimization, Azar et al. 2023): adds regularization to stop DPO overfitting on the preference data.
- **KTO** (Kahneman-Tversky Optimization, Ethayarajh et al. 2024): drops the paired structure entirely and works on unpaired binary feedback — thumbs up / thumbs down.
- **ORPO** (Hong et al. 2024): combines SFT and preference into a single training stage with no reference model — saving a full training pass.
- **SimPO** (Meng et al. 2024, arXiv:2405.14734): reference-free and length-normalized; consistently beats DPO in benchmarks and removes the reference-model memory overhead.
- **R-DPO** and others: further regularizations for stability and generalization; niche.

You do not need to learn all of them deeply. You need the decision tree that picks the right one for your data and your constraints. That is section 13.4.

---

# 13.2 — The DPO Mechanism

*What DPO actually does to your model. The single idea that makes the family tractable.*

## The reparameterization

The core insight, stated plainly: in the RLHF framework, the optimal policy and the reward function are mathematically linked. Given an RLHF-trained policy, you can recover the reward it was optimizing. DPO inverts this — instead of training a reward model and then a policy, it derives a loss function that directly optimizes the policy to satisfy the same preference objective, with no reward model in the loop.

The result is a **contrastive, logistic-style loss on preference pairs**. For each pair `{prompt, chosen, rejected}`, DPO computes the model's log-probability for both responses, subtracts what the *reference* model would have assigned, and applies a logistic function: it pushes the policy to assign relatively more probability to `chosen` than `rejected`, compared to the reference.

The loss (simplified):

```
L_DPO = -log σ( β · [ (log π(chosen) - log π_ref(chosen))
                    - (log π(rejected) - log π_ref(rejected)) ] )
```

Where `π` is the policy being trained, `π_ref` is the frozen reference, `σ` is the sigmoid, and `β` is a temperature controlling how far you can drift from the reference.

Three things to notice:

1. **No reward model.** The model *is* the reward model — its own log-probabilities, relative to the reference, serve as the implicit reward.
2. **No RL.** This is a standard supervised loss. Backprop, optimizer step, done. Same training mechanics as SFT.
3. **The reference model is load-bearing.** The contrast is always against `π_ref`. This is what keeps DPO from drifting arbitrarily — the reference anchors it. DPO needs *something* to be the reference.

## What the reference model is — and why it must be your SFT model

The reference model `π_ref` is, by construction and by intent, **the SFT model you are improving.** DPO is not a base-model technique. The whole pipeline assumes:

- You already have an SFT'd model that can follow instructions and produce reasonable responses.
- You want to *shift* its preferences — make it prefer helpful over evasive, cite sources, decline harmful requests politely, avoid a specific failure mode.

The reference is the SFT model frozen at the start of DPO training. The policy is a copy of it, trainable. Every gradient step pushes the policy's preferences away from the reference in the direction your data points. The KL-style anchor (via the `log π_ref` terms and `β`) keeps the drift bounded.

This is why DPO-on-a-base-model is a cardinal anti-pattern (section 13.6). A base model has not been taught to produce coherent responses in the format you want; "preference" over its outputs is meaningless because the baseline distribution is wrong. DPO needs the SFT starting point. SFT-then-DPO is the standard pipeline.

## β — the temperature that controls drift

`β` (beta) is DPO's most important hyperparameter. It controls how strongly the loss anchors the policy to the reference:

- **Low β** (e.g., 0.01): weak anchor. The policy can drift far from the reference. Faster alignment to your data — and faster over-optimization, reward hacking, and capability degradation. The model enthusiastically matches your preference data, including its flaws.
- **High β** (e.g., 0.5): strong anchor. The policy barely moves. Safe but weak — you may not shift behavior enough to matter.

Typical values: `0.1` to `0.5`, with `0.1` as a common starting point. The right β is empirical — too low and you over-optimize; too high and nothing changes. Raschka's SFT-vs-DPO work and Phil Schmid's 2025 DPO guide both treat β tuning as the central practical question.

---

# 13.3 — Building a Preference Dataset

*The dataset is the steering wheel. DPO on bad data steers you into a wall.*

## The format

Every DPO-family method (except KTO) consumes pairs:

```json
{
  "prompt": "Explain recursion to a junior developer.",
  "chosen": "Recursion is when a function calls itself to solve a smaller version of the same problem. Think of it like Russian nesting dolls...",
  "rejected": "Recursion is a programming concept. It is used in computer science. You can read more about it in a textbook."
}
```

The `prompt` is shared. The `chosen` and `rejected` are two responses to it — one better, one worse, by your definition of better. DPO trains the model to assign relatively more probability to the *kind* of response in `chosen`.

## Three sources for pairs

1. **Human annotation.** Present a prompt and two candidate responses to a human; ask which is better. Expensive, slow, high quality. The original RLHF datasets (OpenAssistant, Chatbot Arena conversations) were built this way.

2. **AI feedback (model-as-judge).** Use a strong model (GPT-4-class, or a capable open model like Llama 3.1 70B / Qwen 2.5 72B) to label which response is better, given a rubric. Cheap, fast, scalable — and the basis of most modern preference datasets. Constitutional AI, RLAIF, and the UltraFeedback dataset all use this. Quality depends on the judge model and rubric; weak judges produce noisy labels that cap your quality ceiling.

3. **Synthetic construction.** Build pairs without a judge at all. The most common pattern: `chosen` = the model's good response (from your SFT data), `rejected` = a degraded version — the base model's lower-quality output, a truncated response, or a response with a specific failure injected (e.g., refusal where helpfulness was wanted, or the wrong format). This is what the lab in this module does. It is cheap, controllable, and excellent for targeting a specific behavior you want to fix.

## The signal test

The single most important property of a preference dataset: **the pairs must contain a real, learnable signal.** If `chosen` and `rejected` are nearly identical, or if the "preference" is random noise, DPO will overfit the noise and degrade the model. This is the same lesson as every other module: the algorithm is only as good as the steering wheel.

Before training, sample 20 pairs and ask: *could a reasonable person tell which is better, and why?* If you can't articulate the difference, your model can't learn from it.

---

# 13.4 — The DPO Variant Decision Tree

*Six methods, one decision. Pick the one that matches your data and constraints.*

## The tree

| Method | Needs reference? | Data shape | When to reach for it |
| --- | --- | --- | --- |
| **DPO** | Yes | Paired `{chosen, rejected}` | The default baseline. Start here. Standard preference pairs. |
| **IPO** | Yes | Paired | When DPO overfits your dataset (eval wins diverge from train wins). Adds regularization. |
| **KTO** | Yes | **Unpaired** binary (good/bad per example) | When you only have thumbs-up/thumbs-down, not pairs. |
| **ORPO** | **No** | Paired, combined with SFT | When you want one training stage (SFT + preference) and no reference model. Saves compute. |
| **SimPO** | **No** | Paired | A strong modern default. Reference-free, length-normalized. Beats DPO in benchmarks. |
| **R-DPO** | Yes | Paired | Regularized DPO for stability/generalization. Niche; reach for it when you see instability. |

## The decision logic

1. **Do you have preference *pairs*, or only binary good/bad labels?**
   - Pairs → DPO / IPO / SimPO / ORPO / R-DPO.
   - Binary only → **KTO**. This is the one method built for unpaired feedback. If your data is a stream of 👍/👎 with no pairing, KTO is the answer.

2. **Do you want a reference model?**
   - Reference-free methods (**SimPO**, **ORPO**) cut memory and compute roughly in half — no frozen second copy of the model in VRAM. For consumer-GPU work this matters.
   - Reference-based methods (DPO, IPO, KTO, R-DPO) are more theoretically grounded and behave well on noisy data, at the cost of the reference overhead.

3. **Do you already have an SFT'd model, or are you starting from base?**
   - If you have an SFT model and want to add preferences → DPO / SimPO / IPO.
   - If you want to skip the separate SFT stage → **ORPO**. It does SFT and preference in one pass. One stage, one model, no reference. The tradeoff: less control over the SFT quality, since it's entangled with the preference loss.

4. **Is DPO overfitting?** (Train reward rising, eval reward falling, model getting weirdly sycophantic or degenerate.)
   - Switch to **IPO** (adds a regularizer that caps how much the implicit reward can grow).
   - Or raise β in DPO.
   - Or improve the data (the usual answer — overfitting on bad data is a data problem).

5. **Looking for a strong modern default with no reference overhead?**
   - **SimPO.** Published in 2024 (arXiv:2405.14734), it removes the reference model and adds length normalization (so the model can't win by just producing shorter/longer responses). In the benchmarks the SimPO authors ran, it consistently beat DPO. If you're starting a new preference pipeline today and have no specific reason to pick DPO, SimPO is a defensible first choice.

> **The honest summary.** DPO is the baseline you understand first because it makes the mechanism clear. SimPO and ORPO are what many production teams reach for now because they remove the reference overhead. IPO and KTO are the specialists for overfitting and unpaired data respectively. R-DPO is for when you hit instability. You will use DPO or SimPO 90% of the time.

---

# 13.5 — DPO vs GRPO: The Line Between Preferences and Verifiable Rewards

*The most important judgment in Pillar 3. Get this wrong and you pick the wrong tool entirely.*

The DPO family optimizes **offline preferences** — fixed pairs from a dataset. It cannot explore. It cannot try a response, see it fail, and try a different one. It learns only from the pairs you gave it.

RL methods — specifically **GRPO** (Group Relative Policy Optimization, the modern default, Module FT14) — optimize **on-policy with verifiable rewards.** The model generates multiple candidate responses to a prompt; a verifier (a math checker, a test runner, a tool-call validator) scores them; the policy is updated to favor the winners.

| Dimension | DPO family | GRPO (FT14) |
| --- | --- | --- |
| Reward type | Subjective preference (helpful, on-style) | Verifiable (math correct, tests pass) |
| Data | Offline, fixed pairs | Generated on-policy during training |
| Exploration | None — can only learn from given pairs | Yes — generates and tries |
| Memory | One or two models (policy + optional ref) | Policy + ref + generation rollout buffer |
| Stability | High (supervised loss) | Lower (RL dynamics, reward variance) |
| Best for | General alignment, style, tone, refusal behavior | Math, code, tool use, agentic tasks |

The rule:

> **If the reward can be computed by a verifier — the answer is checkable — use GRPO (FT14). If the reward is a human/aesthetic judgment — "this response is better" — use the DPO family.**

Pure offline methods (DPO family) cannot do on-policy exploration. This is their fundamental limitation. For a math problem where you can check the answer, exploring is strictly better — the model discovers correct reasoning paths it didn't know it could produce. For "be more concise and less sycophantic," there's no verifier; you need preferences, and DPO is the tool.

This line — subjective preference vs verifiable reward — is the central architectural decision of alignment. Everything in FT13 (DPO family) and FT14 (GRPO) follows from it.

---

# 13.6 — The Standard Pipeline and Its Anti-Patterns

## SFT, then DPO

The canonical pipeline:

1. **Start with a base model** (Layer 1, FT03).
2. **SFT** it on demonstration data to teach format and instruction-following (FT12). This is your reference model.
3. **DPO** (or SimPO / ORPO) on top, using a preference dataset, to shift the model's preferences — more helpful, better refusals, on-brand tone, fewer specific failure modes.

This is the pipeline almost every aligned instruct model you've used went through. SFT establishes the behavior distribution; DPO refines it.

## The anti-patterns

### DPO on a base model (not SFT'd)

DPO assumes a coherent reference distribution. A base model's outputs are not coherent in the instruct sense — they're continuations, not responses. "Preference" over them is meaningless. DPO on a base model produces garbage. **Always SFT first.** (ORPO is the exception that combines them, and even it bakes in an SFT loss term.)

### Over-optimization (β too low)

A β that's too low lets the policy drift far from the reference. The model enthusiastically matches your preference data — including its biases, its noise, its quirks. You see this as rising train reward but degrading eval, increasing sycophancy, or the model breaking on out-of-distribution inputs. The fix: raise β, switch to IPO, or (usually) fix the data.

### Preference data with no real signal

If your `chosen` and `rejected` are nearly indistinguishable, or the labels are random, DPO fits noise. The model degrades. This is the "steering without a steering wheel" anti-pattern from FT00 — the most expensive version. The data quality bar for DPO is *higher* than for SFT, because the signal is subtler (a preference, not a target).

### Treating DPO as a knowledge tool

DPO steers preference. It does not teach facts. If your model doesn't know something, DPO will not fix it — it'll just make the model more confidently wrong, or steer it to refuse more eagerly. Knowledge is RAG or continued pretraining; preference is DPO. (The FT00 thesis, restated for this layer.)

---

## Anti-Patterns (recap)

| Anti-pattern | Why it fails | Fix |
| --- | --- | --- |
| DPO on a base (non-SFT) model | No coherent reference distribution | SFT first, always |
| β too low (over-optimization) | Policy drifts, fits noise/bias | Raise β, switch to IPO, fix data |
| Preference data with no signal | Fits noise, model degrades | Audit 20 pairs; ensure learnable difference |
| DPO as knowledge injection | Steers preference, doesn't teach facts | Use RAG for knowledge; DPO for behavior |
| Wrong method for the data shape | KTO data in DPO, or vice versa | Match method to data (13.4 tree) |

---

## Key Terms

| Term | Definition |
| --- | --- |
| **RLHF / PPO** | The original alignment pipeline: SFT → reward model → PPO. Three models, RL instability. Largely retired for general alignment. |
| **DPO** | Direct Preference Optimization (Rafailov et al. 2023). Reparameterizes RLHF as a logistic classification loss on preference pairs. No reward model, no RL. |
| **Reference model (π_ref)** | The frozen SFT model that DPO contrasts against. Anchors the policy to prevent drift. Load-bearing. |
| **β (beta)** | DPO temperature controlling drift from the reference. Low = aggressive, high = conservative. Typically 0.1–0.5. |
| **IPO** | Interior Point Optimization. Regularized DPO to prevent overfitting the preference data. |
| **KTO** | Kahneman-Tversky Optimization. Works on unpaired binary (thumbs up/down) feedback, not pairs. |
| **ORPO** | Combines SFT and preference into one stage, no reference model. Saves a full training pass. |
| **SimPO** | Reference-free, length-normalized DPO variant. A strong modern default; beats DPO in benchmarks (arXiv:2405.14734). |
| **Preference pair** | `{prompt, chosen, rejected}` — the unit of data for most DPO-family methods. |
| **GRPO** | Group Relative Policy Optimization (FT14). On-policy RL with verifiable rewards. For math/code/tool use. |
| **Verifiable reward** | A reward computable by a checker (math correct, tests pass). The domain of RL/GRPO, not DPO. |
| **On-policy** | Generating fresh samples from the current model during training. DPO is offline; GRPO is on-policy. |

---

## Lab Exercise

See `07-lab-spec.md`. The "SFT then DPO" lab: take your SFT'd model from FT12 (or a provided one), build a 500-pair synthetic preference dataset, run DPO via TRL's `DPOTrainer`, and measure the win-rate improvement on a held-out set using a model judge. Consumer-GPU runnable.

---

## References

1. **Rafailov et al. (2023)** — *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*. arXiv:2305.18290, NeurIPS 2023. The DPO paper. The reparameterization that collapsed RLHF into a supervised loss.
2. **Meng et al. (2024)** — *SimPO: Simple Preference Optimization with a Reference-Free Reward*. arXiv:2405.14734. Reference-free, length-normalized; consistently beats DPO.
3. **Azar et al. (2023)** — *A General Theoretical Paradigm to Understand Learning from Human Feedback* (IPO). arXiv:2310.12036. Regularized DPO for overfitting.
4. **Ethayarajh et al. (2024)** — *KTO: Model Alignment as Prospect Theoretic Optimization* (Kahneman-Tversky). arXiv:2402.01306. Unpaired binary feedback.
5. **Hong et al. (2024)** — *ORPO: Monolithic Preference Optimization without Reference Model*. arXiv:2403.07691. One-stage SFT + preference.
6. **Raschka (2024)** — *Preference Fine-Tuning via DPO: An Illustrated Guide* (SFT-vs-DPO analysis). Practical β tuning and the SFT-then-DPO pipeline.
7. **Phil Schmid (2025)** — *DPO Guide*. The modern practitioner's reference for running DPO with TRL.
8. **Course 3, Module FT14** — *GRPO and Verifiable Rewards*. The RL counterpart to this module; where DPO ends and on-policy RL begins.