Diagrams — Module FT13: The DPO Family and Preferences

Module: FT13 — The DPO Family and Preferences Diagram count: 5 Tool: Mermaid (primary). Each diagram validated in Mermaid Live Editor.

Diagram 1 — The Trajectory: RLHF → DPO → Variants

Type: Timeline / evolution Purpose: The single diagram that shows how preference alignment evolved, what each stage fixed, and why PPO retired for general alignment (but survived for verifiable rewards). Reading the diagram: Left to right, chronologically. Each arrow is a reduction — fewer moving parts, fewer models in memory. The branch at the end shows where PPO survived.

flowchart LR
  RLHF["RLHF / PPO\n(2022)\nSFT → reward model → PPO\n3 models in memory · RL instability"]
  DPO["DPO\n(2023)\nreparameterizes RLHF\nno reward model · no RL\n1 supervised loss"]
  VARS["DPO Variants\n(2023–2024)\nIPO · KTO · ORPO · SimPO\nfix specific DPO failure modes"]
  SIMPO["Modern defaults\nSimPO · ORPO\nreference-free · length-normalized\nbeats DPO in benchmarks"]

  RLHF -->|"collapses to"| DPO
  DPO -->|"specializes into"| VARS
  VARS -->|"converges on"| SIMPO

  RLHF -.->|"PPO survives here\nfor VERIFIABLE rewards\n(math, code, tools)"| GRPO["GRPO (FT14)\non-policy RL\nverifiable rewards only"]

  style RLHF fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8
  style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style VARS fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style SIMPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style GRPO fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa

Diagram 2 — The DPO Mechanism

Type: Process / data flow Purpose: What DPO actually does to your model. A preference pair goes in; a contrastive loss pushes the policy to prefer chosen over rejected, relative to the reference. Reading the diagram: The {prompt, chosen, rejected} pair feeds into two models — the trainable policy and the frozen reference (your SFT model). The loss compares the policy's relative preference to the reference's. No reward model in the loop.

flowchart TD
  Pair["Preference pair\n{prompt, chosen, rejected}"]
  Pair --> Policy["POLICY π (trainable)\ncompute log π(chosen), log π(rejected)"]
  Pair --> Ref["REFERENCE π_ref (frozen)\nthe SFT model\ncompute log π_ref(chosen), log π_ref(rejected)"]

  Policy --> Contrast["CONTRAST\nlog π(chosen) − log π_ref(chosen)\nminus\nlog π(rejected) − log π_ref(rejected)"]
  Ref --> Contrast

  Contrast --> Sigmoid["× β (temperature)\n→ sigmoid → logistic loss"]
  Sigmoid --> Loss["L_DPO = −log σ(β · contrast)\nstandard supervised loss · backprop · optimizer step"]

  Loss -.->|"pushes π to assign\nmore relative prob to CHOSEN\nthan REJECTED, vs the reference"| Policy

  style Pair fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style Policy fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Ref fill:#14141f,stroke:#f0a868,color:#f0a868
  style Contrast fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8
  style Sigmoid fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0
  style Loss fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4

Diagram 3 — The DPO Variant Decision Tree

Type: Decision tree Purpose: How to pick the right method. The choice is driven by your data shape (pairs vs binary), your reference-model budget, and whether you want to combine with SFT. Reading the diagram: Start at the top. Each question routes you to a method. The leaves are the method and the one-sentence reason.

flowchart TD
  Start["Choosing a preference method"] --> Q1{Data shape?}

  Q1 -->|"unpaired binary\n(thumbs up/down only)"| KTO["KTO\nthe only method built for\nunpaired good/bad feedback"]
  Q1 -->|"paired\n{chosen, rejected}"| Q2{Want a reference model?}

  Q2 -->|"no — save the VRAM"| Q3{Combine SFT + preference\nin one stage?}
  Q2 -->|"yes — grounded on noisy data"| Q4{DPO overfitting?}

  Q3 -->|"yes, skip SFT stage"| ORPO["ORPO\nSFT + preference, one pass\nno reference model"]
  Q3 -->|"no, already SFT'd"| SIMPO["SimPO\nreference-free, length-normalized\nbeats DPO in benchmarks"]

  Q4 -->|"no"| DPO["DPO\nthe baseline default\nstart here"]
  Q4 -->|"yes — train reward rising,\neval degrading"| IPO["IPO\nregularized DPO\nstops over-optimization"]

  style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8
  style KTO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style ORPO fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style SIMPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style IPO fill:#14141f,stroke:#5eead4,color:#e4e4e8

Diagram 4 — The SFT → DPO Pipeline

Type: Linear pipeline Purpose: The canonical alignment pipeline. Two stages: SFT establishes the behavior distribution, DPO refines the preferences. The reference model in DPO is the SFT output. Reading the diagram: Left to right. Stage 1 (SFT) produces the reference. Stage 2 (DPO) trains a copy of it on preference pairs, anchored to the reference. The output is a preference-aligned model.

flowchart LR
  Base["BASE model\nLayer 1 (FT03)"] --> SFT["1. SFT\n(FT12)\ndemonstration data\nformat + instruction-following"]
  SFT --> Ref["SFT model = REFERENCE π_ref\nfrozen for DPO"]
  Ref --> DPO["2. DPO\npreference pairs\n{prompt, chosen, rejected}"]
  Pref["Preference dataset\nhuman · AI-judge · synthetic"] --> DPO
  DPO --> Out["PREFERENCE-ALIGNED model\nmore helpful · better refusals\non-brand tone · fewer failure modes"]

  style Base fill:#14141f,stroke:#9494a0,color:#9494a0
  style SFT fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Ref fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868
  style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4
  style Pref fill:#14141f,stroke:#5eead4,color:#e4e4e8
  style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4

Diagram 5 — DPO Family vs GRPO: Preferences vs Verifiable Rewards

Type: Comparison Purpose: The central judgment of Pillar 3. DPO family for subjective preference (offline, no exploration); GRPO for verifiable rewards (on-policy, exploration). Misplace this line and you pick the wrong tool entirely. Reading the diagram: Two tracks. Left: the DPO family — fixed dataset, no verifier possible, the reward is a human/aesthetic judgment. Right: GRPO — on-policy generation, a verifier scores outputs, the model explores and discovers correct reasoning. The divider is "can a checker compute the reward?"

flowchart TD
  subgraph DPOTrack["DPO FAMILY (FT13) — subjective preference"]
    D1["Data: fixed preference pairs\n{prompt, chosen, rejected}"]
    D2["Reward: human/aesthetic judgment\n'which response is better?'"]
    D3["Mode: OFFLINE\nlearns only from given pairs\nno exploration"]
    D4["Best for: helpfulness, tone,\nrefusals, style, alignment"]
    D1 --> D2 --> D3 --> D4
  end

  subgraph GRPTrack["GRPO (FT14) — verifiable rewards"]
    G1["Data: generated ON-POLICY\nmodel produces N candidates"]
    G2["Reward: a VERIFIER checks\nmath correct? tests pass?"]
    G3["Mode: ONLINE\nexplores, discovers\ncorrect reasoning paths"]
    G4["Best for: math, code,\ntool use, agentic tasks"]
    G1 --> G2 --> G3 --> G4
  end

  Divider{"Can a checker\ncompute the reward?"}
  Divider -->|"NO — it's a judgment"| DPOTrack
  Divider -->|"YES — answer is checkable"| GRPTrack

  style DPOTrack fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8
  style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style D2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4
  style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8
  style GRPTrack fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#e4e4e8
  style G1 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8
  style G2 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8
  style G3 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#82e0aa
  style G4 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8
  style Divider fill:#08080c,stroke:rgba(255,255,255,0.15),color:#f0a868

Validation notes

All five diagrams use the course design system colors: #14141f panel fill, #5eead4 accent for primary, #82e0aa (ok, GRPO) / #f08080 (danger) / #f0a868 (warn, reference model) for semantic emphasis, rgba(255,255,255,0.12) for secondary borders, #e4e4e8 / #9494a0 for text, #08080c for inner panel fills.
Paste each into Mermaid Live Editor to render. All use stable Mermaid syntax (flowchart with TD/LR, subgraph, dashed -.-> links) supported in current Mermaid (v10.4+).
The dashed links in Diagram 1 (RLHF -.-> GRPO) and Diagram 2 (Loss -.-> Policy) denote a side-effect / feedback relationship, not a data-flow dependency.
For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.

# Diagrams — Module FT13: The DPO Family and Preferences **Module**: FT13 — The DPO Family and Preferences **Diagram count**: 5 **Tool**: Mermaid (primary). Each diagram validated in [Mermaid Live Editor](https://mermaid.live). --- ## Diagram 1 — The Trajectory: RLHF → DPO → Variants **Type**: Timeline / evolution **Purpose**: The single diagram that shows how preference alignment evolved, what each stage fixed, and why PPO retired for general alignment (but survived for verifiable rewards). **Reading the diagram**: Left to right, chronologically. Each arrow is a reduction — fewer moving parts, fewer models in memory. The branch at the end shows where PPO survived. ```mermaid flowchart LR RLHF["RLHF / PPO\n(2022)\nSFT → reward model → PPO\n3 models in memory · RL instability"] DPO["DPO\n(2023)\nreparameterizes RLHF\nno reward model · no RL\n1 supervised loss"] VARS["DPO Variants\n(2023–2024)\nIPO · KTO · ORPO · SimPO\nfix specific DPO failure modes"] SIMPO["Modern defaults\nSimPO · ORPO\nreference-free · length-normalized\nbeats DPO in benchmarks"] RLHF -->|"collapses to"| DPO DPO -->|"specializes into"| VARS VARS -->|"converges on"| SIMPO RLHF -.->|"PPO survives here\nfor VERIFIABLE rewards\n(math, code, tools)"| GRPO["GRPO (FT14)\non-policy RL\nverifiable rewards only"] style RLHF fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#e4e4e8 style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style VARS fill:#14141f,stroke:#5eead4,color:#e4e4e8 style SIMPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style GRPO fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#82e0aa ``` --- ## Diagram 2 — The DPO Mechanism **Type**: Process / data flow **Purpose**: What DPO actually does to your model. A preference pair goes in; a contrastive loss pushes the policy to prefer `chosen` over `rejected`, *relative to the reference*. **Reading the diagram**: The `{prompt, chosen, rejected}` pair feeds into two models — the trainable policy and the frozen reference (your SFT model). The loss compares the policy's relative preference to the reference's. No reward model in the loop. ```mermaid flowchart TD Pair["Preference pair\n{prompt, chosen, rejected}"] Pair --> Policy["POLICY π (trainable)\ncompute log π(chosen), log π(rejected)"] Pair --> Ref["REFERENCE π_ref (frozen)\nthe SFT model\ncompute log π_ref(chosen), log π_ref(rejected)"] Policy --> Contrast["CONTRAST\nlog π(chosen) − log π_ref(chosen)\nminus\nlog π(rejected) − log π_ref(rejected)"] Ref --> Contrast Contrast --> Sigmoid["× β (temperature)\n→ sigmoid → logistic loss"] Sigmoid --> Loss["L_DPO = −log σ(β · contrast)\nstandard supervised loss · backprop · optimizer step"] Loss -.->|"pushes π to assign\nmore relative prob to CHOSEN\nthan REJECTED, vs the reference"| Policy style Pair fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style Policy fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style Ref fill:#14141f,stroke:#f0a868,color:#f0a868 style Contrast fill:#08080c,stroke:rgba(94,234,212,0.4),color:#e4e4e8 style Sigmoid fill:#08080c,stroke:rgba(94,234,212,0.3),color:#9494a0 style Loss fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 ``` --- ## Diagram 3 — The DPO Variant Decision Tree **Type**: Decision tree **Purpose**: How to pick the right method. The choice is driven by your data shape (pairs vs binary), your reference-model budget, and whether you want to combine with SFT. **Reading the diagram**: Start at the top. Each question routes you to a method. The leaves are the method and the one-sentence reason. ```mermaid flowchart TD Start["Choosing a preference method"] --> Q1{Data shape?} Q1 -->|"unpaired binary\n(thumbs up/down only)"| KTO["KTO\nthe only method built for\nunpaired good/bad feedback"] Q1 -->|"paired\n{chosen, rejected}"| Q2{Want a reference model?} Q2 -->|"no — save the VRAM"| Q3{Combine SFT + preference\nin one stage?} Q2 -->|"yes — grounded on noisy data"| Q4{DPO overfitting?} Q3 -->|"yes, skip SFT stage"| ORPO["ORPO\nSFT + preference, one pass\nno reference model"] Q3 -->|"no, already SFT'd"| SIMPO["SimPO\nreference-free, length-normalized\nbeats DPO in benchmarks"] Q4 -->|"no"| DPO["DPO\nthe baseline default\nstart here"] Q4 -->|"yes — train reward rising,\neval degrading"| IPO["IPO\nregularized DPO\nstops over-optimization"] style Start fill:#14141f,stroke:rgba(255,255,255,0.12),color:#e4e4e8 style KTO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style ORPO fill:#14141f,stroke:#5eead4,color:#e4e4e8 style SIMPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style IPO fill:#14141f,stroke:#5eead4,color:#e4e4e8 ``` --- ## Diagram 4 — The SFT → DPO Pipeline **Type**: Linear pipeline **Purpose**: The canonical alignment pipeline. Two stages: SFT establishes the behavior distribution, DPO refines the preferences. The reference model in DPO *is* the SFT output. **Reading the diagram**: Left to right. Stage 1 (SFT) produces the reference. Stage 2 (DPO) trains a copy of it on preference pairs, anchored to the reference. The output is a preference-aligned model. ```mermaid flowchart LR Base["BASE model\nLayer 1 (FT03)"] --> SFT["1. SFT\n(FT12)\ndemonstration data\nformat + instruction-following"] SFT --> Ref["SFT model = REFERENCE π_ref\nfrozen for DPO"] Ref --> DPO["2. DPO\npreference pairs\n{prompt, chosen, rejected}"] Pref["Preference dataset\nhuman · AI-judge · synthetic"] --> DPO DPO --> Out["PREFERENCE-ALIGNED model\nmore helpful · better refusals\non-brand tone · fewer failure modes"] style Base fill:#14141f,stroke:#9494a0,color:#9494a0 style SFT fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Ref fill:#14141f,stroke:#f0a868,stroke-width:1.5px,color:#f0a868 style DPO fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 style Pref fill:#14141f,stroke:#5eead4,color:#e4e4e8 style Out fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#5eead4 ``` --- ## Diagram 5 — DPO Family vs GRPO: Preferences vs Verifiable Rewards **Type**: Comparison **Purpose**: The central judgment of Pillar 3. DPO family for subjective preference (offline, no exploration); GRPO for verifiable rewards (on-policy, exploration). Misplace this line and you pick the wrong tool entirely. **Reading the diagram**: Two tracks. Left: the DPO family — fixed dataset, no verifier possible, the reward is a human/aesthetic judgment. Right: GRPO — on-policy generation, a verifier scores outputs, the model explores and discovers correct reasoning. The divider is "can a checker compute the reward?" ```mermaid flowchart TD subgraph DPOTrack["DPO FAMILY (FT13) — subjective preference"] D1["Data: fixed preference pairs\n{prompt, chosen, rejected}"] D2["Reward: human/aesthetic judgment\n'which response is better?'"] D3["Mode: OFFLINE\nlearns only from given pairs\nno exploration"] D4["Best for: helpfulness, tone,\nrefusals, style, alignment"] D1 --> D2 --> D3 --> D4 end subgraph GRPTrack["GRPO (FT14) — verifiable rewards"] G1["Data: generated ON-POLICY\nmodel produces N candidates"] G2["Reward: a VERIFIER checks\nmath correct? tests pass?"] G3["Mode: ONLINE\nexplores, discovers\ncorrect reasoning paths"] G4["Best for: math, code,\ntool use, agentic tasks"] G1 --> G2 --> G3 --> G4 end Divider{"Can a checker\ncompute the reward?"} Divider -->|"NO — it's a judgment"| DPOTrack Divider -->|"YES — answer is checkable"| GRPTrack style DPOTrack fill:#14141f,stroke:#5eead4,stroke-width:1.5px,color:#e4e4e8 style D1 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8 style D2 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8 style D3 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#5eead4 style D4 fill:#08080c,stroke:rgba(94,234,212,0.3),color:#e4e4e8 style GRPTrack fill:#14141f,stroke:#82e0aa,stroke-width:1.5px,color:#e4e4e8 style G1 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8 style G2 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8 style G3 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#82e0aa style G4 fill:#08080c,stroke:rgba(130,224,170,0.3),color:#e4e4e8 style Divider fill:#08080c,stroke:rgba(255,255,255,0.15),color:#f0a868 ``` --- ## Validation notes - All five diagrams use the course design system colors: `#14141f` panel fill, `#5eead4` accent for primary, `#82e0aa` (ok, GRPO) / `#f08080` (danger) / `#f0a868` (warn, reference model) for semantic emphasis, `rgba(255,255,255,0.12)` for secondary borders, `#e4e4e8` / `#9494a0` for text, `#08080c` for inner panel fills. - Paste each into [Mermaid Live Editor](https://mermaid.live) to render. All use stable Mermaid syntax (`flowchart` with `TD`/`LR`, `subgraph`, dashed `-.->` links) supported in current Mermaid (v10.4+). - The dashed links in Diagram 1 (`RLHF -.-> GRPO`) and Diagram 2 (`Loss -.-> Policy`) denote a side-effect / feedback relationship, not a data-flow dependency. - For the slide deck (artifact 03), these are rendered as static captures from Mermaid Live, inlined into reveal.js.