Lab Specification — Module FT13: The DPO Family and Preferences

# Create a clean venv
python3.11 -m venv ft13-env && source ft13-env/bin/activate

# The full TRL stack. TRL 1.0+ pulls in transformers, accelerate, peft, datasets.
pip install -q "trl>=1.0.0" transformers accelerate peft datasets bitsandbytes torch
# Logging backend — pick ONE:
pip install -q wandb        # Weights & Biases
#   OR
pip install -q trackio      # HuggingFace's lighter, Spaces-backed tracker

import torch, trl, transformers, peft
print(f"PyTorch: {torch.__version__}")
print(f"TRL: {trl.__version__}")              # must be >= 1.0.0
print(f"Transformers: {transformers.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
    print(f"BF16 supported: {torch.cuda.is_bf16_supported()}")

# The SFT'd model — this becomes BOTH the starting policy AND the frozen reference.
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # or your FT12 output, or Qwen2.5-3B-Instruct
OUTPUT_DIR = "./dpo-out"

import torch
bf16_ok = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
print(f"Reference/policy model: {MODEL_ID}")
print(f"BF16 supported: {bf16_ok}")

from datasets import Dataset
import random, json

# A set of prompts spanning instruction-following, explanation, and coding.
PROMPTS = [
    "Explain what a hash function is.",
    "Write a Python function to reverse a string.",
    "What is the difference between TCP and UDP?",
    "Summarize the plot of Hamlet in two sentences.",
    "How do I center a div in CSS?",
    "Explain recursion with a simple example.",
    "What are the ACID properties in databases?",
    "Write a SQL query to find duplicate rows.",
    "What is the time complexity of binary search?",
    "Explain what a REST API is.",
    # ... (the full script includes 50+ diverse prompts; cycle to reach 500 pairs)
]

# For the lab, we use a provided preference dataset that already encodes the
# concise-vs-verbose preference. This is the fastest path and guarantees signal.
# In production you would generate chosen/rejected with your own model + a judge.

from datasets import load_dataset

# Option A: provided dataset with a clear preference signal (concise vs verbose)
# trl-lib/preference-dataset is a small illustrative DPO-format dataset.
# For a real run, substitute your own {prompt, chosen, rejected} data.
try:
    pref = load_dataset("trl-lib/preference-dataset", split="train")
except Exception:
    # Fallback: build a minimal synthetic dataset in-code (see solution key)
    pref = None

if pref is None or len(pref) < 200:
    # Synthetic construction fallback — chosen = direct, rejected = verbose/hedging
    # This is the pattern from the teaching doc: encode the preference in construction.
    data = []
    for p in PROMPTS * 20:   # cycle to ~500+ rows
        data.append({
            "prompt": p,
            "chosen": f"Here is a direct, concise answer to '{p[:40]}...': [helpful content].",
            "rejected": f"Well, that's an interesting question about '{p[:40]}...'. Let me think about this. There are many ways to approach this. [verbose, hedging content].",
        })
    pref = Dataset.from_list(data[:500])

# Hold out 10% for eval — NEVER DPO on everything (flying blind).
split = pref.train_test_split(test_size=0.1, seed=42)
train_pref, eval_pref = split["train"], split["test"]
print(f"Train pairs: {len(train_pref)}  Eval pairs: {len(eval_pref)}")
print("\nSample pair:")
print(json.dumps(train_pref[0], indent=2)[:500])

from trl import DPOConfig
from peft import LoraConfig

# QLoRA config for the policy (FT08). The reference stays frozen and full-precision-ish.
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)

# DPO config — the levers. Note beta (the drift temperature).
training_args = DPOConfig(
    output_dir=OUTPUT_DIR,
    # --- The model and reference ---
    model=MODEL_ID,                  # the SFT'd model — policy starts here
    ref_model=MODEL_ID,             # the SAME model, frozen as the reference
    # --- Batch + accumulation ---
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,  # effective batch = 16
    # --- LR: LoRA band, slightly lower than SFT (DPO is sensitive) ---
    learning_rate=5e-5,             # DPO typically uses 1e-5 to 5e-5 (lower than SFT LoRA)
    num_train_epochs=1,
    # --- β — THE DPO HYPERPARAMETER. Controls drift from the reference. ---
    beta=0.1,                       # common starting point (0.1–0.5)
    # --- Warmup + scheduler ---
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    # --- Precision ---
    bf16=bf16_ok,
    fp16=not bf16_ok,
    # --- Memory + speed ---
    gradient_checkpointing=True,
    # --- Logging + eval ---
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    # --- Experiment tracking ---
    report_to="wandb",              # or "trackio" / "tensorboard"
    # --- DPO specifics ---
    max_length=1024,
    max_prompt_length=512,
)

from trl import DPOTrainer

# Log in to your tracking backend once in your shell:
#   wandb:  export WANDB_API_KEY=...   (or run `wandb login`)

trainer = DPOTrainer(
    args=training_args,
    train_dataset=train_pref,
    eval_dataset=eval_pref,
    peft_config=peft_config,
)

# Count trainable params (the thesis, quantified)
trainable = sum(p.numel() for p in trainer.model.parameters() if p.requires_grad)
total = sum(p.numel() for p in trainer.model.parameters())
print(f"Trainable params: {trainable:,} / {total:,} = {100*trainable/total:.3f}%")

# THE DPO LOOP
trainer.train()

# Save the DPO'd adapter
trainer.save_model(f"{OUTPUT_DIR}/best-dpo-adapter")
print(f"Saved DPO adapter to {OUTPUT_DIR}/best-dpo-adapter")

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Reload the SFT model fresh (this is the base for the merge)
base = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto"
)
# Load the DPO adapter on top
model = PeftModel.from_pretrained(base, f"{OUTPUT_DIR}/best-dpo-adapter")
# MERGE
merged = model.merge_and_unload()
print("Merged. DPO adapter folded into the SFT model.")

MERGED_DIR = f"{OUTPUT_DIR}/merged-dpo-model"
merged.save_pretrained(MERGED_DIR)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.save_pretrained(MERGED_DIR)
print(f"Saved standalone merged DPO model to {MERGED_DIR}")

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def generate(model, tokenizer, prompt, max_new_tokens=120):
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Held-out eval prompts (NOT from the training set)
EVAL_PROMPTS = [
    "Explain what an API gateway does.",
    "Write a function to check if a string is a palindrome.",
    "What is the difference between supervised and unsupervised learning?",
    "How does HTTPS encryption work?",
    "Explain the concept of Big O notation.",
]

# Generate from BOTH models on the same prompts
sft_tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
# (sft_model already loaded as `base`; merged is the DPO'd model)

print("Generating responses from both models...")
sft_responses = [generate(base, sft_tokenizer, p) for p in EVAL_PROMPTS]
dpo_responses = [generate(merged, sft_tokenizer, p) for p in EVAL_PROMPTS]

# --- Win-rate via model-as-judge ---
# Use a capable model to judge which response is better.
# For the lab, use a simple heuristic judge (length + directness) or a real model.
# The solution key shows a proper model-judge using the same model or an API.

def simple_judge(sft_resp, dpo_resp):
    """A simple proxy judge for the lab: prefer shorter, non-hedging responses.
    In production, replace with a real model-as-judge (GPT-4-class or large open model)."""
    sft_score = -len(sft_resp) - (10 if sft_resp.lower().startswith(("well", "that's", "let me")) else 0)
    dpo_score = -len(dpo_resp) - (10 if dpo_resp.lower().startswith(("well", "that's", "let me")) else 0)
    return "dpo" if dpo_score > sft_score else ("sft" if sft_score > dpo_score else "tie")

wins = {"dpo": 0, "sft": 0, "tie": 0}
for i, p in enumerate(EVAL_PROMPTS):
    verdict = simple_judge(sft_responses[i], dpo_responses[i])
    wins[verdict] += 1
    print(f"\nPrompt: {p}")
    print(f"  SFT:  {sft_responses[i][:100]}...")
    print(f"  DPO:  {dpo_responses[i][:100]}...")
    print(f"  Judge: {verdict}")

total = sum(wins.values())
print(f"\n=== WIN RATE ===")
print(f"DPO wins: {wins['dpo']}/{total} ({100*wins['dpo']/total:.0f}%)")
print(f"SFT wins: {wins['sft']}/{total} ({100*wins['sft']/total:.0f}%)")
print(f"Ties:     {wins['tie']}/{total} ({100*wins['tie']/total:.0f}%)")

# Lab Specification — Module FT13: The DPO Family and Preferences

**Course**: Course 3 — LLM Fine-Tuning Masterclass
**Module**: FT13 — The DPO Family and Preferences
**Lab**: "SFT then DPO"
**Duration**: 45–60 minutes (the first lab of Pillar 3 — you run a preference-alignment job on top of an SFT'd model)
**Environment**: Python 3.11+. A consumer NVIDIA GPU (RTX 4090 / 24GB recommended; RTX 3090 24GB works) OR free Google Colab T4/A100. ~25GB free disk for the SFT model + reference + DPO checkpoints.

> This is the hands-on lab for Pillar 3's preference-alignment module. You will take an SFT'd model (yours from FT12, or a provided one), build a 500-pair synthetic preference dataset, run DPO via TRL's `DPOTrainer` on top of it, and measure the win-rate improvement on a held-out set using a model judge. By the end you will see the model's preferences shift measurably — alignment, felt.

---

## Learning objectives

By the end of this lab you will have:

1. **Built a preference dataset** in the `{prompt, chosen, rejected}` format using synthetic construction (chosen = good response, rejected = degraded/base-model response).
2. **Run DPO** via TRL's `DPOTrainer` on top of an SFT'd model, with a frozen reference, and observed the reward/reward-margin curves.
3. **Measured win-rate improvement** on a held-out set using a model-as-judge, before and after DPO.
4. **Diagnosed over-optimization** (or its absence) by comparing train reward to eval win rate.
5. **Seen the steering effect** — the model's preferences shifted because of the data you gave it, with no new knowledge injected.

---

## Phase 0 — Environment setup (5 min)

```bash
# Create a clean venv
python3.11 -m venv ft13-env && source ft13-env/bin/activate

# The full TRL stack. TRL 1.0+ pulls in transformers, accelerate, peft, datasets.
pip install -q "trl>=1.0.0" transformers accelerate peft datasets bitsandbytes torch
# Logging backend — pick ONE:
pip install -q wandb        # Weights & Biases
#   OR
pip install -q trackio      # HuggingFace's lighter, Spaces-backed tracker
```

Verify the stack and your GPU:

```python
import torch, trl, transformers, peft
print(f"PyTorch: {torch.__version__}")
print(f"TRL: {trl.__version__}")              # must be >= 1.0.0
print(f"Transformers: {transformers.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
if torch.cuda.is_available():
    print(f"GPU: {torch.cuda.get_device_name(0)}")
    print(f"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB")
    print(f"BF16 supported: {torch.cuda.is_bf16_supported()}")
```

You need CUDA (Colab T4 or a consumer NVIDIA GPU). DPO keeps two models in memory (policy + reference), so this lab uses QLoRA for the policy to fit a 3–4B model on 24GB. Apple Silicon MPS is not well-supported for DPO training — use Colab if you lack an NVIDIA GPU.

---

## Phase 1 — Get your SFT'd model (the reference) (5 min)

DPO requires an SFT'd model as the reference (the FT13 thesis: DPO is not a base-model technique). You have two options:

**Option A — Use your FT12 output (preferred).** If you completed the FT12 lab and produced an SFT'd model (a saved adapter or merged checkpoint), use it here. Point `MODEL_ID` at your checkpoint directory or Hub repo. This is the preferred path — you steer a model you built.

**Option B — Use a provided SFT'd model.** If you skipped FT12, use a capable open instruct model that has already been SFT'd. Recommended: `Qwen/Qwen2.5-1.5B-Instruct` (small, fits easily on a T4/24GB with DPO+QLoRA) or `Qwen/Qwen2.5-3B-Instruct` (better quality, needs 24GB).

```python
# The SFT'd model — this becomes BOTH the starting policy AND the frozen reference.
MODEL_ID = "Qwen/Qwen2.5-1.5B-Instruct"   # or your FT12 output, or Qwen2.5-3B-Instruct
OUTPUT_DIR = "./dpo-out"

import torch
bf16_ok = torch.cuda.is_available() and torch.cuda.is_bf16_supported()
print(f"Reference/policy model: {MODEL_ID}")
print(f"BF16 supported: {bf16_ok}")
```

**Record**: the model ID you chose. If you used your FT12 output, note that — you are preference-aligning a model you SFT'd yourself.

> **Teaching moment:** This IS the SFT-then-DPO pipeline from the teaching doc. Your SFT model is the reference `π_ref`. DPO will train a copy of it (the policy) on preference pairs, anchored to this reference. If you tried to run DPO on a base (non-SFT) model, the loss would decrease but the output would be incoherent — the cardinal anti-pattern.

---

## Phase 2 — Build the preference dataset (10 min)

You will build 500 synthetic preference pairs. The construction pattern from the teaching doc: `chosen` = a good response, `rejected` = a degraded version. This lab targets **verbosity and evasiveness** — we want the model to prefer concise, direct, helpful responses over long-winded, hedging, or refusal-like ones.

We'll generate both halves synthetically using the model itself (or a provided dataset if you prefer speed). The key: the difference between `chosen` and `rejected` must be a clear, articulable preference.

```python
from datasets import Dataset
import random, json

# A set of prompts spanning instruction-following, explanation, and coding.
PROMPTS = [
    "Explain what a hash function is.",
    "Write a Python function to reverse a string.",
    "What is the difference between TCP and UDP?",
    "Summarize the plot of Hamlet in two sentences.",
    "How do I center a div in CSS?",
    "Explain recursion with a simple example.",
    "What are the ACID properties in databases?",
    "Write a SQL query to find duplicate rows.",
    "What is the time complexity of binary search?",
    "Explain what a REST API is.",
    # ... (the full script includes 50+ diverse prompts; cycle to reach 500 pairs)
]

# For the lab, we use a provided preference dataset that already encodes the
# concise-vs-verbose preference. This is the fastest path and guarantees signal.
# In production you would generate chosen/rejected with your own model + a judge.

from datasets import load_dataset

# Option A: provided dataset with a clear preference signal (concise vs verbose)
# trl-lib/preference-dataset is a small illustrative DPO-format dataset.
# For a real run, substitute your own {prompt, chosen, rejected} data.
try:
    pref = load_dataset("trl-lib/preference-dataset", split="train")
except Exception:
    # Fallback: build a minimal synthetic dataset in-code (see solution key)
    pref = None

if pref is None or len(pref) < 200:
    # Synthetic construction fallback — chosen = direct, rejected = verbose/hedging
    # This is the pattern from the teaching doc: encode the preference in construction.
    data = []
    for p in PROMPTS * 20:   # cycle to ~500+ rows
        data.append({
            "prompt": p,
            "chosen": f"Here is a direct, concise answer to '{p[:40]}...': [helpful content].",
            "rejected": f"Well, that's an interesting question about '{p[:40]}...'. Let me think about this. There are many ways to approach this. [verbose, hedging content].",
        })
    pref = Dataset.from_list(data[:500])

# Hold out 10% for eval — NEVER DPO on everything (flying blind).
split = pref.train_test_split(test_size=0.1, seed=42)
train_pref, eval_pref = split["train"], split["test"]
print(f"Train pairs: {len(train_pref)}  Eval pairs: {len(eval_pref)}")
print("\nSample pair:")
print(json.dumps(train_pref[0], indent=2)[:500])
```

**Record**: train/eval pair counts, and one sample pair. Confirm the `prompt`, `chosen`, `rejected` fields exist.

> **Signal test (do this before training):** Read 5 random pairs. Can you articulate *why* `chosen` is better than `rejected`? If the difference is clear (e.g., chosen is concise and helpful, rejected is verbose and evasive), you have signal. If they look identical, your data has no signal and DPO will fit noise — fix the construction before proceeding.

---

## Phase 3 — Configure DPO (5 min)

DPO keeps two models in memory: the trainable policy and the frozen reference. To fit on 24GB, we use QLoRA for the policy. The `DPOConfig` holds the levers.

```python
from trl import DPOConfig
from peft import LoraConfig

# QLoRA config for the policy (FT08). The reference stays frozen and full-precision-ish.
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    task_type="CAUSAL_LM",
)

# DPO config — the levers. Note beta (the drift temperature).
training_args = DPOConfig(
    output_dir=OUTPUT_DIR,
    # --- The model and reference ---
    model=MODEL_ID,                  # the SFT'd model — policy starts here
    ref_model=MODEL_ID,             # the SAME model, frozen as the reference
    # --- Batch + accumulation ---
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,  # effective batch = 16
    # --- LR: LoRA band, slightly lower than SFT (DPO is sensitive) ---
    learning_rate=5e-5,             # DPO typically uses 1e-5 to 5e-5 (lower than SFT LoRA)
    num_train_epochs=1,
    # --- β — THE DPO HYPERPARAMETER. Controls drift from the reference. ---
    beta=0.1,                       # common starting point (0.1–0.5)
    # --- Warmup + scheduler ---
    warmup_ratio=0.1,
    lr_scheduler_type="cosine",
    # --- Precision ---
    bf16=bf16_ok,
    fp16=not bf16_ok,
    # --- Memory + speed ---
    gradient_checkpointing=True,
    # --- Logging + eval ---
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=50,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    # --- Experiment tracking ---
    report_to="wandb",              # or "trackio" / "tensorboard"
    # --- DPO specifics ---
    max_length=1024,
    max_prompt_length=512,
)
```

**Record**: your effective batch size, your LR, your β, and your precision (BF16/FP16). The three things that most affect a DPO run are: the data quality (Phase 2), β (here), and the LR.

> **Teaching moment:** Every lever here answers a question from the teaching doc. `beta=0.1` is the drift temperature — too low and the policy over-optimizes; too high and nothing moves. `learning_rate=5e-5` is lower than the SFT LoRA band because DPO is sensitive and you're refining an already-good model. `ref_model=MODEL_ID` makes the SFT model the reference — the anchor. None of these are arbitrary.

---

## Phase 4 — Run DPO training (15–25 min)

```python
from trl import DPOTrainer

# Log in to your tracking backend once in your shell:
#   wandb:  export WANDB_API_KEY=...   (or run `wandb login`)

trainer = DPOTrainer(
    args=training_args,
    train_dataset=train_pref,
    eval_dataset=eval_pref,
    peft_config=peft_config,
)

# Count trainable params (the thesis, quantified)
trainable = sum(p.numel() for p in trainer.model.parameters() if p.requires_grad)
total = sum(p.numel() for p in trainer.model.parameters())
print(f"Trainable params: {trainable:,} / {total:,} = {100*trainable/total:.3f}%")

# THE DPO LOOP
trainer.train()
```

**While it runs**, open your tracking dashboard and watch:

- **rewards/accuracies** — the fraction of pairs where the policy assigns higher implicit reward to chosen than rejected. Should rise from ~0.5 (chance) toward ~0.8–0.9.
- **rewards/margins** — the difference between chosen and rejected implicit rewards. Should grow positive.
- **rewards/chosen** and **rewards/rejected** — chosen should rise; rejected should fall or stay flat. If BOTH rise, the model is drifting (β may be too low).
- **eval metrics** — every 50 steps. Watch for eval reward accuracy diverging from train (overfitting signal).

**Record**: a screenshot or text description of the reward curves. Note the final `rewards/accuracies` (train and eval), and whether the margin grew positive.

> **If rewards/accuracies plateaus at ~0.5:** the model isn't learning a preference. Check your data for signal (the Phase 2 signal test). If chosen and rejected are too similar, there's nothing to learn. This is the "no-signal data" anti-pattern.
>
> **If the model degrades (outputs become weird or ultra-verbose):** over-optimization. β is too low. Raise it to 0.3 or 0.5, or switch to IPO. The train reward may keep rising while eval quality drops — that's the overfitting signal.

---

## Phase 5 — Save and merge the DPO'd model (5 min)

```python
# Save the DPO'd adapter
trainer.save_model(f"{OUTPUT_DIR}/best-dpo-adapter")
print(f"Saved DPO adapter to {OUTPUT_DIR}/best-dpo-adapter")
```

For deployment, merge the adapter back into the base SFT model:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Reload the SFT model fresh (this is the base for the merge)
base = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, torch_dtype=torch.bfloat16, device_map="auto"
)
# Load the DPO adapter on top
model = PeftModel.from_pretrained(base, f"{OUTPUT_DIR}/best-dpo-adapter")
# MERGE
merged = model.merge_and_unload()
print("Merged. DPO adapter folded into the SFT model.")

MERGED_DIR = f"{OUTPUT_DIR}/merged-dpo-model"
merged.save_pretrained(MERGED_DIR)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.save_pretrained(MERGED_DIR)
print(f"Saved standalone merged DPO model to {MERGED_DIR}")
```

**Record**: adapter dir size vs merged model dir size.

---

## Phase 6 — Measure win-rate improvement (10 min)

The payoff. You will measure whether the DPO'd model is preferred over the original SFT model on held-out prompts, using a model-as-judge. This is the standard preference-alignment eval.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def generate(model, tokenizer, prompt, max_new_tokens=120):
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tokenizer(text, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
    return tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

# Held-out eval prompts (NOT from the training set)
EVAL_PROMPTS = [
    "Explain what an API gateway does.",
    "Write a function to check if a string is a palindrome.",
    "What is the difference between supervised and unsupervised learning?",
    "How does HTTPS encryption work?",
    "Explain the concept of Big O notation.",
]

# Generate from BOTH models on the same prompts
sft_tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
# (sft_model already loaded as `base`; merged is the DPO'd model)

print("Generating responses from both models...")
sft_responses = [generate(base, sft_tokenizer, p) for p in EVAL_PROMPTS]
dpo_responses = [generate(merged, sft_tokenizer, p) for p in EVAL_PROMPTS]

# --- Win-rate via model-as-judge ---
# Use a capable model to judge which response is better.
# For the lab, use a simple heuristic judge (length + directness) or a real model.
# The solution key shows a proper model-judge using the same model or an API.

def simple_judge(sft_resp, dpo_resp):
    """A simple proxy judge for the lab: prefer shorter, non-hedging responses.
    In production, replace with a real model-as-judge (GPT-4-class or large open model)."""
    sft_score = -len(sft_resp) - (10 if sft_resp.lower().startswith(("well", "that's", "let me")) else 0)
    dpo_score = -len(dpo_resp) - (10 if dpo_resp.lower().startswith(("well", "that's", "let me")) else 0)
    return "dpo" if dpo_score > sft_score else ("sft" if sft_score > dpo_score else "tie")

wins = {"dpo": 0, "sft": 0, "tie": 0}
for i, p in enumerate(EVAL_PROMPTS):
    verdict = simple_judge(sft_responses[i], dpo_responses[i])
    wins[verdict] += 1
    print(f"\nPrompt: {p}")
    print(f"  SFT:  {sft_responses[i][:100]}...")
    print(f"  DPO:  {dpo_responses[i][:100]}...")
    print(f"  Judge: {verdict}")

total = sum(wins.values())
print(f"\n=== WIN RATE ===")
print(f"DPO wins: {wins['dpo']}/{total} ({100*wins['dpo']/total:.0f}%)")
print(f"SFT wins: {wins['sft']}/{total} ({100*wins['sft']/total:.0f}%)")
print(f"Ties:     {wins['tie']}/{total} ({100*wins['tie']/total:.0f}%)")
```

**Record**: the win-rate table (DPO wins / SFT wins / ties), and 2–3 before/after response pairs with a one-sentence observation of what changed.

> **Note on the judge:** The lab uses a simple proxy judge (conciseness + directness) so it runs without external dependencies. In production, replace `simple_judge` with a real model-as-judge — call a GPT-4-class model or a large open model (Llama 3.1 70B / Qwen 2.5 72B) with a rubric prompt asking which response is better and why. The win rate is only as trustworthy as the judge.

---

## Deliverables

Submit `ft13-lab-report.md`:

- [ ] **Phase 1**: model chosen (your FT12 output or provided), and confirmation it is SFT'd.
- [ ] **Phase 2**: train/eval pair counts, one sample pair, and the result of the signal test (could you articulate the preference?).
- [ ] **Phase 3**: effective batch size, LR, β, precision (BF16/FP16), and a one-line justification for each.
- [ ] **Phase 4**: reward-curve screenshot or description (rewards/accuracies and rewards/margins); final train and eval reward accuracy; whether the margin grew positive.
- [ ] **Phase 5**: adapter dir size vs merged model dir size.
- [ ] **Phase 6**: win-rate table (DPO / SFT / ties) and 2–3 before/after response pairs with a 1–2 sentence observation of what changed.

---

## Solution key

- **Phase 1**: For `Qwen2.5-1.5B-Instruct`: a capable open instruct model, already SFT'd by Qwen. If the student used their FT12 output, even better — they're aligning their own model.
- **Phase 2**: For the provided `trl-lib/preference-dataset`: ~200–500 pairs. Train ~90%, eval ~10%. Each example has `prompt`, `chosen`, `rejected`. The signal test should pass — the difference between chosen and rejected must be articulable (concise/direct vs verbose/hedging, or helpful vs refusal). If the student used the synthetic fallback, the preference is encoded directly in construction (chosen is concise, rejected is verbose/hedging).
- **Phase 3**: effective batch = 2 × 8 = 16. LR = 5e-5 (DPO is more sensitive than SFT; lower LR). β = 0.1 (the common starting point). On a 4090/A100: BF16. On a Colab T4: FP16. Trainable params for QLoRA r=16 on a 1.5B model: ~0.1–0.3%.
- **Phase 4**: A healthy DPO run shows `rewards/accuracies` rising from ~0.5 to ~0.75–0.9, with `rewards/margins` growing positive. `rewards/chosen` rises; `rewards/rejected` falls or stays flat. Eval accuracy should track train accuracy; a divergence (train rising, eval stalling) indicates overfitting — raise β or improve data. If accuracies plateau at 0.5, the data has no signal. If the model outputs degrade (ultra-verbose or weird), β is too low.
- **Phase 5**: adapter ~20–80 MB (1.5B model); merged model ~3 GB (1.5B params × 2 bytes BF16). The size difference is PEFT.
- **Phase 6**: With a well-constructed dataset (concise vs verbose preference), the DPO'd model should win the majority of held-out comparisons — DPO win rate of 60–80% is healthy. The before/after should show the DPO'd model producing more concise, direct responses. If the win rate is ~50% (no improvement), either the data lacked signal or β was too high (no drift). If win rate is very high (>95%) but outputs are degenerate, β was too low (over-optimization).
- **On the judge**: The simple proxy judge (conciseness + directness) is a teaching tool. For a real evaluation, use a model-as-judge with a rubric. Win-rate numbers from the proxy judge are biased toward the preference the data encoded — which is the point (we encoded that preference), but real-world eval needs a more neutral judge.

---

## Stretch goals

1. **Tune β.** Re-run with β = 0.3 and β = 0.5. Compare the reward curves and the final win rate. Which β gave the best win rate without over-optimization? (You are building intuition for the drift temperature.)
2. **Try SimPO (reference-free).** Swap `DPOTrainer` for `SimPOTrainer` (if available in your TRL version) and remove the `ref_model`. Compare VRAM usage (SimPO should use roughly half) and win rate. Does SimPO match or beat DPO on your data?
3. **Try IPO.** Re-run with `IPOTrainer` (or the `loss_type="ipo"` option in `DPOConfig`). Compare overfitting behavior — does IPO's regularization keep eval accuracy closer to train accuracy on noisy data?
4. **Use a real model-as-judge.** Replace `simple_judge` with a call to a capable model (via an API or a locally-served large model) using a rubric prompt. How does the win rate change with a more neutral judge? (You are building the eval skills for production preference alignment.)
5. **Build pairs with AI feedback.** Instead of the synthetic construction, use a strong model as a judge to label which of two generated responses is better, given a rubric. This is the RLAIF / UltraFeedback pattern. Compare the resulting DPO win rate to the synthetic-construction approach. (You are learning the tradeoff between control and judge-quality.)