"What is RLHF/PPO and why was it the original alignment method?"	"RLHF (Reinforcement Learning from Human Feedback) with PPO was the original alignment pipeline: (1) SFT a base model, (2) train a separate reward model on human preference pairs, (3) run PPO where the policy generates, the reward model scores, and an RL loop updates the policy with a KL penalty to the reference. Used for InstructGPT and the original ChatGPT. Three models in memory, RL instability. Largely retired for general alignment — replaced by the DPO family."	c3::ft13::recall
"What is DPO and what problem did it solve?"	"DPO (Direct Preference Optimization, Rafailov et al. 2023, arXiv:2305.18290) reparameterizes the RLHF objective into a logistic classification loss on preference pairs. No reward model, no RL loop. The model IS its own reward model — its log-probabilities relative to the reference serve as the implicit reward. Collapsed a 3-model RL pipeline into a single supervised loss runnable on one GPU."	c3::ft13::recall
"State the DPO loss in simplified form and name its components."	"L_DPO = -log σ( β · [ (log π(chosen) - log π_ref(chosen)) - (log π(rejected) - log π_ref(rejected)) ] ). π = trainable policy, π_ref = frozen reference (your SFT model), β = drift temperature, σ = sigmoid. It's a contrastive loss that pushes π to assign more relative probability to chosen than rejected, compared to the reference."	c3::ft13::recall
"What is the reference model (π_ref) in DPO, and why is it load-bearing?"	"The reference model is the frozen SFT model you are improving. The DPO contrast is always against π_ref — it computes the policy's log-probs minus the reference's log-probs for both chosen and rejected. This anchor keeps the policy from drifting arbitrarily. Without it, the model could match the preference data in degenerate ways. DPO is NOT a base-model technique — it needs the SFT starting point."	c3::ft13::recall
"What is β (beta) in DPO, what range is typical, and what goes wrong at each extreme?"	"β is the drift temperature controlling how strongly the policy is anchored to the reference. Typical range: 0.1–0.5, with 0.1 a common start. Too low (e.g., 0.01) = weak anchor, policy drifts far → over-optimization, reward hacking, capability degradation. Too high (e.g., 0.5+) = strong anchor, policy barely moves → weak alignment effect. The right β is empirical."	c3::ft13::recall
"Name the six DPO-family methods and the one-sentence reason to reach for each."	"DPO (default baseline). IPO (regularized — when DPO overfits). KTO (unpaired binary feedback — thumbs up/down). ORPO (combines SFT+preference in one stage, no reference — saves compute). SimPO (reference-free, length-normalized — beats DPO in benchmarks, modern default). R-DPO (regularized for stability/generalization, niche)."	c3::ft13::application
"What is SimPO and why is it a strong modern default?"	"SimPO (Simple Preference Optimization, Meng et al. 2024, arXiv:2405.14734) is reference-free and length-normalized. It removes the reference-model memory overhead (no frozen second copy in VRAM) and adds length normalization so the model can't win by producing shorter/longer responses. In benchmarks it consistently beats DPO. Many production teams have adopted it as the default."	c3::ft13::application
"What is KTO and when do you use it instead of DPO?"	"KTO (Kahneman-Tversky Optimization, Ethayarajh et al. 2024) works on UNPAIRED binary feedback — thumbs up / thumbs down per example — not preference pairs. Use it when your data is a stream of good/bad labels with no pairing (e.g., user feedback logs). It's the one method built for unpaired data. Based on prospect theory from behavioral economics."	c3::ft13::application
"What is ORPO and what makes it different from DPO?"	"ORPO (Hong et al. 2024) combines SFT and preference optimization into a SINGLE training stage with NO reference model. You skip the separate SFT pass — one stage, one model. Use it when you want to save compute or start from a base model in one shot. Tradeoff: less control over SFT quality since it's entangled with the preference loss."	c3::ft13::application
"What is IPO and when do you reach for it?"	"IPO (Interior Point Optimization, Azar et al. 2023) adds regularization to DPO to stop overfitting/over-optimization. Reach for it when DPO overfits your dataset — you see train reward rising while eval degrades, or the model gets weirdly sycophantic. IPO caps how much the implicit reward can grow. Alternatively, raise β in DPO, or (usually) fix the data."	c3::ft13::application
"What is the format of a preference dataset for DPO-family methods (except KTO)?"	"Each example is a triple: {prompt, chosen, rejected}. The prompt is shared. Chosen and rejected are two responses to it — one better, one worse, by your definition of better. DPO trains the model to assign relatively more probability to the kind of response in chosen. KTO is the exception — it takes unpaired binary labels, not pairs."	c3::ft13::recall
"What are the three sources for preference pairs, and the tradeoff of each?"	"(1) Human annotation — present prompt + 2 responses, human picks better. Expensive, slow, high quality (OpenAssistant, Chatbot Arena). (2) AI feedback / model-as-judge — a strong model labels with a rubric (UltraFeedback, RLAIF). Cheap, fast, scalable; quality capped by the judge model. (3) Synthetic — chosen = good response, rejected = degraded/base output. Cheap, controllable, targets specific behaviors. The lab uses this."	c3::ft13::application
"What is the 'signal test' for a preference dataset, and why does it matter?"	"Sample 20 pairs and ask: could a reasonable person tell which is better, and WHY? If you can't articulate the difference, the model can't learn from it. Preference data with no real signal causes DPO to fit noise and degrade the model. The quality bar for DPO is HIGHER than for SFT because the signal is subtler (a preference, not a target). This is the 'steering without a steering wheel' anti-pattern."	c3::ft13::analysis
"Why is DPO on a base (non-SFT'd) model a cardinal anti-pattern?"	"DPO assumes a coherent reference distribution. A base model's outputs are continuations, not instruct responses — the baseline distribution is wrong. 'Preference' over base-model outputs is meaningless. DPO on a base model produces garbage. The reference must be an SFT'd model. SFT-then-DPO is the standard pipeline. (ORPO is the exception that bakes SFT into the same loss.)"	c3::ft13::analysis
"What is the standard SFT-then-DPO pipeline?"	"(1) Start with a base model (Layer 1). (2) SFT on demonstration data to teach format and instruction-following (FT12) — this SFT model becomes the reference π_ref. (3) DPO (or SimPO/ORPO) on top using a preference dataset to shift preferences. Output: a preference-aligned model. This is the pipeline almost every aligned instruct model went through."	c3::ft13::recall
"Where does the DPO family end and GRPO begin? State the line."	"The line is the reward type. DPO family: subjective preference ('which response is better?'), offline, fixed pairs, no exploration — for helpfulness, tone, refusals, style. GRPO (FT14): verifiable rewards (math correct? tests pass?), on-policy, generates and explores — for math, code, tool use. Rule: if a checker can compute the reward, use GRPO; if it's a human/aesthetic judgment, use the DPO family."	c3::ft13::analysis
"Why can pure offline methods (DPO family) NOT do on-policy exploration?"	"DPO learns only from the fixed pairs you gave it — it cannot try a response, see it fail, and try a different one. There's no generation during training and no verifier scoring the outputs. On-policy RL (GRPO) generates candidates, has a verifier score them, and updates the policy toward the winners — discovering correct reasoning paths the model didn't know it could produce. This exploration is strictly better when rewards are verifiable."	c3::ft13::analysis
"Why did PPO largely retire for general alignment but survive for verifiable rewards?"	"PPO is expensive (3 models in memory), unstable (finicky RL dynamics), and hard to debug (reward model has its own failure modes — reward hacking, misgeneralization). DPO replaced it for general preference alignment because it's a simple supervised loss. But PPO-style RL survives for verifiable rewards (math, code, tools) because on-policy exploration discovers solutions that offline methods cannot — there, the RL overhead pays off. That's GRPO in FT14."	c3::ft13::analysis
"What are the four anti-patterns of the DPO family, and the fix for each?"	"(1) DPO on a base (non-SFT) model → no coherent reference; fix: SFT first, always. (2) Over-optimization (β too low) → policy drifts, fits noise/bias; fix: raise β, switch to IPO, fix data. (3) Preference data with no real signal → fits noise, model degrades; fix: audit 20 pairs for learnable difference. (4) Treating DPO as a knowledge tool → it steers preference, doesn't teach facts; fix: use RAG for knowledge, DPO for behavior."	c3::ft13::analysis
"What does it mean that 'the model is secretly a reward model' in DPO?"	"In RLHF, you train a separate reward model to score responses. DPO's reparameterization shows the reward function can be expressed directly in terms of the policy. So DPO uses the policy's OWN log-probabilities (relative to the reference) as the implicit reward — no separate reward model is trained or stored. The contrastive loss pushes these implicit rewards to rank chosen above rejected. The paper's title is literally 'Your Language Model is Secretly a Reward Model.'"	c3::ft13::analysis
"Why does DPO need a held-out eval split, and what signal do you watch?"	"Without eval you can't distinguish alignment from over-optimization. Train reward may keep rising while the model degrades (sycophancy, capability loss, reward hacking). On a held-out preference set, watch the win rate or reward margin: if train reward rises but eval win rate stalls or drops, you've over-optimized (β too low, or noisy data). Same lesson as SFT (FT11): the gap between train and eval is the overfitting signal. Eval is not optional."	c3::ft13::application
"A team has 10k user interactions labeled thumbs-up/thumbs-down (no pairing). Which DPO-family method do they use, and why not DPO?"	"KTO. Their data is unpaired binary feedback, not preference pairs — DPO requires {prompt, chosen, rejected} triples where two responses to the same prompt are compared. KTO (Kahneman-Tversky Optimization) is specifically designed for unpaired good/bad labels. Running DPO would require artificially constructing pairs from the binary labels, introducing noise. KTO uses the labels directly via prospect-theoretic loss."	c3::ft13::analysis
"In the lab, you build synthetic pairs where chosen = good response and rejected = a base-model refusal. What behavior is this targeting, and why does synthetic construction work here?"	"This targets over-refusal — the model declines requests it could safely help with. By pairing a good response (chosen) against a refusal (rejected), DPO learns to assign more probability to helpful responses and less to evasive ones. Synthetic construction works because the signal is clear and controllable: the difference between chosen and rejected IS the behavior you want to instill. No judge model needed; the construction encodes the preference directly."	c3::ft13::analysis