Preference data and DPO

Direct preference optimization, or DPO, learns from comparisons.

One example contains a prompt, a preferred response, and a rejected response. The difference should represent a behavior you genuinely want.

For example:

Prompt: Give one release risk.
Preferred: The update path has not been tested from version 1.0.
Rejected: Everything looks great! Here are five unrelated ideas...

This pair expresses focus and evidence. A pair where the preferred response is merely longer teaches an accidental length preference.

Open Direct preference optimization after creating an SFT checkpoint. The preference ballot asks the same prompt twice. Choose the answer you would rather receive, and the app saves the prompt, chosen answer, and rejected answer.

If both answers are identical, the pair is not saved. Sample again. Reject ambiguous pairs where both answers are equally good or where the preferred answer is only longer.

Four saved pairs unlock DPO. That is enough to see the machinery work, not enough to define a broad preference. Apply the same judgment consistently and collect more varied pairs when you want a clearer signal.

The app holds some pairs back to measure preference accuracy. With a tiny set, that number can jump sharply. Record the training and held-out pair counts instead of treating one percentage as stable.

Run a small DPO experiment after SFT. Repeat the same evaluation prompts and settings.

DPO adjusts the trained policy relative to the frozen SFT checkpoint. The two models start with identical weights, so the app’s logistic DPO loss begins near ln(2), or 0.693. Record the pair count and training steps rather than treating every DPO run as equivalent.

Record both improvements and regressions. Preference optimization can move behavior in an unwanted direction when the pairs are weak or narrow.

Check for reward-like shortcuts: excessive brevity, repeated disclaimers, copied phrases, or a style that wins the training pairs but fails new prompts.

The clean comparison is SFT checkpoint → DPO checkpoint, with the same chat template, prompts, seeds, and sampler. Do not attribute a difference to DPO when the generation settings also changed.