Compare checkpoints fairly

Create a table in your notebook with one row per prompt and one column per checkpoint.

Use the same baseline prompts, generation length, temperature, top-k, top-p, min-p, and seed where available.

Compare model stages separately:

random initialization → pre-trained checkpoints
best pre-trained checkpoint → SFT checkpoint
SFT checkpoint → DPO checkpoint

Each transition answers a different question. Combining them into “the final model is better” hides which training stage changed the behavior.

Score only things you can explain. Useful columns include complete sentence, repetition, follows requested format, stays on topic, and unsupported claims.

Define the rubric before reading final samples. For a binary item such as “follows requested JSON format,” write the exact passing condition.

Use several fixed seeds when sampling is stochastic. Report a count such as 4 of 5 outputs completed the sentence, not only the best example.

Keep the raw samples under the table. A short score without its output is hard to audit.

Add loss and compute context beside behavioral results:

Checkpoint	Training step	Validation loss	Training time	Notes
Early
Best validation
Final

Check evaluation prompts against the training data when possible. A copied answer is evidence of memorization, not held-out generalization.

Choose the checkpoint you would keep and write one evidence-based reason. The latest checkpoint does not have to be the best one.

If two checkpoints are close, say the experiment cannot distinguish them. Honest uncertainty is more useful than choosing a winner from noise.