Compare and report
Write the model lab report
Finish a practical report that records configuration, data, loss, samples, fine-tuning results, and limitations without requiring submission or grading.
20 minute lesson
Copy this outline into model-lab.md and complete every section:
# Model lab report
## Goal
## Mac and app version
## Run manifest
## Model configuration
## Dataset and tokenizer
## Data preparation and split
## Parameter count and estimates
## Compute used
## Training observations
## Training and validation loss
## Checkpoint samples
## Evaluation method
## SFT results
## DPO results
## Final comparison
## Limitations
## Next experiment
Include exact configuration values, dataset source and license, parameter count, run duration, important loss points, and unedited sample outputs.
Connect every conclusion to evidence:
Claim: the middle checkpoint generalizes better than the final checkpoint.
Evidence: lower validation loss and higher fixed-prompt rubric scores across five seeds.
Limit: the test set contains only short story prompts.
Report compute and data honestly. Include processed training tokens, step count, checkpoint interval, elapsed time, and approximate energy or hardware details if you measured them. Do not convert an estimate into a measured value.
If you skipped DPO, say so and explain why. A skipped optional stage is better than an invented result.
End with one next experiment that changes a single variable. Example: keep the blueprint and tokenizer, then compare two datasets with the same step budget.
State what stays fixed and what result would support the hypothesis:
Change: replace the story dataset with the encyclopedia sample.
Fixed: tokenizer, blueprint, token budget, seed, and evaluation settings.
Expected evidence: lower held-out loss on encyclopedia text, weaker story-style scores.
Read the report once and remove any claim the recorded evidence does not support.
Attach artifacts or stable identifiers for the tokenizer, data manifest, checkpoints, raw outputs, and app version. The report should let a future run reproduce the comparison without relying on memory.
Lesson completed