What a small model can and cannot do

A small model can learn token patterns, spelling, sentence shapes, recurring names, and simple relationships present in its data.

It can also produce broken grammar, repeat itself, contradict the prompt, or invent details. These failures are part of the experiment.

Do not evaluate it as a general assistant. Ask whether it improves from noise toward the distribution it trained on.

Choose three baseline prompts now. For a story dataset, you might use:

• Once upon a time, a small fox
• The robot opened the door and
• Mia looked at the sky because

Save those exact prompts. We will reuse them at several checkpoints with the same sampling settings.

Add a small rubric before seeing any trained output:

Check	Score
Forms recognizable words	0 or 1
Completes a sentence	0 or 1
Stays on the prompt topic	0 or 1
Avoids obvious repetition	0 or 1

This rubric is intentionally modest. “Factually correct” is a poor primary target for a tiny story model with narrow data.

Generate from the untrained model if the app allows it. Save the raw output, checkpoint state, prompt, sampling settings, and seed. Do not keep only the funniest failure.

Our later claim will be narrow: under fixed generation settings, the trained checkpoints score better on this defined story rubric than the initial state.

What to observe: before training, output should be random or unusable. That gives us a real baseline.