Run the tokenizer playground

The tokenizer playground starts with small pieces, then repeatedly merges frequent neighboring pairs.

Reset the playground. Run one merge at a time for the first five merges. Record the pair, its frequency, and how the token count changes.

For a simplified corpus containing low low lower, the first representation might look like this:

l o w   l o w   l o w e r

If l o is the most frequent adjacent pair, merging it creates lo. A later merge can create low. The algorithm follows counts, not dictionary definitions.

Then run ten or fifty merges. Compare the final token stream with the first one.

A larger vocabulary can encode familiar text with fewer tokens. It also increases the embedding table and the number of output scores predicted at every position.

Measure both sides in your notebook:

compression = characters in sample / tokens in sample
embedding parameters = vocabulary size × embedding width

For example, growing from 2,000 to 4,000 tokens at width 256 adds roughly 2,000 × 256 = 512,000 embedding values. Weight tying and architecture details can change the exact total, but the tradeoff remains.

Run the same sample from the training domain and an out-of-domain sample. A tokenizer tuned to stories may compress code or another language poorly.

What to observe: the algorithm follows frequency. It does not understand that a piece is a noun or verb.