See what the model sees
Tokens and token IDs
Turn text into the numbered pieces a language model receives and notice that tokens are not the same as words.
8 minute lesson
A model does not receive a string or a list of English words. A tokenizer converts text into pieces and maps each piece to an integer ID.
Imagine this tiny vocabulary:
0: <end>
1: " the"
2: " cat"
3: " sat"
4: "."The text the cat sat. could become [1, 2, 3, 4]. The model sees those IDs, then looks up a learned vector for each one.
Open Tokenization in the app. Enter one of your baseline prompts and inspect the token stream.
Try a common word, an unusual name, punctuation, and an emoji. Record which inputs use one token and which split into several pieces.
Check round-trip behavior too:
decode(encode(text)) = textIf normalization changes spaces, punctuation, or Unicode, document it. The tokenizer defines the exact text the model can reconstruct.
The model predicts one next token from its vocabulary. It does not first decide on a complete sentence.
A token can include a leading space or only part of a word. That is why token counts do not match word counts.
Sequence length affects compute. If a 60-character prompt becomes 20 tokens, a context length of 64 leaves room for 44 more input or generated tokens. A different tokenizer may split the same prompt into 35 tokens and leave less room.
What to observe: familiar patterns tend to use fewer reusable pieces. Rare text often splits more.
Lesson completed