Understand AI models
Tokens and generation
See how text becomes tokens and why tokenization affects context, cost, and the model output.
8 minute lesson
Models do not read text as whole sentences. A tokenizer splits input into smaller units called tokens. A token can be a whole word, part of a word, punctuation, whitespace, or a piece of source code.
The exact split depends on the model. A common English word may use one token. An unusual name, a long number, or a word from another language may use several. Do not rely on the idea that one token always means one word.
Tokens share one context window
A model can consider only a limited number of tokens at once. This limit is the context window.
The window includes more than your latest message:
- system and project instructions
- earlier conversation messages
- files and images you attached
- results returned by tools
- the response being generated
If you attach an entire repository, the model does not receive infinite memory. Irrelevant files consume space that could hold important code, requirements, or tool results.
A large context window also does not guarantee equal attention to every detail. Critical constraints can be missed when they are buried inside a long conversation. Put important requirements close to the task and remove irrelevant material.
Tokens affect cost and speed
API providers commonly charge for input and output tokens. A request with 100,000 tokens costs more and takes longer to process than a focused request with 2,000.
Output tokens matter too. Asking for a short table is cheaper and faster than asking for a long report. In an agent loop, every tool result may become input to the next model call, so verbose logs can multiply the cost.
Generation is sequential
The model produces one token, adds it to the context, and predicts again. A small early choice can send the rest of the answer in a different direction. This is one reason retrying can help, but a retry is not a substitute for fixing missing context.
A practical habit
When a conversation becomes confused, do not keep adding corrections forever. Start a fresh task with the goal, the current state, the relevant files, and the decisions that still matter. A clean context is often better than a larger one.
Lesson completed