AI breaks text into smaller pieces before it processes it.
Those pieces are called tokens. A token can be a whole word, part of a word, punctuation or another text fragment.
Watch a sentence become tokens
The split below is illustrative. Exact token boundaries depend on the tokenizer used by a specific model.
One word is not always one token
Choose an example. The coloured blocks represent one possible tokenizer split, not a universal count.
Both sides of the conversation use tokens
Your prompt plus the context included for the model.
The response the model generates.
Tokens become IDs, and tokenisation is not universal.
After tokenisation, each token maps to an ID used by the model. Different model families can use different tokenizers, so the same text may produce different token counts.
Text → token → ID
The ID is illustrative. Token IDs depend on the tokenizer.
Languages and scripts
Character count, word count and token count are different measures. The relationship can vary substantially across languages and scripts.
Why tokens matter
Tokens connect directly to context-window limits and, for many APIs, usage cost.