Token
In shortA token is a small chunk of text, often a word piece, that an AI model reads, writes and bills.
In plain words
An AI model does not read letters or whole words. A program called a tokenizer first cuts your text into small pieces, the tokens. A token can be a short word, part of a long word, a number, or a punctuation mark. The model then works only with these pieces. Most tokenizers learn their pieces from data, so common words stay whole and rare words split into parts (subword units).
Google's Gemini docs give a rule of thumb: a token is about 4 characters, and 100 tokens are about 60 to 80 English words. Anthropic adds that the characters per token depend on the model and the language, and that newer Claude models produce about 30 percent more tokens for the same text than earlier ones. Treat any ratio as a guess and count with the tool of the provider you use.
Why it matters
- Cost: API providers bill per token, input and output at separate rates. See price per million tokens and the table in AI API prices.
- Limits: The context window and many rate limits are counted in tokens, not words.
- Language: Other languages, code and numbers often need more tokens than plain English. The same content can cost more in German than in English.
- Speed: A model writes one token at a time, so a long answer takes longer than a short one.
Example
A 1,000-word English email is about 1,250 to 1,700 tokens if you apply Google's rule for Gemini models. You ask for a 150-word summary, which is roughly 190 to 250 output tokens. You pay once for the input tokens and once for the output tokens, each at its own rate. For a quick count before you send, use the token estimator.
Common misconceptions
- "One token is one word." Not so. Short common words are often one token, but long or rare words split into several.
- "All models count the same." Each model family has its own tokenizer. The same text gives different counts at different providers.
- "Spaces and punctuation are free." They are part of the text and often part of a token.
Questions and answers
How many tokens is 1,000 words?
For English text about 1,250 to 1,700 tokens, using Google's rule that 100 tokens are 60 to 80 words. Code and other languages need more. Use your provider's counter for an exact number.
Is a token the same as a word?
No. A token is often a word piece. Short common words are usually one token; long or rare words split into several.
How do I count tokens?
Use the provider's token counter or the token estimator on this site. Providers also report exact counts in every API response.
Read next
- Context window
The context window is the maximum amount of text, counted in tokens, that a model can read and write in one request.
- Price per million tokens
Price per million tokens is the usual way AI APIs state their rates: what you pay for each million tokens sent in or generated.
- Large language model (LLM)
A large language model (LLM) is an AI system trained on huge amounts of text to predict the next token, which lets it write, summarise and answer.
- Token estimator: roughly count tokens in your text
A free tool that estimates how many tokens a text holds. It is a rough guide, not an exact count.