Tokens explained: what a token is, where the idea came from and what it costs you

What a token is

A language model does not read letters or words. Before it sees anything, your text is cut into pieces called tokens by a separate program, the tokenizer. A token can be a whole short word (fun), a piece of a longer word (Token and ization), a number fragment, a punctuation mark, or, for unusual characters, a few raw bytes. The model reads tokens in, and writes tokens out one at a time. Providers bill by them, and they limit prompts by them.

Two details surprise people. A space belongs to the token that follows it ( fun is one token, with its space), so you pay for spacing without seeing it. And numbers get chopped up: $4.56 became four tokens, which is why price tables and logs are expensive to feed into a model. The same word can also be cut differently by different tokenizers, as the last example shows.

How many tokens is a word?

Gemini's documentation puts it at about 4 characters per token, and says 100 tokens is about 60 to 80 English words. I counted instead of quoting: the prose of this site's 62 guides (104,577 words, with code, tables and figures removed) came to 1.23 tokens per word, or 4.79 characters per token, using the o200k_base encoding. That is a little more generous than the documentation's range, because these guides use plain, mostly short words. Treat it as a rule for English prose, not for code, numbers or other languages (next sections).

Why models use tokens instead of letters or words

The model has a fixed list of allowed pieces, its vocabulary, and at every step it scores every entry on that list to decide what comes next. That forces a trade-off, and tokens are the compromise.

Letters make sequences long, and in the original Transformer design the work of comparing every position with every other grows quickly with length, so long sequences are slow and expensive. Whole words make sequences short but break on anything the list has not seen. Subword pieces keep the list a manageable size and still let the model spell out a word it has never met, one piece at a time. For scale, I measured the vocabularies of three public OpenAI encodings: 50,257 entries for gpt2, 100,277 for cl100k_base and 200,019 for o200k_base.

How tokens were invented

Nobody sat down and designed “the token” for chatbots. The idea was borrowed from data compression, then rediscovered by people who needed machine translation to cope with rare words, and finally scaled up.

The 1994, 2012, 2015/2016 and 2018 entries were checked against paper records during this write-up. The GPT-2 entry rests on my measurement of the gpt2 encoding in tiktoken (50,257 entries, byte fallback) and my recollection of the GPT-2 paper, which I did not reopen. Which text each tokenizer was trained on is not stated on the pricing pages I read.

How byte pair encoding builds a vocabulary

The method is simple enough to follow by hand:

1. Start with single characters (or bytes) as the vocabulary, and mark where each word ends.
2. Count every pair of neighbouring pieces across a large amount of text.
3. Merge the most common pair into one new piece and add it to the vocabulary.
4. Repeat until the vocabulary reaches the size you chose. Real tokenizers do this tens of thousands of times.

I ran it on a toy corpus in the style of the one in the translation paper: low five times, lower twice, newest six times, widest three times. These are the first ten merges the program picked:

Notice what happens. The ending est is common, so it is merged early and becomes one piece. By step 8, newest is a single token. widest is rarer, so it ends up as wi + d + est: three tokens. That is the whole idea. Frequent text becomes cheap and rare text becomes expensive. It is also why the same word can cost different amounts in different tokenizers: each one learned its merges from different text.

The byte-level version used since GPT-2 starts from bytes instead of letters, so it never meets a character it cannot encode. Unusual characters simply fall back to their raw bytes, which is why one emoji can cost several tokens.

How many tokens is my text? Words, code, JSON and other languages

Token counts depend on what you write. I counted ten kinds of text with two open-source OpenAI encodings. One sample each, so read the pattern, not the decimals:

• English and Spanish are cheap, near 20 tokens per 100 characters in both encodings.
• Code and JSON cost about twice as much per character (34 to 40), because symbols, indentation and short names break into many small tokens.
• The tokenizer can matter more than the language. The Hindi sentence cost 126 tokens in the older encoding and 38 in the newer one, a 3.3× difference for identical text. A bigger vocabulary that has seen more of a language packs it into fewer tokens.
• Minifying JSON saved about a third. The pretty-printed object cost 72 tokens (cl100k) or 74 (o200k); the minified one 45 or 48. Part of the saving is the indentation, which tokenizers charge for.
• Numbers and emoji are token-heavy: 52 tokens per 100 characters for the number sample.

Why tokens are the unit you pay in

Every provider charges per million tokens, with separate prices for input and output, and most add a third, much lower price for cached input. Here are 14 current models from the three big providers, copied from their own pricing pages:

• Output costs 5 to 8.3 times input. The ratio is exactly 5× on 11 of the 14 rows, 6× on two and 8.3× on one. A long answer is the expensive part of a request.
• Why output costs more is not stated on any of the pages. A reasonable guess, and only a guess, is that the model produces output one token at a time while it can take input in together.
• Hidden thinking counts as output. Gemini's page labels its output price “including thinking tokens”, so a reasoning model can bill far more output than the answer you see.
• The spread is huge. Between the cheapest and dearest rows the input price differs by 100×. The next figure shows the same job priced on all 14.
• Some prices are about to change. Gemini 4 Argon's listed prices double on 1 December 2026 and Gemini 3.8 Flash's on 1 January 2027, per the Gemini page.

Text is not the only thing measured in tokens. Gemini's tokens page says an image up to 384 pixels on both sides counts as 258 tokens, larger images are split into 768-pixel tiles of 258 tokens each, and audio counts as 32 tokens per second, so one minute of audio is 1,920 tokens.

Context windows: how many tokens fit, and what long prompts cost

The context window is the most tokens a model can handle in one request: your prompt, the conversation so far, any documents, and the answer it writes back. Windows are now around a million tokens, but a bigger window is not a free one:

The bigger risk is quiet. A chat model has no memory between calls, so apps resend the whole conversation every turn, and you are billed for it every time.

That is why chat products summarise old turns, why agents trim their tool lists (a four-tool server cost about 750 tokens of context before any call), and why coding agents fetch files on demand instead of loading a whole codebase. The remedy for the repeated part is caching.

How caching changes the maths: input, output and cached tokens

Caching is a discount on the part of the prompt you repeat. The mechanics guide explains what is actually stored; here is what it costs, with the provider numbers from their own pages:

A write costs more than a normal request and a read costs far less, so the question is how often you read. Call the write price w and the read price r, both as multiples of the normal input price, and the share of requests that find a live cache h. The average price of a cached token is h×r + (1 − h)×w, and caching beats not caching when that is below 1. Solving gives a break-even hit rate of (w − 1) ÷ (w − r). For a 5-minute Claude cache (w = 1.25, r = 0.1) that is 0.25 ÷ 1.15, or 21.7%.

• A 5-minute or 30-minute cache pays off once about one request in five is a hit. Two requests inside one lifetime are enough.
• Long caches need more reuse. A 1-hour Claude write (2×) needs about half of requests to hit, and a 12-hour OpenAI write (2.5×) about 61%.
• With no write premium, caching cannot lose. OpenAI's earlier models list no cache-write price and Gemini's implicit caching states none, so any hit saves.
• Short prompts are not cached at all, and you are not told. Anthropic's page says prompts below the minimum “cannot be cached” and no error is returned.

Free caching break-even calculator

Pick a model, describe one typical request, and say how often you expect the cache to still be alive. Everything is calculated in your browser and nothing is sent anywhere. The presets use list prices read on 8 November 2026, which may be out of date by the time you read this. Treat the results as estimates: type in your provider's current prices, or choose “Custom”, and check their page before you budget.

How to count tokens before you pay

The estimates above are for planning. To know, count. For OpenAI's open-source encodings this takes four lines, and I ran them to produce the numbers in this guide:

import tiktoken

enc = tiktoken.get_encoding("o200k_base")
text = open("prompt.txt").read()
print(len(enc.encode(text)), "tokens (o200k_base)")

Two cautions. The pages I read do not say which encoding each current OpenAI model uses, so treat the count as an estimate for those models. And Claude and Gemini tokenizers are not something you can download from the pages I read: use each provider's own token-counting method in its documentation. In production, log the token usage reported in each API response, which is what you are actually billed on.

Seven ways to spend fewer tokens, ranked by size of effect

In rough order of how much they usually save, each with the evidence behind it:

The first three change the unit price or the volume of the biggest line, which is why they come first. Everything else is smaller, but the JSON and tool-list changes are free.

Documented vs. inferred vs. unsupported