Prompt caching stores the model's computation over a repeated prompt prefix so a later request sharing that prefix skips recomputing it. A cache write costs more than a normal request; a cache hit is billed at a steep discount. It only works when the shared content sits first in the prompt, before anything that changes per request.
Key takeaways
- What's cached is computed state, not response text — the model's internal representation of having processed a specific prefix, reused instead of recomputed.
- Cache writes cost more, cache reads cost less than a normal (uncached) request — the discount only shows up on a hit, not on the request that first populates the cache.
- Prefix order is the whole mechanism. Only content that's byte-identical from the start of the prompt can be reused, so static content has to come before anything that varies per request.
- Cache lifetimes are short and provider-specific — typically minutes, refreshed on each hit — not a persistent store you can rely on across sporadic requests.
- Caching helps most with a large, stable prefix reused often: long system instructions, a big reference document, or a large tool-definition block sent on every call in a session.
What actually gets cached
A transformer model processes a prompt by building up an internal representation as it reads through the tokens in order. Prompt caching stores that intermediate representation for a specific prefix — the exact sequence of tokens from the start of the prompt up to some cut point — so that a later request sharing the identical prefix can reuse the stored computation instead of redoing it. It is not caching the model's eventual response; two requests with the same cached prefix but different final questions still get two different, freshly-generated answers. Only the shared prefix's processing is skipped.
Why a cache write costs more than a normal request
Populating the cache means doing the prefix's computation and then storing the result somewhere retrievable — extra work compared to a normal request that computes and discards. Providers price this as a premium on cache-write tokens relative to standard input token pricing. The payoff comes later: a cache-read token, on a subsequent request that hits the same stored prefix, is billed at a steep discount versus a standard input token, because the actual computation is skipped entirely. The economics only work out if the prefix gets reused enough times that the read discounts outweigh the initial write premium.
| Token category | What happened | Relative cost |
|---|---|---|
| Cache write | Prefix computed and stored for the first time | Premium over standard input |
| Cache read (hit) | Identical prefix found in cache, computation skipped | Steep discount vs. standard input |
| Standard input | Not part of any cached prefix, processed normally | Baseline rate |
Why prompt order decides whether caching helps
Because caching works on a literal, byte-identical prefix match, only content that stays fixed across requests and sits at the very start of the prompt can ever be served from cache. A common mistake is putting the user's specific, ever-changing question first and static system instructions or reference material after it — that ordering guarantees a cache miss on every request, because the "prefix" up to any point includes the part that changed. Putting the stable material first — system prompt, tool definitions, a large reference document — and the variable, per-request content last is what actually makes a hit possible.
Documented vs. inferred vs. unsupported
| Claim | Status | Basis |
|---|---|---|
| Cache-read tokens are billed at a discount vs. standard input tokens | Documented | Published directly in major LLM providers' API pricing pages |
| Prefix order determines whether a cache hit is possible | Documented | Follows directly from how prefix-based caching is described in provider documentation |
| Caching always reduces total cost for any workload | Unsupported | A workload with no repeated prefixes pays the write premium with no offsetting reads — the net effect depends on reuse pattern, not a universal win |
| Cached prompts stay available indefinitely once written | Unsupported | Published cache lifetimes are short and provider-specific, refreshed on each hit rather than persistent |