Prompt caching
In shortPrompt caching lets an AI provider reuse the already processed start of your prompt, so repeated parts are cheaper and faster.
In plain words
Many requests begin with the same long text: a system prompt, tool descriptions, a manual, or the earlier turns of a chat. Processing it again each time wastes work. With prompt caching, the provider keeps the processed form of that fixed start for a short time. If your next request begins with exactly the same text, it reads from the cache and only processes the new part in full.
Details differ by provider:
- Anthropic: The cache lasts 5 minutes by default and is refreshed each time it is used. A 1-hour option exists at extra cost. Cache reads cost far less than normal input, roughly a tenth of the base price for most models. Minimum prompt lengths apply and vary by model.
- OpenAI: Reuse needs the whole rendered prefix to match. Cached input is discounted, and minimum lengths and lifetimes depend on the model.
- Google: Gemini offers implicit caching, which is on by default for newer models, and passes savings on automatically when a request hits the cache.
Why it matters
- Cost: The repeated part is billed at a lower rate. Check the current numbers in AI API prices.
- Speed: Less to process means a faster start of the answer.
- Rate limits: For most Claude models, cached reads do not count toward the input rate limit, as Anthropic's limits page explains.
- Design: Put fixed content first and changing content last. A change early in the prompt breaks the match for everything after it.
- Context: Cached text still fills the context window. Caching changes the price, not the space.
Example
A support bot sends an 8,000-token product manual plus the system prompt with every question. Without caching, each question pays full price for all of it. With caching, questions that arrive within the cache lifetime read the manual from the cache and pay full price only for the customer's short question and the answer.
Common misconceptions
- "The provider caches my answers." It caches the processing of the prompt start. The answer is generated fresh each time.
- "It applies everywhere automatically." Some providers cache on their own; others need markers in your request. Read the docs.
- "Any similar text hits the cache." The start must match exactly. One changed character early on ends the match.
Questions and answers
What is prompt caching?
A feature where the provider keeps the processed start of a prompt for a short time and reuses it, so repeated text costs less and responds faster.
How long does a prompt cache last?
It depends on the provider and the setting. Anthropic's default is 5 minutes, refreshed on each use, with a 1-hour option at extra cost.
How do I get cache hits?
Keep the start of your prompt identical between requests: system prompt, tools and documents first, the changing question last.
Read next
- Context window
The context window is the maximum amount of text, counted in tokens, that a model can read and write in one request.
- Token
A token is a small chunk of text, often a word piece, that an AI model reads, writes and bills.
- Rate limit
A rate limit caps how many requests or tokens you may send to an AI API in a set time; going over returns an error, usually HTTP 429.
- System prompt
A system prompt is a block of standing instructions, set by an app or developer, that shapes how a model behaves across a whole conversation.
- Price per million tokens
Price per million tokens is the usual way AI APIs state their rates: what you pay for each million tokens sent in or generated.