I asked a team what their prompt cache hit rate was. The tech lead said “we cache responses, right?” That isn’t what cache means here, and that single misunderstanding was costing them about fourteen thousand dollars a month.
Prompt caching has been generally available across the major providers for over a year. The discount is real — typically ninety percent off cached tokens after the first hit, sometimes more. The catch is that almost nobody designs their prompts to be cacheable, so almost nobody gets the discount.
The rule is mechanical: caches match by prefix, not by content. If the first token of two requests differs, neither hits cache. If you put a timestamp at the top of the system prompt — “Today is May 19, 2026” — every single call is a cold cache. If you interleave user history before the system prompt, every conversation is a cold cache. The expensive bits — the system prompt, the tool definitions, the few-shot examples — are exactly the bits you want frozen at the start of every request.
What cache-shaped looks like
Three rules cover most of it:
- Static block first. System prompt, role, formatting rules, tool definitions, few-shot examples — all of it before anything that varies. Order matters. A 4 KB static prefix that changes once a quarter is an asset; one buried under the user message is dead weight.
- Dynamic block last, and clearly delimited. User input, current state, retrieved context — at the end, in a single block, so the cache boundary is obvious to humans reviewing the prompt later.
- No clocks, no UUIDs, no random salts in the prefix. They look harmless. They cost you the entire month’s caching benefit.
The diagnostic
Look at your cache hit rate. If it’s under seventy percent on a high-volume agent, your prompt isn’t cache-shaped. The fix is usually a one-day refactor of the prompt template plus a tweak to your retry/log code so it doesn’t perturb the prefix.
The savings won’t show up as a dramatic moment. They’ll show up as next month’s bill being a third of last month’s, and nobody noticing because the dashboards weren’t tracking it. Track it.