A finance lead builds next quarter's inference budget off a vendor's cost-reduction headline, then watches the actual bill land 3x over model. The gap isn't a billing error. It's the difference between a benchmark number and a production number, and the two are measuring different things.
Anthropic advertises "up to 90%" cost reduction and "up to 85%" latency reduction from prompt caching, for long, static prompts reused verbatim. That's a real number for that specific shape of workload. It is not what most agentic systems see in production. Anthropic's own multi-turn conversation benchmark already shows -53% cost, well below the headline, before a single tool call or context-window compaction enters the picture.
Microsoft Research's production telemetry on GitHub Copilot puts a harder floor under the gap. Drawing on 3.2 million users, 13 million sessions, and 761 million LLM calls from June 2026, prompt-prefix caching accounts for 90% of tokens within a single turn, on average. Cross a turn boundary and that drops to 55%. Switch models mid-session and it collapses to 8%, a 67% average decline. Context compaction produces a median 66.1% drop on the very next call. Cache hit rate is not a property of the vendor's infrastructure. It's a property of how your system prompt is built, and most teams build it wrong by default.
The vendor headline is a best case, not a baseline
Treat Anthropic's 90% figure as a ceiling, reachable under one specific condition: a long, stable prompt sent once and reused unchanged across every subsequent call. Most production agents don't work that way. They inject tool results, update conversation history, and carry per-turn state, and each of those changes the bytes the cache is keyed on.
The Microsoft Copilot numbers are the closer reference class for anyone running tool-calling agents in production, because they're measured on exactly that workload at scale rather than on a synthetic long-document benchmark:
| Event | Cache hit rate | Drop from single-turn baseline |
|---|---|---|
| Single turn | 90% | baseline |
| Cross-turn boundary | 55% | -39% |
| After model switch | 8% | -67% (average) |
| First call after context compaction | median -66.1% | steepest single-event drop |
The 90-to-55 gap is a fixed cost of turn-boundary structure. Every agentic system pays it, whether or not the team knows it's paying, unless the system prompt is deliberately architected to hold the prefix stable across turns.
What actually breaks the cache
Prompt caching works by hashing a prefix of the request and reusing the computed key-value state on the next call if that prefix matches byte-for-byte. Anything that changes the prefix, even one token, forces a full recompute. Lumer et al., in the first systematic cross-provider evaluation of prompt caching for multi-turn agentic tasks (OpenAI, Anthropic, and Google), confirm cost savings of 41-80% and time-to-first-token gains of 13-31% are achievable, but their central finding cuts the other way:
— Lumer et al., 'Don't Break the Cache,' PwC 2026Strategic prompt cache block control, such as placing dynamic content at the end of the system prompt, avoiding dynamic traditional function calling, and excluding dynamic tool results, provides more consistent benefits than naive full-context caching, which can paradoxically increase latency.
Naive caching can make a system slower, not just cheaper to run badly, when dynamic content sits ahead of the stable prefix and forces a rewrite on every turn. Their recommendation is specific and actionable: put dynamic content at the end of the system prompt, exclude dynamic tool results from the cached block, and avoid injecting per-turn tool schemas ahead of the user message. Tool definitions that change based on session state, feature flags, or user permissions are the single most common way teams unknowingly rebuild the prefix on every request. If your tool list is computed per-request and placed before the cached block, you've built a system that can never sustain a high hit rate no matter how long the static portion of the prompt is.
The same discipline applies to anything else that's technically stable in meaning but unstable in bytes. Timestamps, request IDs, and session identifiers embedded near the top of a system prompt will silently bust a prefix that would otherwise be reusable across the whole session. This isn't a finding from a specific paper; it's the direct, mechanical consequence of how prefix hashing works, and it's worth treating as a design checklist item rather than something you discover by comparing a cache_read_input_tokens field against your expectations after the fact.
Hit rate has a floor even when you do everything right
There's a second constraint the design-discipline framing above doesn't cover: prompt length itself. Song's 2026 research from PayPal on cache-aware prompt compression analyzed Claude Sonnet 4.6's caching behavior and found it isn't a simple hit-or-miss mechanism. It's a two-tier architecture with a sharp threshold near 3,500 tokens:
— Song, 'Cache-Aware Prompt Compression,' PayPal 2026Anthropic's cache has a two-tier architecture with a sharp threshold near 3,500 tokens, below which the hit rate plateaus at ρ≈0.83 across 30-call sessions.
Below that threshold, hit rate plateaus around 0.83 regardless of how disciplined the prefix ordering is. Above it, hit rate approaches 1.0. A short, tightly-scoped system prompt, the kind a lean single-purpose agent might reasonably ship with, structurally cannot reach the 90%+ headline no matter how well it follows the "static content first" rule. That's a useful number to have in your back pocket the next time a cost projection assumes near-100% hit rates are just a matter of good prompt hygiene.
The same research debunks a common instinct: that shrinking the prompt (compression) is the obvious lever when caching alone isn't hitting target savings. Query-aware compression, the dominant academic approach, changes the prompt's content based on the incoming query, which changes the prefix on every single call. That breaks caching entirely. Song's CAPC (cache-aware prompt compression) method, which compresses in a way that preserves prefix stability, achieved 49% savings over cache-only, 64% over query-aware compression, and 90% over vanilla approaches with no caching or compression. The lesson generalizes past this one paper: any optimization applied to your prompt needs to be evaluated for its effect on prefix stability, not just token count. A technique that shrinks tokens but changes the prefix on every call can cost more than doing nothing.
Multi-tenant caching is a correctness question, not just a cost question
Everything above treats hit rate as a cost and latency problem. In multi-tenant deployments, it's also a correctness problem, and it deserves separate engineering attention rather than getting folded into the cost conversation.
When a system prompt or cached prefix is shared across tenants for cache efficiency, and dynamic per-tenant context (permissions, feature flags, account-specific instructions) is layered on afterward, the door is open for cached state computed for one tenant's context to be reused incorrectly against another tenant's request if the boundary between "shared, cacheable" and "tenant-specific, must-recompute" isn't drawn precisely. No paper in this research set documents a specific incident here; treat it as an open engineering question your architecture review should ask explicitly rather than as a citable finding. The practical guidance: any content that varies by tenant, not just by turn, belongs outside the shared cached prefix by construction, with the cache key itself scoped to include tenant identity. Get that boundary wrong and the failure mode isn't a slow response. It's tenant A's cached reasoning influencing tenant B's answer.
That's the failure mode in the abstract. In practice it shows up as a handful of concrete implementation mistakes, and they're worth naming because they're easy to make without noticing. The first is keying the cache purely on prompt content hash with no tenant identifier folded into the key at all, on the assumption that byte-identical prefixes are safe to share because they're byte-identical. Two tenants on the same pricing tier with the same boilerplate system prompt will produce that exact collision, and nothing in a content-only hash distinguishes "these two requests happen to look the same" from "these two requests are the same." The second is caching at a layer that sits below your application's tenant-scoping logic, a shared inference gateway or a provider-level cache that multiple services route through, where the team that owns the caching layer isn't the team that owns tenant isolation and neither one is checking the other's assumptions. The third, subtler mistake is caching tool results rather than just the tool schema: a static tool definition is safe to share across tenants because it doesn't contain tenant data, but a cached result from calling that tool is a snapshot of one tenant's account state, and reusing it against a different tenant's session is a data leak dressed up as a performance win. The fix for all three is the same discipline as the turn-boundary problem above, applied one layer down: draw the tenant boundary at the cache key, not at the application logic that happens to run after the cache returns, and treat "is this safe to share" as a property you verify per field, not one you assume because the bytes match.
Treat token spend like a resource with a design contract
The pattern across all three sources is the same: cache hit rate is not something that happens to your system. It's something your system prompt's structure determines, the same way memory locality determines cache performance in any other system you've built. The 2026 research formalizes this as a two-tier cost model precisely because "just enable caching" and "just compress the prompt" both turn out to be the wrong level of abstraction. The right level is prefix ordering: static content first, ordered from least to most volatile, dynamic content and tool results last, tenant-specific state scoped outside the shared block entirely.
That's a design contract, not a runtime flag. It belongs in the same review your team already does for schema changes, not in a post-launch optimization sprint after finance asks why the bill doesn't match the vendor slide deck.
Before next sprint
Pull your own trace logs and measure single-turn versus cross-turn cache hit rate the way Microsoft measured Copilot's, using whatever cache-read/cache-write token fields your provider exposes. If cross-turn hit rate lags single-turn by a wide margin, the fix is prefix ordering: static system prompt and tool schema first, session-specific and per-tenant content last, not a bigger cache budget. A cost model built on a hit rate you haven't measured is a guess wearing a spreadsheet.
References
- Anthropic — Prompt caching with Claude (2025)
- Microsoft Research — Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale (2026)
- Lumer et al. — Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks (PwC, 2026)
- Song — Cache-Aware Prompt Compression: A Two-Tier Cost Model for LLM API Caching (PayPal, 2026)