How to Fix Prompt Cache Misses in OpenAI and Anthropic

If your LLM bill assumes a high cache-hit rate but telemetry shows mostly uncached input, fix the request layout before changing models. The highest-value move is usually simple: make one large prefix stable, place volatile data after it, and prove the change with provider usage fields.

Provider behavior, limits, and pricing multipliers below were checked against official documentation as of 2026-07-20.

First, understand what must match

Both providers cache prompt prefixes, not arbitrary repeated fragments. A prefix is everything from the start of the rendered prompt through a cache boundary. If an early item changes, identical material later in the request cannot rescue that prefix.

OpenAI’s prompt-caching guide says eligible prompts are cached automatically from 1,024 tokens and require exact prefix matches. Static instructions, examples, images, tools, and schemas belong first; user-specific content belongs last. On GPT-5.6 and later families, a stable prompt_cache_key improves matching, and explicit breakpoints can mark reusable prefixes.

Anthropic’s prompt-caching guide evaluates the full prefix in this order: tools, system, then messages. You can enable automatic caching at request level or put cache_control on a specific content block. The default cache lifetime is five minutes; a one-hour TTL costs more to write. Minimum cacheable length varies by model and platform, so check the official table for the exact combination you deploy.

Find the first rendered-prefix difference

Capture two consecutive request bodies after your SDK has constructed them. Remove secrets, then compare their model, tools, system content, message order, images, output schema, and prompt-affecting options. Start at the beginning and stop at the first difference.

The usual causes are:

• Dynamic metadata: timestamps, trace IDs, user IDs, locale strings, or deployment hashes interpolated into the system prompt.
• Reordered content: tools built from a map, retrieved documents sorted by score, or examples assembled in nondeterministic order.
• Schema churn: generated JSON Schema changes property order, descriptions, defaults, or tool versions between requests.
• History edits: summarization or redaction rewrites an earlier message instead of appending a new message.
• Configuration drift: model routing, tool choice, image detail, structured-output settings, or beta headers change inside one conversation.
• Expired or unwarmed entries: the request matches, but the previous cache entry has aged out or concurrent requests arrived before the first response began.

Anthropic also offers a beta cache diagnostics guide for the Claude API. It compares a request with a previous response ID and reports the earliest structural divergence. Treat its missed-token estimate as a debugging magnitude, not a billing value.

Refactor the request: before and after

This pattern looks convenient but puts churn before reusable material:

const system = `Request ${crypto.randomUUID()} at ${new Date().toISOString()}
${longPolicy}
${examples}`;

const tools = discoveredTools; // arrival order can vary
const input = customerQuestion;

Instead, version and serialize the stable prefix deliberately. Move request metadata and the current question after it:

const tools = [...discoveredTools].sort((a, b) =>
  a.name.localeCompare(b.name)
);

const stableSystem = `${longPolicy}\n${examples}\nSchema version: 7`;
const dynamicInput = JSON.stringify({
  requestId,
  timestamp,
  question: customerQuestion
});

For OpenAI, send a stable key such as support:v7 with requests sharing that exact prefix. On GPT-5.6 and later, place an explicit breakpoint after stableSystem or a stable file when you want precise control. Do not use one key for unrelated prefixes; OpenAI’s official guide says to keep traffic across all prefixes for a key to approximately 15 requests per minute and partition higher-volume traffic with a stable mapping, as of 2026-07-20.

For Anthropic, put cache_control: { "type": "ephemeral" } on the last stable system or message block. Use ttl: "1h" only when expected reuse falls outside five minutes. Keep tool definitions and their order fixed because a tool change invalidates tools, system, and message caches.

Read telemetry instead of inferring a hit

OpenAI Responses exposes cache reads under usage.input_tokens_details.cached_tokens; Chat Completions uses usage.prompt_tokens_details.cached_tokens. GPT-5.6 and later also report cache_write_tokens. Log both values with prompt version, cache key, model, and request time.

A healthy repeated request should show a write on the first call and cached tokens on later calls. Zero cached tokens can mean a short prompt, a changed prefix, an expired entry, or poor routing. A partial hit is useful evidence: the stable prefix matched until a later divergence.

Anthropic separates three quantities:

total input = cache_read_input_tokens
            + cache_creation_input_tokens
            + input_tokens

Here, cache_read_input_tokens is reused prefix content, cache_creation_input_tokens is newly written prefix content, and input_tokens is content after the final breakpoint. Do not compare only input_tokens with your pre-caching total; that field no longer represents the whole prompt.

Track two ratios per prompt version over a fixed time window: sum(cache_read) / sum(total_input) and sum(cache_read) / sum(cache_creation). The first describes request coverage. The second helps reveal prefixes that are repeatedly written but seldom reused; handle a zero write denominator explicitly.

Calculate the break-even point

Let U be the uncached input price, W the cache-write price, R the cache-read price, and h the number of successful reads after one write. For the cacheable prefix only:

cached cycle cost   = W + hR
uncached cycle cost = (h + 1)U

Caching is cheaper when W + hR < (h + 1)U. Dynamic tail and output costs cancel from this comparison if they are unchanged.

As of 2026-07-20, Anthropic documents five-minute writes at 1.25U, one-hour writes at 2U, and reads at 0.1U. One five-minute write plus one hit costs 1.35U, versus 2U uncached. A one-hour entry needs at least two hits: one write plus two reads costs 2.2U, versus 3U uncached.

OpenAI documents GPT-5.6-and-later writes at 1.25U; the read rate depends on the model’s official cached-input price. Substitute that rate rather than assuming every model has the same discount. Earlier model families do not add a cache-write fee, but you should still measure whether the cacheable prefix is large and reused often enough to matter.

Troubleshooting checklist

1. Confirm the deployed model and platform support caching and meet the documented token minimum.
2. Log cache reads, cache writes, total input, model, prompt version, key, and timestamps.
3. Compare fully constructed request bodies and locate the first prefix difference.
4. Move IDs, dates, user state, retrieved context, and current questions after the stable boundary.
5. Sort tools and examples deterministically; pin schema and tool versions.
6. Keep model and prompt-affecting options constant within a cached conversation.
7. Send one request, wait until its response begins, then test an identical second request.
8. Test again within the TTL before blaming prefix construction.
9. Calculate write-versus-read cost from actual telemetry, not headline discount percentages.
10. Alert on falling cache-read coverage and rising writes per read after each prompt deployment.

The practical stopping point

Do not contort unique, short prompts to qualify for caching. Optimize the large prefixes reused across support flows, coding agents, document analysis, or tool-heavy assistants. Once telemetry shows stable reads, compare cost per accepted result before and after the change; lower input cost is not useful if a stale schema or missing dynamic context reduces answer quality.