Semantic Caching for LLM Apps: A Practical Implementation Guide

Semantic caching lets an LLM application reuse an earlier response when a new request has the same meaning, even if the wording is different. It can reduce model calls and user-visible latency, but it also introduces a failure mode that an exact cache does not have: a plausible-looking response can be returned for the wrong question.

The safe implementation is therefore not “put a vector database in front of the model.” It is a gated decision system. It must decide which requests are eligible, separate users and response variants, reject uncertain matches, expire entries when their dependencies change, and measure wrong hits as carefully as cost savings.

This article’s implementation details were checked on 2026-07-23 against official Redis semantic-cache documentation, the RedisVL SemanticCache API, and Microsoft’s LLM semantic-cache lookup policy. Those products use different distance or score conventions, so their example thresholds are not interchangeable.

Semantic cache versus exact cache

An exact cache computes a deterministic key from the request and returns a hit only when that key matches. It works well for repeated API requests, normalized templates, and any output where one changed character should produce a new result. It is easy to reason about because equality defines the boundary.

A semantic cache adds an embedding step:

1. Convert the incoming request into a vector.
2. Search previously stored request vectors.
3. Apply hard metadata filters.
4. Compare the nearest result with a similarity or distance threshold.
5. Return the stored response on a qualified hit; otherwise call the LLM and store the result.

That process can match “How do I reset my password?” with “I forgot my password—what now?” It can also confuse “How do I reset another user’s password?” with the self-service question. Vector proximity captures related meaning, not application authorization or factual equivalence.

Use both cache layers when practical. Check a canonical exact key first because it is cheaper and stricter. Run semantic lookup only after an exact miss. The semantic layer should never weaken tenant, locale, permission, model, tool, or data-version boundaries encoded in the exact key.

Semantic caching is also different from provider-side prompt caching. Prompt caching may reuse a shared prefix while still asking the model to generate a fresh answer. A semantic response cache skips generation and returns old output. If repeated prefixes are the main cost, start with fixing prompt cache misses before accepting semantic wrong-hit risk.

Cache layer	Reuses	Best fit	Main failure mode
Exact response cache	A response for the same canonical key	Repeated deterministic requests	An incomplete key serves the wrong variant
Semantic response cache	A response for a meaningfully similar request	Stable, low-risk paraphrases	A related but non-equivalent request gets a plausible wrong answer
Provider prompt cache	Repeated prompt-prefix processing	Long shared instructions or context	Prefix changes prevent a hit; generation still runs

This table describes behavior, not a required stack. An application may use all three, but each layer needs separate hit-rate, latency, and quality measurements.

Choose safe cacheable workloads

Start with requests that are repetitive, low-risk, and stable. Good pilot candidates include public FAQ answers, documentation explanations tied to a version, product-navigation help, and deterministic summaries of immutable text. These workloads have many paraphrases and a clear way to judge whether two requests may share one response.

Exclude a request from semantic caching when any of these conditions applies:

• The answer depends on the caller’s identity, permissions, account state, or private conversation history.
• The request can execute a tool, create a transaction, send a message, or change external state.
• The answer depends on live inventory, current prices, availability, news, or another rapidly changing source.
• The user requests creative variation, a fresh sample, or personalized advice.
• The response contains secrets, personal data, contractual terms, or regulated decisions.
• The model output is nondeterministic by design and variation is part of the product value.
• The request is ambiguous enough that a cached answer could hide the need for clarification.

Treat tool calls separately from model responses. A read-only tool result may be cacheable for a short period if its arguments, authorization scope, and data version are in the key. A state-changing tool call is not. Official Redis integration guidance makes the same distinction with a read-oriented weather example while warning that an action such as sending email is not idempotent (semantic caching integration, checked 2026-07-23).

Build an explicit eligibility function rather than scattering exclusions through prompt text. It should return a reason code such as eligible_public_faq, skip_private, skip_tool, or skip_freshness. Log the code without logging sensitive request content. This makes coverage measurable and lets security rules override any similarity score.

Embed and match incoming requests

Represent the part of the request that determines the answer, not every byte sent to the model. A cache lookup record usually needs:

• canonical user intent text;
• embedding model name and version;
• response and response format;
• tenant or public namespace;
• locale and jurisdiction where relevant;
• application, prompt, model, and knowledge-base versions;
• safety status and validation result;
• creation time, expiry time, and source identifiers.

Do not blindly embed system prompts, hidden instructions, full chat history, or authorization tokens. Instead, put behavior-changing values into exact metadata fields. Microsoft’s current policy supports vary-by values to partition cache entries and specifically recommends user or group identifiers to control cross-user access. Redis documents metadata such as tenant, locale, model version, and safety flags alongside the prompt, embedding, and response.

A read-through flow can remain small:

if not eligible(request):
    return generate(request)

scope = exact_scope(request)
vector = embed(canonical_intent(request))
candidate = nearest_neighbor(vector, filters=scope)

if candidate exists and passes_threshold(candidate):
    if still_valid(candidate) and response_contract_matches(candidate):
        return candidate.response

response = generate(request)
if validated(response):
    store(vector, scope, response, dependencies, expiry)
return response

Keep the embedding model fixed within an index. Changing models can change vector dimensions and neighborhood geometry, so store the model version and rebuild or separate the index during migration. Keep an exact hash of the canonical input as well; it helps distinguish exact hits, semantic hits, and duplicate entries during evaluation.

If embedding calls become a material part of the bill, measure them separately and follow the ingestion controls in reducing embedding costs. A semantic cache is useful only when the avoided generation cost and latency exceed lookup, embedding, storage, and operational overhead.

Set similarity thresholds with labeled data

Do not copy a threshold from a tutorial. Vendors expose different metrics and directions. RedisVL’s current API documents cosine distance in the range 0 to 2, with lower values meaning a stricter match and a default distance_threshold of 0.1. Microsoft’s LLM cache policy exposes a score threshold from 0.0 to 1.0, but its documentation also says lower values require higher similarity. Those verified conventions are product-specific, not a universal “90% similar” scale.

Create a calibration set from real traffic with three labels:

• safe hit: both requests can return the same response without modification;
• unsafe hit: the requests are related but require different answers;
• uncertain: a reviewer needs more context.

Include paraphrases, negation, changed entities, changed numbers, permission differences, locale differences, and near-neighbor intents. For every cached prompt, retrieve several candidates across a range of thresholds. Measure:

semantic precision = safe semantic hits / all semantic hits
semantic recall = safe semantic hits / all reusable request pairs
wrong-hit rate = unsafe semantic hits / all semantic hits

Prioritize precision. A miss merely pays for another model call; a wrong hit silently returns the wrong answer. Select a threshold that keeps the observed wrong-hit rate below the product’s explicit risk budget, then validate it on a holdout set that was not used for tuning.

Use per-route thresholds when intents differ. A narrow FAQ route may tolerate a broader matching boundary than troubleshooting, financial, medical, legal, or account-specific content. Add a second-stage verifier for borderline candidates if the extra latency is justified: compare extracted entities, required facts, answer schema, and dependency versions before accepting the hit.

Redis publishes a threshold-optimization workflow rather than treating its example value as universal (official threshold optimization guide, checked 2026-07-23). Follow that principle even if your storage layer is different.

Prevent stale or private responses

Similarity is never an authorization check. Apply tenant, user group, visibility, locale, jurisdiction, response format, and policy version as filters before vector ranking or as a separate namespace. Do not retrieve broadly and remove forbidden results afterward; even internal candidate access can cross a data boundary.

Set a time-to-live (TTL) from the volatility of the answer’s dependencies:

Dependency	Cache policy
Immutable, versioned documentation	Long TTL plus invalidation when the version changes
Public policy or product configuration	Shorter TTL plus a source-version key
User or account state	Prefer no semantic cache; otherwise isolate per user and expire aggressively
Live or transactional data	Bypass response caching

RedisVL supports a per-record TTL and filterable fields in its SemanticCache API. Microsoft’s paired store policy uses a duration value. These are mechanisms, not freshness guarantees: the application must know which source or prompt change makes an entry invalid.

Store dependency fingerprints such as document revision, retrieval index version, system-prompt version, tool schema version, safety-policy version, and output-schema version. On lookup, reject mismatches even if the vectors are close. On deployment, either bump the namespace or invalidate affected entries. This avoids a dangerous half-migration where old answers are served under new behavior.

Cache only validated responses. Reject model errors, refusals caused by temporary outages, truncated output, failed citations, malformed structured output, and responses that contain restricted data. If a cached answer uses retrieved evidence, preserve source IDs and re-check that the sources remain permitted and current. The evidence controls in reducing RAG hallucinations still apply when the generation step is skipped.

Measure savings and wrong-hit risk

A rising hit rate is not sufficient evidence of success. Track exact hits and semantic hits separately, because they have different cost and risk profiles. At minimum, record:

• eligibility rate and skip reasons;
• exact-hit, semantic-hit, and miss rates;
• embedding, lookup, generation, and end-to-end latency;
• avoided input and output tokens;
• embedding, storage, and cache-service cost;
• stale-hit, cross-scope-hit, and wrong-hit rates;
• cache entry age and invalidation reason;
• fallback rate when cache infrastructure is unavailable.

Calculate net savings rather than gross avoided calls:

net savings =
  avoided generation cost
  - embedding cost
  - cache lookup and storage cost
  - validation cost
  - operational cost

For quality measurement, sample cache hits and replay them through an offline evaluator. Compare the cached response with a fresh response and, more importantly, with a route-specific correctness rubric. Seed adversarial pairs where one word changes the answer: “enable” versus “disable,” “my account” versus “all accounts,” or one product version versus another.

Add a release gate for cache behavior just as you would for model behavior. The guide to evaluating RAG retrieval before the LLM provides a useful pattern: isolate the retrieval decision, label evidence, and test it independently from generation. Here, the retrieval decision is whether an earlier response is safe to reuse.

Production rollout plan

Roll out semantic caching in stages with a kill switch:

1. Shadow mode: perform lookup but always call the model. Log the candidate, distance, scope, and whether the fresh answer would have accepted the cached response.
2. Exact cache only: verify key construction, versioning, invalidation, privacy filters, and metrics before semantic reuse.
3. Internal traffic: enable semantic hits for employees or test tenants on one low-risk route.
4. Small production cohort: serve only high-confidence hits while reviewing a statistically useful sample.
5. Route expansion: add workloads one at a time, each with its own labels, threshold, TTL, and risk budget.
6. Continuous audit: retune after embedding, prompt, model, policy, or corpus changes.

Define rollback before launch. The application should bypass the cache when lookup times out, metrics disappear, a namespace is suspect, or the wrong-hit budget is exceeded. Rate limiting still needs to protect the model backend during a cache outage; Microsoft’s policy documentation explicitly recommends placing rate limiting after cache lookup for this reason.

Before enabling a route, require evidence that tenant isolation works, invalidation reaches affected entries, cached output passes the current response contract, and shadow-mode precision clears the release threshold. Keep the cache optional in the request path. The safest semantic cache is one that can be disabled quickly without making the LLM application unavailable.

Frequently asked questions

What similarity threshold should I use for semantic caching?

There is no portable default. Start strict, label safe and unsafe request pairs from your own route, and choose the threshold on a holdout set. Confirm whether your system exposes similarity or distance and which direction is stricter before comparing numbers from different products.

Can semantic caching leak one user’s data to another?

Yes, if lookup is allowed across an inappropriate scope. Partition or pre-filter entries by tenant, visibility, authorization group, locale, and other response-changing dimensions. For private or account-specific answers, bypass semantic caching unless isolation and invalidation have been explicitly tested.

How long should an LLM semantic cache live?

Set TTL from the fastest-changing dependency, then invalidate on source, prompt, policy, model, or schema version changes. Immutable versioned documentation can use a longer TTL; live, transactional, or user-state answers generally should bypass semantic response caching.

Does semantic caching replace RAG or prompt caching?

No. RAG retrieves evidence for a new answer, provider prompt caching reuses prefix processing, and semantic response caching returns an earlier answer without generation. They solve different problems and can be combined when their keys, versions, and metrics remain separate.

How do I know whether semantic caching is saving money?

Measure avoided generation cost minus embedding, lookup, storage, validation, and operational costs. Report exact and semantic hits separately, and pair net savings with wrong-hit and stale-hit rates; a higher hit rate is not useful if response quality falls outside the route’s risk budget.

Official sources

Verified on 2026-07-24:

• Redis: Semantic cache use case — architecture, metadata filtering, TTL, and non-authoritative cache behavior.
• RedisVL: SemanticCache API — the current distance_threshold range, default, filter fields, and per-record TTL.
• RedisVL: Threshold Optimizers API — first-party support for calibrating cache thresholds rather than copying a tutorial value.
• Microsoft: llm-semantic-cache-lookup policy — score range, vary-by partitioning, and the warning that similar cached responses may be wrong, stale, or unsafe.
• Microsoft: llm-semantic-cache-store policy — required TTL duration and paired lookup/store behavior.

For adjacent implementation work, see reducing embedding costs, evaluating RAG retrieval, and preventing RAG hallucinations. These internal links point to existing articles in this site’s source tree.