Retrieval-augmented generation does not prevent hallucinations by itself. The model can still receive the wrong passages, overlook a limitation, merge incompatible sources, or add details the passages never state.
The practical remedy is to make evidence handling an explicit pipeline. Retrieve passages that could answer the question, preserve enough source structure to cite them, require every material claim to map to a passage, and abstain when that mapping fails. Then measure unsupported answers separately from general answer quality.
This guide treats a hallucination as a material claim that is not supported by the evidence available to the generation step. That operational definition is narrower than “any factually wrong statement,” but it is testable: the verifier can inspect exactly what the model saw.
NIST uses the broader term “confabulation” for confidently presented erroneous or false content, including output that diverges from its input. Its Generative AI Profile recommends measuring and reviewing such risks in context rather than treating them as a single prompt defect (NIST AI 600-1, verified 2026-07-24). This guide narrows that risk into controls a RAG team can test.
Product capabilities and evaluation terminology mentioned below were verified on 2026-07-24 against the official sources linked in the text.
1. Identify retrieval and generation failures
Start by separating failures that are often grouped under one label. A polished but unsupported answer may originate several steps before generation.
| Failure | What happened | Diagnostic signal | First place to intervene |
|---|---|---|---|
| Corpus gap | The answer is absent from the indexed sources | No relevant source exists | Content ownership and ingestion |
| Retrieval miss | A relevant source exists but was not returned | Relevant document is outside the retrieved set | Query rewriting, search, filters, or ranking |
| Context failure | The right source was returned in an unusable fragment | Passage omits a qualifier, heading, table label, or date | Parsing and chunking |
| Synthesis failure | Useful evidence was present but the answer distorted it | Claim conflicts with or exceeds a passage | Prompt, claim extraction, and verification |
| Citation failure | The answer is supported but points to the wrong location | Citation does not entail its attached claim | Citation mapping and rendering |
Log the stages separately: normalized query, filters, retrieved passage IDs, reranked passages, exact model context, generated claims, citations, and the gate decision. Keep sensitive source text out of broad logs.
Do not respond to every failure by changing the prompt. A stricter prompt cannot recover a passage that retrieval missed, while changing chunk size will not enforce a missing synthesis rule. Microsoft’s official guidance likewise distinguishes retrieval evaluation from system-level groundedness, relevance, and completeness evaluation (official RAG evaluators, verified 2026-07-23).
2. Require answer-supporting passages
Retrieval should return evidence that can support an answer, not merely text that resembles the question. Similarity is a candidate-generation signal; it is not proof of entailment.
Define a passage contract before generation. Each passage should carry:
- a stable
passage_idanddocument_id; - a title or section path that explains its local context;
- source version or effective date when freshness matters;
- access-control metadata inherited from the source;
- the exact text supplied to the model;
- a retrieval or reranking score used for debugging, not presented as confidence.
After retrieval, rerank candidates against the question. Apply metadata filters when it implies a product, region, policy version, customer, or time period. For multi-part questions, retrieve for each part.
Add an evidence sufficiency check: does the retrieved set directly support every required part? A high top score is not enough, and repeated partial answers do not cover a missing condition.
A simple internal representation might be:
{
"question_parts": ["eligibility", "deadline", "exceptions"],
"coverage": {
"eligibility": ["p-104"],
"deadline": ["p-221"],
"exceptions": []
},
"decision": "abstain"
}Begin with a deterministic rule plus a small classifier. Require support per question part, reject passages outside scope, and route ambiguity to abstention. Tune on labeled queries rather than adopting a universal similarity threshold. If retrieval architecture is the bottleneck, Pinecone vs Turbopuffer for RAG describes related operational trade-offs.
Before changing the generator, confirm that the required passages are retrievable. How to Evaluate RAG Retrieval Quality covers recall, ranking, and labeled relevance judgments in more detail.
3. Design citation-friendly context
A citation is useful only when a reader or verifier can locate the passage that supports the nearby claim. Design the context and the output format together.
Give each passage a short opaque ID, then preserve boundaries:
[P104]
Source: Employee Handbook > Leave > Eligibility
Version: 2026-06-01
Text: ...
[P221]
Source: Employee Handbook > Leave > Deadlines
Version: 2026-06-01
Text: ...Instruct the model to attach one or more passage IDs to each material sentence, not merely add a source list at the end. Keep quotations distinguishable from paraphrases. If two sources disagree, require the answer to expose the disagreement and the relevant dates instead of blending them into one statement.
Chunks should retain headings, list labels, table headers, units, footnotes, and effective dates. Do not concatenate unrelated passages into one block; doing so obscures which source supports which words.
Do not let the model invent URLs. Resolve passage IDs to canonical links in application code from trusted metadata. Amazon Bedrock’s RetrieveAndGenerate documentation, for example, describes citation objects that associate a generated text span with retrieved references (official response and citation structure, verified 2026-07-23). Preserve that claim-to-passage relationship in any implementation.
4. Add an explicit abstention path
“Answer only from context” is incomplete unless the system specifies when and how to refuse. Give abstention a first-class output state rather than treating it as a generation error.
Define a compact policy:
- Answer only the parts directly supported by supplied passages.
- Do not fill gaps from model memory.
- If a required part lacks support, either provide a clearly labeled partial answer or abstain.
- State what evidence is missing without guessing.
- Suggest a safe next step, such as refining the query or consulting the source owner.
Use structured output so application code can enforce the decision:
{
"status": "answer | partial | abstain",
"answer": "",
"claims": [
{"text": "", "passage_ids": ["P104"]}
],
"missing_evidence": ["exceptions"],
"reason": "No retrieved passage describes exceptions."
}The generator should not judge its own evidence alone. A downstream validator should reject answer when a material claim lacks a citation, cites an unknown passage, or depends on a missing question part.
Set abstention thresholds by consequence. A documentation assistant might return a visibly limited partial answer. A workflow affecting money, access, health, or legal rights should require stronger evidence and may need human review.
5. Verify claims against retrieved text
Verification works better at claim level than at whole-answer level. Split the draft into atomic claims: one subject, one predicate, and any conditions, quantities, dates, or exceptions needed to preserve meaning.
For each claim, check:
- Citation validity: every referenced passage ID exists in the retrieved set.
- Entailment: the cited text supports the claim without importing outside knowledge.
- Scope: product, tenant, jurisdiction, version, and date match.
- Completeness: qualifiers and exceptions are not dropped.
- Conflict: no equally authoritative retrieved passage contradicts the claim.
Use deterministic checks first for passage IDs, dates, quotations, units, and citation presence. An entailment model or LLM judge can handle paraphrases, but require a supporting span and reason. No span means the claim fails.
Calibrate any model-based judge against human labels from your own domain. Google Cloud’s official judge-evaluation guidance uses human ratings as ground truth for pointwise metrics and human preferences for pairwise metrics, then compares judge output with measures such as balanced accuracy, F1, and a confusion matrix (official judge evaluation guide, verified 2026-07-24). A judge that agrees on general examples can still miss domain-specific qualifiers.
Then apply an answer policy:
if any critical claim is unsupported:
abstain or send to review
elif any noncritical claim is unsupported:
remove it and regenerate the affected sentence
else:
publish the answer with resolved citationsDo not verify against the open web unless the product promises web research and records those sources as evidence. Otherwise the verifier silently changes the evidence boundary.
For a broader pattern for machine-enforced release checks, see How to Set Up a Self-Hosted Eval Gate for AI Output. The same principle applies here: deterministic failures should not be averaged away by a strong style or relevance score.
6. Test adversarial and no-answer queries
A useful RAG test set contains more than answerable questions. Build labeled slices that expose different failure modes:
- answerable questions with one clear passage;
- questions requiring evidence from multiple passages;
- questions whose answer is not in the corpus;
- near-match questions where the corpus covers a different product or version;
- false-premise questions that retrieved text explicitly contradicts;
- conflicting or stale sources;
- requests to ignore sources or reveal hidden context;
- malformed queries, ambiguous names, and unsupported languages.
For each case, label the relevant passages, required answer points, forbidden claims, and expected status: answer, partial, or abstain. Keep some cases private to reduce overfitting.
Measure retrieval and generation independently. Retrieval recall asks whether the needed passage appeared in the candidate set. Groundedness asks whether claims stayed within supplied evidence. Citation precision asks whether attached passages actually support their claims. Completeness asks whether the answer covered the supported parts of the question. Abstention precision and recall show whether the system refuses the right cases.
| Metric | Numerator | Denominator | What a poor result suggests |
|---|---|---|---|
| Retrieval recall | Questions where all required passages were retrieved | Answerable labeled questions | Search, filtering, or ranking misses evidence |
| Citation precision | Citations that entail their attached claim | Citations reviewed | Citation mapping or synthesis is unreliable |
| Claim groundedness | Supported material claims | Material claims reviewed | Generation exceeds the supplied evidence |
| Abstention precision | Correct abstentions | All abstentions | The system refuses answerable questions |
| Abstention recall | Unanswerable questions correctly refused | Unanswerable labeled questions | The system answers despite missing evidence |
| Unsupported-answer rate | Answers with at least one unsupported material claim | Answers reviewed for support | At least one control failed before delivery |
Define “required passage,” “material claim,” and “correct abstention” in the annotation guide. Without those definitions, teams can calculate the same metric names from incompatible labels. If the dataset itself needs work, use How to Build an LLM Evaluation Dataset to structure examples and review labels.
Run the suite whenever you change parsing, chunking, embeddings, query rewriting, filters, ranking, generation, prompts, or verification. Compare failure slices across repeated runs rather than trusting one clean example. The machine-gate design guide shows how to turn these expectations into blocking tests.
7. Monitor unsupported-answer rate
Production monitoring should preserve the evidence trail without retaining unnecessary user or source data. Sample interactions and record query class, corpus version, passage IDs, decision state, validator failures, and user feedback.
Make unsupported-answer rate a primary metric:
unsupported-answer rate =
answers containing at least one unsupported material claim
÷ answers reviewed for supportReport the denominator and review method. A rate based only on user complaints is not comparable to one based on systematic sampling. Track the metric by query type, source, language, and pipeline version; an overall average can hide a failing slice.
Pair it with retrieval miss rate, citation precision, abstention rate, false-abstention rate, conflict rate, and verifier failure rate. A falling unsupported-answer rate is not automatically good if the system now refuses every difficult question. Likewise, a low abstention rate can conceal unsupported answers.
When a failure appears, add it to the regression set and assign it to the earliest responsible stage: source, ingestion, retrieval, context construction, or verification.
Frequently asked questions
Does RAG eliminate hallucinations?
No. RAG supplies external evidence, but retrieval can miss the right source and generation can still distort a retrieved passage. Claim-level citation checks and an abstention path address failures that retrieval alone does not.
What similarity-score threshold prevents RAG hallucinations?
There is no universal threshold. Scores vary by embedding model, index, query type, corpus, and reranker. Label representative queries, measure retrieval recall and unsupported answers, then select thresholds for each risk class.
Should a RAG system answer when only part of the question is supported?
Use a partial state only when the supported and unsupported parts can be separated clearly. Identify the missing evidence and do not infer it from model memory. For high-consequence workflows, route partial evidence to review or abstain.
Can an LLM evaluate its own groundedness?
It can be one signal, but not the only gate. Check citation IDs, quotations, dates, and units deterministically; require the judge to identify a supporting span; and compare judge decisions with human labels from the deployment domain.
How should prompt-injection attempts in retrieved documents be handled?
Treat retrieved text as evidence, not as instructions. Preserve trust boundaries, test adversarial documents, and prevent source content from overriding system rules. See How to Test Prompt Injection in RAG Systems for a focused test plan.
The goal is not to make the model sound cautious. It is to make the system auditable: every material claim has inspectable support, missing evidence produces an explicit state, and each escaped failure becomes a test. That turns “reduce hallucinations” from a prompt-writing aspiration into an engineering control.