A retrieval-augmented generation system can produce a fluent wrong answer for two very different reasons: the retriever supplied the wrong evidence, or the language model mishandled good evidence. Testing only the final answer hides that distinction. A more useful evaluation starts by freezing generation, inspecting the ranked retrieval results, and measuring whether the evidence needed to answer each question appears early enough.
This guide builds a retrieval-only test from queries, relevance labels, and recorded search results. It covers recall, precision, mean reciprocal rank (MRR), normalized discounted cumulative gain (NDCG), metadata filters, unanswerable questions, failure analysis, and a release gate. It does not prescribe a universal score threshold. Your acceptable score depends on the consequences of missing evidence, the amount of context the generator can consume, and the document types in your corpus.
Separate retrieval from generation
Evaluate the retriever as a function with a narrow contract:
query + permitted metadata -> ranked evidence unitsAn evidence unit may be a chunk, section, page, record, or complete document. Use one unit consistently. If annotators label documents while the system returns overlapping chunks, one relevant page can appear as several “correct” hits and distort precision. Preserve stable document and chunk IDs.
For the retrieval run, record at least:
- test-case ID and exact query text;
- corpus, index, embedding, chunking, and reranker versions;
- access-control and business filters applied;
- ordered result IDs, ranks, scores, and source document IDs;
- the configured
k, meaning the maximum number of returned results; - latency and any timeout, empty-result, or fallback event.
Do not pass retrieved text to the LLM during this test. The question is narrower: did the system return the labeled evidence, in a useful order, under the correct permissions?
Keep a later test for generation quality. Once retrieval passes, feed the captured evidence to the generator and evaluate groundedness, completeness, refusal behavior, and answer format. Microsoft’s official RAG evaluator documentation likewise separates document retrieval from groundedness and response evaluation (official RAG evaluator documentation).
This separation is not specific to one platform. Amazon Bedrock documents distinct retrieve only and retrieve and generate RAG evaluation modes, and its retrieve-only evaluation can accept recorded output from an external RAG system (official RAG evaluation documentation, checked 2026-07-24).
If your evaluation needs to block deployment rather than merely create a dashboard, connect these measurements to a self-hosted evaluation gate.
Build answerable test questions
Start with real information needs, not paraphrases created because one chunk contains a convenient sentence. Sources include anonymized support queries, search logs, task transcripts, expert interviews, and recurring procedure questions. Remove confidential content before it enters the dataset.
Each positive test case should contain:
id: policy-017
query: "When does the contractor access review expire?"
expected_scope: "security-policy"
answerable: true
relevant_evidence:
- document_id: policy-2026-04
section_id: access-reviewInclude direct lookups, questions requiring two passages, terminology mismatches, abbreviations, date-sensitive policies, and answers found in tables. Include misspellings only if production should handle them.
Split cases by source before tuning. Keep questions from one policy family in either development or holdout data so near-duplicate wording does not inflate results. Tune on development data; run holdout data after choices are fixed.
Review synthetic questions: generators often copy source vocabulary and produce easier queries than users ask. Confirm that each question is natural, answerable, and supported by its label.
Chunk boundaries affect whether answerable evidence can be retrieved at all. Before evaluating a major chunking change, use the RAG chunk-size guide to estimate how the new boundaries and overlap alter the evidence units.
Label relevant evidence
Relevance labels are the ground truth for retrieval metrics, so their definition must be written before annotation begins. A practical graded scale is:
0 — irrelevant: does not help answer the query;1 — related: discusses the topic but cannot support the answer;2 — useful: supports part of the answer or provides necessary context;3 — sufficient: directly supports the answer without requiring an unsupported inference.
Binary precision and recall can treat grades 2 and 3 as relevant. NDCG can retain all grades and reward systems that rank sufficient evidence above merely related text. Record the threshold in the dataset version; changing it changes the meaning of every score.
Annotators should judge evidence, not search scores or writing quality. Give them the query, candidate passage, enough neighboring context, and the rubric. For multi-evidence questions, label each necessary passage.
To reduce annotation cost, pool top results from several configurations, merge duplicate IDs, and label the pool. Track unjudged separately from irrelevant; if many top results are unjudged, expand the pool.
Double-label a sample and adjudicate disagreements to expose ambiguous questions and rubric gaps. Rewrite cases where experts cannot agree on sufficient evidence.
Version the query, labels, corpus snapshot, and annotation rubric together. A document revision can invalidate an old relevance judgment even when the query text has not changed.
Calculate recall, precision, MRR, and NDCG
The metric definitions and formulas in this section were checked on 2026-07-23 against Microsoft’s official information-retrieval guidance, Elasticsearch’s official ranking-evaluation reference, and the U.S. National Institute of Standards and Technology’s TREC evaluation-measures reference.
Calculate every metric at a stated cutoff such as @3, @5, or @10. A score without k is incomplete because the generator usually receives only a limited number of results.
Precision@k asks how much of the returned context is relevant:
Precision@k = relevant results in top k / results examined in top kIf two of five returned chunks are relevant, precision@5 is 2/5 = 0.40. Precision exposes context pollution, but it does not reveal whether other necessary evidence was missed.
Recall@k asks how much known relevant evidence was recovered:
Recall@k = relevant results in top k / all known relevant resultsIf three of four labeled passages appear in the top ten, recall@10 is 3/4 = 0.75. Call it recall against the labeled pool, not proof that no relevant evidence exists elsewhere.
Reciprocal rank rewards placing the first relevant result early:
Reciprocal rank = 1 / rank of first relevant result
MRR = mean reciprocal rank across queriesRank 1 scores 1.0; rank 4 scores 0.25; no relevant result scores 0. MRR suits single-passage questions but can hide missing secondary evidence.
NDCG@k evaluates graded relevance and ranking. First calculate discounted cumulative gain:
DCG@k = sum from rank i=1..k of (2^relevance_i - 1) / log2(i + 1)
NDCG@k = DCG@k / ideal DCG@kThe ideal score sorts labels by descending relevance, making 1.0 a perfect order. Because NDCG depends on the grade scale and gain formula, publish both with the score.
Use the metric that matches the failure you need to detect:
| Metric | Best used for | Main blind spot |
|---|---|---|
| Precision@k | Limiting irrelevant context in the top k | Does not show relevant evidence missed below k |
| Recall@k | Checking whether known evidence was recovered | Treats rank 1 and rank k equally |
| MRR | Single-answer tasks where the first useful hit matters | Ignores additional relevant passages after the first |
| NDCG@k | Graded relevance and ordering quality | Depends on a consistent grading and gain scheme |
| Zero-hit rate | Finding answerable queries with no relevant hit | Does not distinguish near misses from unrelated results |
Report a distribution, not only a mean: median, lower percentile, zero-hit rate, and scores by query class. Twenty easy lookups can conceal a complete failure on five permission-sensitive policy queries.
These label-based ranking metrics and LLM-judged context metrics answer different questions. For example, Amazon Bedrock’s retrieve-only evaluation documents ContextRelevance and ground-truth-dependent ContextCoverage as built-in metrics (official metric documentation, checked 2026-07-24). Keep deterministic rank metrics when you need reproducible regression gates; add judged metrics only when their extra semantic coverage justifies evaluator cost and variability.
Test filters and no-answer cases
Retrieval quality includes returning nothing when nothing permitted can answer the question. Add negative cases for nonexistent entities, unsupported time ranges, requests outside the corpus, and plausible questions whose evidence has been removed or expired.
Specify expected behavior for every negative case: zero results, results below a calibrated threshold, or an insufficient_evidence state. Similarity scores are system-specific signals, not universal probabilities, so calibrate them on your labels.
Test filters as part of the query contract:
- a user with permission retrieves the relevant document;
- a user without permission never receives its ID, snippet, or metadata;
- tenant, language, region, product, and effective-date filters select the intended scope;
- a missing optional filter does not broaden access unexpectedly;
- an empty or conflicting filter returns a controlled empty result.
Maintain separate security assertions. A retriever that finds highly relevant but unauthorized evidence has failed even if its recall appears excellent. Treat leakage as a hard failure, not a small penalty averaged into NDCG.
Test stale documents. If current and archived procedures both match, reward the current source and demote or exclude the archive. Record effective dates so updates can trigger reevaluation.
Diagnose failures by document type
Aggregate metrics tell you that quality changed; slices tell you what to fix. Partition results by document type, query type, language, source system, answer location, update age, and required number of evidence units.
Common patterns have different remedies:
| Failure pattern | Likely retrieval cause | Experiment to isolate it |
|---|---|---|
| Headings rank, but their details do not | Chunks separate labels from content | Attach heading context or change chunk boundaries |
| Exact identifiers disappear | Semantic retrieval underweights literal tokens | Add keyword or hybrid retrieval |
| Tables fail while prose succeeds | Row and header relationships were lost | Index normalized table rows with headers |
| Current and archived policies mix | Date or lifecycle metadata is missing | Add effective-state filters and reindex |
| Multi-part questions find only one passage | k is too small or query is not decomposed | Increase candidate depth or test query decomposition |
| One large manual dominates results | Near-duplicate chunks crowd the ranking | Deduplicate by document or diversify results |
Change one variable at a time when possible: chunking, embedding, query rewrite, candidate depth, filters, hybrid weights, or reranking. Save the configuration and compare it on the same frozen dataset.
Use the broader machine-gate design guide when a failure class should become a deterministic check rather than another dashboard metric.
Set a release gate
A release gate should combine quality, safety, latency, and reproducibility. Define it before the candidate run so the team cannot lower the bar after seeing an inconvenient result.
A compact gate might require:
- no unauthorized document or metadata exposure in any security case;
- no regression beyond the agreed tolerance for recall@k and NDCG@k on the frozen holdout set;
- a minimum score for critical slices rather than only the overall mean;
- a bounded zero-hit rate for answerable questions;
- expected empty or insufficient-evidence behavior for negative cases;
- latency within the application budget at the tested concurrency;
- a saved dataset version, corpus snapshot, configuration, raw ranked results, and evaluation script version.
Choose thresholds from the baseline and failure cost. A compliance assistant may prioritize recall; narrow support search may prioritize early precision. Use separate slice thresholds, and never let relevance offset a security failure.
When a candidate fails, emit its case IDs and slice. A mislabeled case is a dataset defect; a missing permitted document is an ingestion defect; a low-ranked correct chunk is a ranking defect. Assign them accordingly.
Only after retrieval passes should you evaluate the LLM on a fixed set of captured contexts. That sequencing turns “the RAG answer got worse” into a smaller, actionable question: did evidence retrieval regress, or did generation regress while evidence remained stable?
Frequently asked questions
How many queries do I need for a RAG retrieval evaluation?
There is no universal minimum. Start with enough cases to cover every critical query and document class, then report confidence intervals or raw counts alongside averages. A small, balanced set is useful for early iteration; a larger frozen holdout is safer for a release decision.
What is a good recall@k score for RAG?
A good threshold depends on the cost of missing evidence and the generator’s context budget. Establish a baseline on labeled production-like queries, set stricter thresholds for high-risk slices, and require no material regression. Do not copy a threshold from a different corpus.
Should I use recall@5 or recall@10?
Use the cutoff that matches the number of evidence units the generator actually receives. If retrieval returns 20 candidates but reranking passes only 5 to the LLM, report candidate recall@20 and final recall@5 separately.
Can an LLM judge replace relevance labels?
Not completely. An LLM judge can expand coverage, but its output can vary with the model and prompt. Keep a human-labeled holdout for release gates, version the judge configuration, and manually review disagreements or score changes near the threshold.
Should retrieval evaluation include reranking?
Evaluate both stages when a reranker is used. Measure candidate recall before reranking to test whether the relevant evidence entered the pool, then measure precision, MRR, or NDCG after reranking to test the final order. The RAG reranking guide and hybrid search guide cover the two common ways to improve those stages.
Related guides
- Build an LLM evaluation dataset for dataset versioning and holdout design.
- Prevent RAG hallucinations for the generation checks that follow retrieval evaluation.
- Test prompt injection in RAG for adversarial cases that relevance metrics alone do not cover.
Primary sources
All sources below were checked on 2026-07-24: