A reranker can improve a retrieval-augmented generation system when the first-stage retriever finds the right evidence but places it below weaker passages. It cannot recover evidence that retrieval never found. The pattern is: retrieve a broad candidate set, score it more carefully, and send only the strongest passages to the generator.
Treat reranking as an engineering change with a measurable quality-versus-latency tradeoff. Compare the complete pipeline with and without it, using labeled queries and realistic traffic.
Technical and product facts in this article were checked against official documentation and primary research on July 23, 2026. Provider behavior can change, so verify model limits and billing before implementation.
What reranking changes
Most retrieval pipelines separate candidate generation from candidate ordering. A lexical, vector, or hybrid retriever searches the corpus efficiently. A reranker applies a more expensive relevance model only to those candidates.
This division matters because retrieval and reranking solve different problems:
- The retriever is responsible for recall: relevant evidence must enter the candidate set.
- The reranker is responsible for ordering: the most useful evidence should rise toward the positions the generator will actually receive.
- The generator answers from selected context; it should not be expected to repair a failed search.
Sentence Transformers describes CrossEncoder models as slower than bi-encoders because they compute a score for each query-document pair, which is why they are commonly applied only to top-k results. Its retrieve-and-rerank example uses an efficient retriever first and a CrossEncoder second. Sources checked July 23, 2026: CrossEncoder usage and retrieve-and-rerank documentation.
If a relevant passage is ranked 18th and the generator receives five passages, reranking the first 40 may promote it. If it is absent from all 40, the reranker has nothing to promote. Check candidate recall first. Chunk boundaries, filters, query rewriting, embeddings, or retrieval may be the real constraint. Our RAG chunk size guide covers one source of missing evidence. Log both ranks with stable document and chunk IDs.
Choose cross-encoder or LLM reranking
A cross-encoder reads a query and candidate together and returns a relevance score. It is a strong default when relevance means “Does this passage answer or support this query?”
Choose one for predictable pairwise scoring, batchable inference, and a narrow task. Do not assume a public benchmark transfers to internal tickets, policies, code, or tables. Sentence Transformers notes that fine-tuning can be important so a reranker does not reduce performance on a particular use case. Source checked July 23, 2026: CrossEncoder training overview.
An LLM reranker can follow a rubric: prefer current policy, require a jurisdictional match, or reject passages that mention terms without supporting the claim. It can score candidates independently, compare pairs, or order a list. That flexibility adds prompt sensitivity, token use, and latency.
Use one when the relevance rule needs instructions or reasoning a tested cross-encoder misses. Require structured output with candidate IDs and scores or an ordered list. Reject unknown, duplicated, or omitted IDs; never let it rewrite evidence while ranking.
Managed APIs provide reranking without model hosting. Cohere’s v2 endpoint accepts a query and documents and returns ordered relevance scores. Its API reference warns that long documents may be truncated according to max_tokens_per_doc and recommends against more than 1,000 documents per request. Source checked July 23, 2026: Cohere Rerank v2 API reference. These are provider-specific facts, not general limits.
Late-interaction models offer another point on the quality-and-latency spectrum. ColBERT encodes queries and documents separately, then performs a token-level interaction; document representations can therefore be computed before query time. That differs from a cross-encoder, which jointly scores every query-passage pair. The original paper reports its own dataset-specific efficiency and effectiveness results, which should not be treated as production guarantees. Source checked July 24, 2026: ColBERT primary paper.
Reranking option summary
| Option | What it scores | Main advantage | Main cost or risk | Use it when |
|---|---|---|---|---|
| Retriever only | Query against the search index | Lowest complexity and no second-stage latency | Weak candidates may stay above better evidence | The baseline already clears the quality gate |
| Cross-encoder | Each query-passage pair jointly | Focused relevance scoring and batchable inference | Work rises with candidate count and passage length | Relevance is narrow and candidate k is controlled |
| Late interaction | Precomputed document token representations against query tokens | Moves part of the representation work before query time | More indexing, storage, and serving complexity | Cross-encoder latency is too high and the corpus justifies a specialized index |
| LLM reranker | Candidates against an explicit rubric or listwise prompt | Can apply nuanced, domain-specific instructions | Prompt sensitivity, token cost, and variable latency | Relevance depends on rules that simpler models miss |
| Managed rerank API | Provider-hosted scoring model | No model-serving infrastructure | Provider limits, truncation, data handling, and billing | Operational simplicity outweighs external-service constraints |
Pinecone’s current rerank endpoint illustrates common managed-API controls: the request supplies a model, query, documents, optional top_n, fields to rank, and model-specific parameters; the response includes reranked data and usage statistics. Treat the exact schema and hosted models as provider-specific and versioned. Source checked July 24, 2026: Pinecone Rerank API reference.
Set candidate and final top-k values
There are two different top-k values:
- Candidate k is how many results the first stage sends to the reranker.
- Final k is how many reranked passages continue to context assembly.
Set candidate k from recall curves. For each labeled query, check whether relevant evidence appears within the first 5, 10, 20, 40, and 80 candidates. If recall stops improving after 40, scoring 80 adds work without evidence. If recall keeps climbing, broaden or improve retrieval.
A starting experiment might compare candidate k values of 20, 40, and 80 and final k values of 3, 5, and 8. These are test points, not universal settings. Keep the corpus, retriever, filters, and queries fixed.
Final k should reflect evidence coverage and usable context, not the advertised context window. Too few passages omit evidence; too many add duplicates, contradictions, and noise. Deduplicate and consider source diversity after scoring.
Measure whether evidence was available at candidate k and survived final k. The failures require different fixes. Consider storage and query costs with our vector database pricing guide.
If candidate recall is weak across both exact terms and paraphrases, test hybrid search with BM25 and vectors before increasing reranking work. Reranking can reorder a better candidate pool, but it does not replace candidate generation.
Control latency and token cost
Reranking work grows with candidate count and length. Measure p50, p95, and p99 latency, timeouts, and candidates scored. For APIs, log input size and billable units; for self-hosting, measure utilization, batch wait, throughput, and memory.
Control the cost in this order:
- Reduce bad candidates upstream. Apply access and tenant filters before reranking, never after it.
- Send focused passages. Prefer a clean chunk with a title and essential metadata to a document full of boilerplate.
- Batch pair scoring. Cross-encoders generally benefit from batching, but batch wait must fit the latency budget.
- Cap candidate k. Use the smallest candidate set that preserves the required recall.
- Cache carefully. Include query normalization, corpus version, permissions, model, and configuration in the key.
- Define a timeout fallback. Return the original order or use a cheaper reranker, and log the fallback.
Document length behavior deserves explicit testing. Cohere cautions that relevance scores should not be interpreted as ratio measurements—a score of 0.9 is not necessarily “twice as relevant” as 0.45. Its current best-practices page also describes provider-managed document chunking. Source checked July 23, 2026: Cohere reranking best practices. Use scores for ordering and calibrated thresholds only after validating them on your own data.
Run retriever-only and reranked variants under the same concurrency, timeout, cache state, region, and hardware or service tier.
Create a reranking test set
Build the test set from real information needs plus stress cases. Record the query, relevant chunk IDs, optional grades, metadata constraints, and a labeling note. Freeze the corpus snapshot or store content hashes.
Include:
- Direct fact lookups with one sufficient passage
- Questions requiring two or more supporting passages
- Short and underspecified queries
- Exact identifiers, error codes, names, and version strings
- Paraphrases that share few keywords with the answer
- Near-duplicate documents with different dates or authority
- Queries with no answer in the corpus
- Permission-sensitive or historically difficult queries
Label evidence, not merely answers. A passage can contain the right words without supporting the claim. Calibrate reviewers on a small slice and write down the relevance rule.
Mine hard negatives from actual results: high-ranked candidates that are wrong because of a shared term, stale version, or missing condition. Keep a held-out set so tuning does not become memorization.
The BEIR paper evaluated several retrieval architectures across heterogeneous datasets; its reranking gains also came with higher computational cost. Source checked July 23, 2026: BEIR primary paper. Use benchmarks to shortlist approaches, then decide on your labeled distribution.
For a reusable workflow beyond reranking, see how to build an LLM evaluation dataset from real failures.
Measure gains with retrieval metrics
Evaluate the same candidates before and after reranking. Report overall results and slices by query type, document, language, tenant, and answerability.
Use metrics that match the job:
- Recall@candidate-k asks whether relevant evidence entered the reranker. A reranker cannot improve this value when the candidate set is fixed.
- Recall@final-k asks whether relevant evidence survived into context.
- MRR@k emphasizes the rank of the first relevant result. NIST’s TREC explanation defines reciprocal rank as the inverse of the first correct result’s rank, or zero when none is returned, averaged across questions. Source checked July 23, 2026: TREC QA evaluation description.
- NDCG@k is useful when passages have graded relevance and the order of several good results matters.
- Precision@k shows how much of the limited context is relevant.
Add p95 latency, timeout and fallback rates, cost per 1,000 queries, context tokens, and duplicate rate. Then evaluate downstream answers: the goal is supported answers, not prettier rankings.
Define a release gate before seeing results: no credible regression in a critical class, a specified retrieval gain, and latency and cost within budget. Preserve per-query deltas because averages can hide damaged exact-ID lookups.
Our guides to evaluating RAG retrieval before testing the LLM and building a self-hosted evaluation gate provide broader patterns for turning test results into a repeatable release decision.
Know when reranking is not worth it
Do not add reranking because it is common in RAG diagrams. It is the wrong next step when recall is poor, filters exclude answers, chunks split essential context, or the corpus lacks authoritative evidence.
Reranking may also be unnecessary when:
- The corpus is small enough for a simpler search strategy.
- The first-stage ranking already saturates the relevant metric.
- Queries are mostly exact identifiers that lexical retrieval handles reliably.
- The added p95 latency violates the user experience or service objective.
- Relevant candidates are obvious, but generation ignores or misquotes them.
- The team cannot label a representative test set or observe stage-level failures.
- Permission rules cannot be guaranteed before candidates reach an external service.
Run an ablation: retriever only, retriever plus reranker, and improved retrieval without reranking. Compare quality, latency, cost, failures, and maintenance. Prefer the least complex pipeline that clears the gate.
If reranking wins, deploy gradually. Log both rankings, maintain a fallback, watch query segments, and retest after changes to embeddings, chunking, filters, corpus, or model. The goal is a pipeline where each stage has a defined responsibility, measurable benefit, and safe failure behavior.
Frequently asked questions
What is a good candidate k for RAG reranking?
There is no universal value. Plot recall at several depths on labeled queries, then choose the smallest candidate k that preserves the evidence you need within the latency and cost budget. Values such as 20, 40, and 80 are experiment points, not recommendations.
How do I know whether a reranker is actually helping?
Compare retriever-only and reranked pipelines on the same frozen candidates and labels. Check Recall@final-k, MRR or NDCG, downstream supported-answer quality, p95 latency, timeout rate, and cost. Review per-query regressions as well as averages.
Should I use a reranker score threshold?
Only after calibrating it on your own labeled data and model version. Provider scores may be useful for ordering without being calibrated probabilities or comparable across models. When evidence is insufficient, pair retrieval checks with an explicit abstention policy; see how to reduce RAG hallucinations with evidence and abstention.
Does reranking replace hybrid search?
No. Hybrid search expands or improves the candidate pool by combining retrieval signals; reranking orders the candidates it receives. They can be used together, and candidate recall should be measured before attributing failures to the reranker.
What should happen if the reranker times out?
Use a predefined fallback, such as the original retrieval order, and log that the fallback occurred. Apply permissions before any external reranking call, keep stable candidate IDs, and monitor whether fallback traffic changes answer quality.