How to Build Hybrid Search for RAG with BM25 and Vectors

Hybrid search for retrieval-augmented generation runs a lexical query and a vector query over the same eligible chunks, merges the two ranked candidate lists, and sends only the strongest evidence to the generator. BM25 supplies exact-term precision; vector search supplies semantic recall. Reciprocal rank fusion (RRF) is a practical default for combining them because it uses rank positions rather than pretending that their raw scores share a scale.

The implementation is small. The difficult work is preserving stable chunk identities, applying access filters consistently, choosing candidate depths, and proving that hybrid retrieval beats the vector-only baseline on real questions. This guide builds that pipeline without tying it to one database.

Algorithm behavior and product-specific examples in this article were checked against primary research and official Elastic, OpenSearch, and Microsoft documentation through 2026-07-24. No pricing claims are included.

When vector search misses

Dense embeddings are useful when the query and evidence express the same idea with different words. A question about “ending a subscription,” for example, may retrieve a passage that says “cancel your plan.” That semantic matching is the reason vector retrieval is often the first component in a RAG system.

It is not a complete retrieval strategy. Vector search can miss or under-rank passages when the query depends on lexical details such as:

error codes, ticket IDs, product SKUs, function names, and command-line flags;
exact quotations, legal clauses, acronyms, or uncommon proper nouns;
negation or small numeric differences that produce similar embeddings;
newly introduced terminology that the embedding model represents poorly;
short queries whose intent is ambiguous without an exact match.

BM25 complements those weaknesses. It scores lexical matches using term frequency, inverse document frequency, and document-length normalization. Rare query terms can therefore carry more weight than common terms, while repeated words eventually provide diminishing benefit. Elastic documents BM25 as the default similarity in Elasticsearch and Lucene, with k1 = 1.2 for term-frequency saturation and b = 0.75 for length normalization as current defaults (official similarity reference, verified 2026-07-23). Those values are implementation defaults, not universal tuning targets.

Hybrid retrieval is especially useful when a corpus mixes natural-language explanations with structured identifiers. It also gives you a safer migration path: you can retain a known lexical retriever while measuring whether embeddings contribute additional relevant evidence.

Do not add hybrid search merely because two retrievers sound more sophisticated. If almost every production query is an exact identifier lookup, lexical search may already solve the task. If the corpus lacks meaningful searchable text, BM25 adds little. Build a vector-only baseline first, record its misses, and use those misses to justify the second retrieval path.

Build sparse and dense candidate sets

Index one logical chunk as one retrievable unit, and give it a stable chunk_id. Store the same identity, text, source metadata, and authorization metadata in both retrieval paths. If the lexical and vector indexes use different IDs or different chunk boundaries, fusion can neither recognize duplicates nor explain why one passage won.

A minimal chunk record looks like this:

{
  "chunk_id": "handbook:returns:0042",
  "text": "Refund requests must be submitted within...",
  "document_id": "handbook:returns",
  "section": "Refund window",
  "tenant_id": "tenant-17",
  "language": "en",
  "updated_at": "2026-06-18",
  "embedding": [0.012, -0.034, 0.008]
}

The lexical index should analyze prose but preserve fields that require exact matching. Search body text, headings, aliases, and identifier fields deliberately rather than concatenating everything into an opaque blob. The dense index should embed the text that carries the answer, usually with a compact heading or breadcrumb for disambiguation. It should not embed access-control fields and timestamps as if they were prose.

At query time:

Normalize the user query without deleting meaningful punctuation from codes or flags.
Derive the authorization and scope filters from trusted application state.
Run BM25 and vector retrieval independently, preferably in parallel.
Ask each retriever for a candidate pool larger than the final number of chunks.
Return chunk_id, rank, raw score, source metadata, and text from each path.

Microsoft’s current hybrid-search overview describes the same high-level shape: full-text and vector queries run in parallel against an index containing text and embeddings, then their results are merged with RRF (official hybrid-search overview, verified 2026-07-23).

Candidate depth is an evaluation parameter, not a constant copied from a tutorial. If the generator receives eight chunks, each retriever usually needs to retrieve more than eight so fusion has room to rescue evidence that is strong in only one list. Increase the pool until retrieval gains flatten or latency and memory exceed the service budget. If chunking itself is producing fragments that cannot answer a question, fix that upstream with the RAG chunk size workflow before tuning fusion.

Choose the retrieval and fusion strategy

Use the simplest option that fixes measured retrieval failures. The table summarizes what each stage contributes and which parameter should be evaluated rather than assumed.

Component or method	Best fit	Main risk	First parameter to evaluate
BM25 only	Exact identifiers, rare terms, and literal wording	Misses paraphrases and conceptually related wording	Analyzer, field boosts, and lexical candidate depth
Vector only	Paraphrases, conceptual questions, and semantic similarity	Under-ranks codes, names, and small lexical differences	Embedding model, distance metric, and vector candidate depth
RRF hybrid	Heterogeneous retrievers whose score scales are not comparable	Weak candidates can enter when the rank window is too deep	Candidate depth and rank constant; `60` is a common default, not a universal optimum
Normalized weighted hybrid	Labeled data shows one retriever should contribute more	Calibration can drift after corpus, analyzer, or model changes	Normalization method and lexical/vector weights
Hybrid plus reranker	First-stage recall is adequate but top-result ordering remains weak	Adds latency and another model or scoring stage	Rerank depth and end-to-end latency budget

Elasticsearch’s current RRF reference documents a default rank_constant of 60 and states that a larger rank_window_size can improve relevance at a performance cost (official Elasticsearch RRF reference, verified 2026-07-24). These are backend defaults and trade-offs, not evidence that the same values are optimal for a particular corpus.

Normalize incompatible scores

BM25 scores and vector scores do not mean the same thing. A BM25 score depends on term statistics, field boosts, query structure, and document length. A vector engine may return cosine similarity, cosine distance, dot product, Euclidean distance, or a transformed score. Even two queries from the same retriever can have different score distributions.

Therefore, this is unsafe:

hybrid_score = 0.5 * bm25_score + 0.5 * vector_score

The equal weights do not make the inputs comparable. A lexical score range of 0 to 25 can overwhelm a vector range of 0 to 1, while another query may produce a completely different lexical range.

There are two reasonable approaches:

Normalize scores, then combine them. Min-max normalization and L2 normalization can map each list into a comparable range. Weighted arithmetic combination then lets you express a preference for one retriever. This is useful when score magnitudes are calibrated on representative data.
Discard score magnitude and combine ranks. Rank fusion uses where a chunk appears in each list. It is less sensitive to score-scale drift and is a simpler starting point when retrievers are heterogeneous.

OpenSearch’s official normalization processor exists specifically because BM25 and k-nearest-neighbor results use different relevance scales. It supports min-max and L2 normalization plus arithmetic, geometric, and harmonic score combinations (official normalization documentation, verified 2026-07-23).

If you choose score normalization, define behavior for a flat list where every score is equal, negative dot-product values, missing chunks, and outliers. Fit any calibration only on a training or tuning split. Refit it when the embedding model, analyzer, corpus, or index settings change. Otherwise, use RRF as the baseline and add weighting only when labeled evaluation shows a repeatable gain.

Combine results with reciprocal rank fusion

RRF gives each chunk a contribution based on its position in each ranked list:

RRF(chunk) = Σ 1 / (c + rank_i(chunk))

rank_i is the chunk’s one-based position in result list i; a missing chunk contributes nothing from that list. c is a rank constant that reduces the difference between adjacent positions. The original 2009 RRF paper by Cormack, Clarke, and Büttcher used this simple formulation and compared it with other fusion methods (primary paper, verified 2026-07-23). Microsoft’s current implementation documentation uses the same 1 / (rank + k) form and notes 60 as a small constant used in its explanation (official RRF documentation, verified 2026-07-23).

Treat 60 as a starting value, not a requirement. More important decisions are the candidate depth of each list, whether lexical and dense lists receive equal weight, and how ties are resolved.

def reciprocal_rank_fusion(result_lists, rank_constant=60, weights=None):
    weights = weights or [1.0] * len(result_lists)
    fused = {}

    for results, weight in zip(result_lists, weights):
        for rank, hit in enumerate(results, start=1):
            item = fused.setdefault(
                hit["chunk_id"],
                {"chunk": hit, "score": 0.0, "sources": []},
            )
            item["score"] += weight / (rank_constant + rank)
            item["sources"].append({
                "retriever": hit["retriever"],
                "rank": rank,
                "raw_score": hit["score"],
            })

    return sorted(
        fused.values(),
        key=lambda item: (-item["score"], item["chunk"]["chunk_id"]),
    )

Keep the per-retriever ranks and raw scores for debugging even though they do not enter the default fused score. Use a deterministic tie-breaker such as chunk_id. Deduplicate by chunk identity before selecting the final context, and consider limiting near-duplicate chunks from one document so a repeated passage cannot occupy the entire context window.

Weighted RRF can be useful, but a weight should encode measured relevance, not intuition. If exact error codes dominate the workload, a larger lexical weight may help. If users ask paraphrased conceptual questions, the dense list may deserve more influence. Test both query groups separately; one global average can hide a regression in either group.

Add filters without destroying recall

Filters are part of retrieval correctness, not a cleanup step. Tenant boundaries, document permissions, language, product version, and effective dates must apply to both candidate generators. A chunk excluded by authorization must never enter fusion or the generator context.

Where the vector engine applies a filter matters:

Pre-filtering restricts eligible documents during vector search. It protects scope and can preserve filtered recall, but a highly selective predicate may require more graph traversal.
Post-filtering retrieves nearest neighbors first and removes ineligible results afterward. It may be faster or simpler, but it can return too few chunks because relevant eligible items never entered the unfiltered top candidates.

Microsoft’s vector-filter documentation explicitly warns that post-filtering can create false negatives with selective filters or a small candidate count, while its pre-filter mode favors recall at a possible latency and CPU cost (official vector-filter guide, verified 2026-07-23).

Authorization filters should be enforced inside each retrieval query whenever the backend supports it. Do not retrieve across tenants and rely on application-side trimming. For non-security facets, compare pre-filtering with an over-retrieved post-filter approach using the same labeled queries. Record both the number requested and the number remaining after filtering; a system that silently returns two chunks when ten were requested needs an observable warning.

Apply the identical logical filter to BM25 and vectors. Then test empty scopes, deleted documents, version boundaries, mixed-language queries, and users with access to only a tiny fraction of the corpus. These edge cases often reveal recall failures that an unfiltered demo conceals.

Evaluate against vector-only search

Evaluate retrieval before asking an LLM to write answers. Generation quality is too noisy to tell you whether hybrid search supplied stronger evidence.

Build a query set from support logs, search logs, expert questions, and known incidents after removing private data. Label every chunk that contains sufficient evidence for each query. Include at least these query slices:

exact identifiers and rare terms;
paraphrases and conceptual questions;
multi-part questions;
filter-heavy or permission-limited questions;
queries with no answer in the corpus;
recently added terminology.

Run vector-only, BM25-only, and hybrid retrieval against the same index snapshot. Measure recall at the candidate cutoff, mean reciprocal rank for the first relevant chunk, and nDCG when several chunks have graded relevance. Elasticsearch’s official rank-evaluation reference distinguishes these roles: recall at k measures how many relevant results enter the top k, mean reciprocal rank tracks the position of the first relevant result, and normalized DCG accounts for both rank and graded relevance (official ranking-evaluation reference, verified 2026-07-24). Also record the empty-result rate, eligible results remaining after filters, p50 and p95 retrieval latency, and context tokens sent downstream.

Inspect paired failures, not just aggregate scores. Create four buckets: hybrid fixed a vector miss; hybrid harmed a vector success; both missed; both succeeded with a different ranking. The second and third buckets tell you whether to adjust candidate depth, analyzers, embeddings, filters, chunking, or fusion.

Set a release rule before tuning. For example, require hybrid retrieval to improve identifier-query recall without reducing semantic-query recall beyond an agreed tolerance, while staying inside the latency budget. The exact thresholds belong to your application and labeled dataset; they are not portable benchmark numbers.

OpenSearch’s official optimization workflow illustrates why tuning should be dataset-driven: it evaluates normalization methods, three score-combination methods, lexical/vector weights from 0.0 to 1.0 in 0.1 increments, and RRF rank constants of 1, 5, 10, 20, and 60 against a query set and judgment list (official hybrid-search optimization guide, verified 2026-07-24). You do not need to copy that entire search grid, but you do need representative queries and relevance judgments. The RAG retrieval evaluation guide covers how to turn those judgments into a repeatable test set.

If first-stage retrieval has good recall but the best evidence still appears too low, evaluate a second-stage model separately rather than forcing fusion weights to solve every ranking problem. The RAG reranking guide explains that next step.

Frequently asked questions

Does hybrid search always outperform vector search?

No. It is most useful when production queries include both exact lexical details and semantic paraphrases. Compare BM25-only, vector-only, and hybrid results on the same labeled queries; keep the simpler retriever if hybrid does not produce a repeatable gain within the latency budget.

What RRF rank constant should I use?

Start with a documented backend default such as 60, then test alternatives on representative judgments. A larger constant gives lower-ranked candidates relatively more influence, but candidate depth and the quality of each input list can matter as much as the constant itself.

How many candidates should BM25 and vector search retrieve?

Retrieve more candidates per path than the final context needs, then increase the depth until recall gains flatten or the latency budget is exceeded. There is no portable fixed ratio because corpus size, filter selectivity, approximate-nearest-neighbor settings, and chunk quality all affect the result.

Should filters run before or after vector search?

Authorization and tenant filters should be enforced inside both retrieval paths whenever the backend supports it. For non-security filters, pre-filtering generally protects eligible recall, while post-filtering may require over-retrieval and can still return too few eligible chunks.

Do I still need reranking after RRF?

Not necessarily. Add reranking only when evaluation shows that relevant chunks enter the fused candidate set but remain poorly ordered. Measure the quality gain against the added latency, cost, and operational complexity.

Production rollout checklist

Roll out hybrid retrieval behind a versioned configuration and preserve the vector-only path for comparison and rollback.

Shadow traffic is a safe first stage: run hybrid retrieval without changing the answer path, then compare its candidates with the current vector results. Next, expose it to a small traffic segment while monitoring retrieval and answer-support metrics. Only increase the share after both relevance and access-control tests pass.

Hybrid search can increase query work because it executes two retrieval paths and merges them, but the infrastructure cost depends on the backend and workload. Model storage, writes, and query charges with the vector database pricing guide, and compare operational trade-offs in the existing Pinecone versus Turbopuffer RAG review. Keep those purchasing questions separate from the retrieval evaluation: a cheaper system that misses required evidence is not an equivalent design.

The production goal is not to maximize the fused score. It is to return sufficient, authorized, non-duplicative evidence within a predictable latency budget—and to make every regression traceable to a retriever, filter, or fusion decision.