The right RAG chunk size is not a universal number. It is the smallest passage that preserves enough context to answer your real questions without burying the useful sentence in unrelated text. For many English prose collections, 400 to 800 tokens with 10% to 20% overlap is a defensible first experiment, not a production rule. Structured API references may need smaller, boundary-aware chunks; narrative reports may need larger sections.
A RAG chunk size calculator turns that starting hypothesis into operational numbers: total chunks, duplicated embedding tokens, estimated vector and metadata storage, and retrieved context per query. Use those numbers to reject obviously expensive settings. Then choose among the remaining settings with an evaluation set built from your own documents.
Specifications in this article were checked against official documentation on 2026-07-24. Provider limits can change, so recheck them before implementation.
What chunk size changes
Chunk size creates a three-way trade-off among retrieval precision, context preservation, and system cost.
Smaller chunks usually isolate individual facts more cleanly. A query for a single configuration flag is less likely to retrieve an entire chapter. However, a small chunk can separate a qualification from the sentence it modifies, split a table from its heading, or return a function without its surrounding contract. It also creates more records, metadata copies, embedding requests, and candidates for the retriever to rank.
Larger chunks keep more local context together. That helps when an answer depends on several adjacent paragraphs, but a broad chunk can match a query because of one phrase while carrying substantial irrelevant text into the generation prompt. Large chunks also reduce the number of distinct passages the model can inspect within a fixed context budget.
Overlap is a partial safeguard, not a substitute for structural splitting. Repeating text across boundaries can preserve a sentence that lands at the edge of a fixed token window. It cannot reliably repair a split table, code block, list, or section hierarchy. Prefer headings, paragraphs, list items, and function boundaries first; use token limits as the fallback that prevents pathological chunk sizes.
This structure-first approach is consistent with Cohere’s official chunking cookbook, which separates the decision about where to split from the decision about how large to make a chunk. The cookbook also notes that overlap can recover boundary context but introduces redundant content (Cohere chunking strategies, checked 2026-07-24). Treat that as implementation guidance and test it on the document structures in your own corpus.
The practical target is therefore not “maximum recall at any cost.” It is enough retrieval coverage to answer supported questions while keeping evidence focused, traceable, and affordable.
Inputs the calculator uses
The RAG Chunk & Storage Estimator needs six core inputs and one optional retrieval input:
- Total document tokens: the corpus size after cleaning and before overlap is added. If only words are available, treat the token conversion as an estimate and validate it with the tokenizer used by the embedding provider.
- Chunk size in tokens: the maximum token budget for each passage after structural splitting.
- Overlap in tokens: text repeated between adjacent chunks. It must be smaller than the chunk size.
- Embedding dimensions: the number of numeric values in each stored vector.
- Bytes per dimension: use 4 for ordinary 32-bit floating-point vectors unless the actual index uses another representation.
- Metadata bytes per chunk: include identifiers, source paths, headings, timestamps, access-control fields, and any stored text that the database counts as metadata.
- Replication multiplier: use the storage multiplier documented for the deployed architecture, or 1 when estimating raw unreplicated data.
For query planning, also enter the number of chunks retrieved per query. Multiplying that value by chunk size gives a conservative upper estimate of retrieved passage tokens before prompts, citations, reranking output, or deduplication.
The calculator is provider-independent. It estimates logical volume rather than applying a vendor price. That separation matters because a database may bill vector storage, metadata, writes, reads, replicas, or compute differently. If a future tool accepts a unit price, treat the result as a scenario based on that input, not as a live quote.
Chunk-count and storage formulas
For fixed-size token windows, define the forward movement between chunks as:
step = chunk size - overlap
Then estimate the number of chunks as:
chunks = max(1, ceil((total tokens - overlap) / step))
This formula handles the first chunk once and advances each later chunk by the non-overlapping step. It is an estimate because boundary-aware splitting produces variable lengths and short final chunks.
Suppose a corpus contains 1,000,000 tokens. With 600-token chunks and 90-token overlap:
- Step:
600 - 90 = 510 tokens - Estimated chunks:
ceil((1,000,000 - 90) / 510) = 1,961 - Approximate embedded tokens: first chunk plus later steps and repeated overlap, or roughly
1,961 × 600 = 1,176,600before accounting for a short final chunk - Approximate overlap overhead: about 176,600 tokens, or 17.7% above the original corpus
Raw dense-vector storage is:
vector bytes = chunks × dimensions × bytes per dimension
At 1,536 dimensions and 4 bytes per dimension, 1,961 vectors use 12,048,384 raw bytes, about 11.49 MiB. If metadata averages 1,200 bytes per chunk, add 2,353,200 bytes, about 2.24 MiB. A replication multiplier of 2 produces a simple planning estimate of about 27.46 MiB. This is not a billing estimate: indexes, quantization, graph structures, logs, backups, and provider-specific overhead can change physical consumption.
For comparison, Qdrant’s official capacity-planning guide estimates in-memory vector size as number of vectors × dimensions × 4 bytes × 1.5; the extra 50% is intended to cover metadata, indexes, point versions, and temporary optimization segments (Qdrant capacity planning, checked 2026-07-24). Do not add that factor automatically to every provider: use it as a Qdrant planning reference and replace it with measurements from the actual index configuration.
If a query retrieves eight 600-token chunks, the conservative passage budget is 4,800 tokens. That figure should be compared with the generator’s available input budget after reserving space for instructions, conversation history, citations, and output.
Starting points by document type
The following ranges are experiment seeds. They are not benchmark winners, standards, or guarantees.
| Document type | Starting chunk size | Starting overlap | Boundary to preserve | Main risk to test |
|---|---|---|---|---|
| FAQs and support answers | 150–350 tokens | 0–40 tokens | One question and its complete answer | Combining unrelated answers |
| API and product documentation | 250–500 tokens | 25–75 tokens | Heading, endpoint, parameter list, or procedure | Losing definitions inherited from a parent section |
| Source code | 200–600 tokens | 0–80 tokens | Function, class, module, or symbol | Splitting signatures from behavior and tests |
| Policies and contracts | 400–800 tokens | 50–150 tokens | Clause and referenced qualification | Retrieving a rule without its exception |
| Research papers and technical reports | 500–900 tokens | 75–150 tokens | Section, experiment, or argument | Mixing methods, results, and limitations |
| Long narrative or meeting transcripts | 600–1,000 tokens | 100–200 tokens | Topic or speaker turn group | Broad chunks matching several unrelated topics |
Start with one row, then create a smaller and larger variant—roughly 25% in each direction. For a 600-token baseline, compare about 450, 600, and 750 tokens while holding the embedding model, retrieval method, reranker, prompt, and evaluation questions constant.
Provider defaults are useful compatibility facts but should not be copied as universal guidance. As checked on 2026-07-23, OpenAI’s official vector-store API documentation says its automatic strategy currently uses 800-token maximum chunks with 400-token overlap; its static strategy permits maximum chunk sizes from 100 to 4,096 tokens and limits overlap to no more than half the chunk size (official OpenAI vector-store reference). That 800/400 setting describes one product’s current behavior, not a general retrieval benchmark.
Embedding input limits are also model-specific. Google’s official Vertex AI documentation currently lists a 2,048-token maximum sequence length for gemini-embedding-001, text-embedding-005, and text-multilingual-embedding-002. The same page states a maximum of 250 input texts and 20,000 input tokens per request, with individual text beyond 2,048 tokens truncated unless truncation is disabled (official Vertex AI embedding guide). Check the exact model and endpoint instead of assuming that the generator’s context window is also the embedding limit.
Overlap cost versus recall
For a large corpus with mostly full chunks, overlap overhead is approximately:
overlap overhead ratio ≈ overlap / (chunk size - overlap)
With a 600-token chunk and 60-token overlap, the ratio is about 11.1%. At 120-token overlap it rises to 25%. At 300-token overlap it approaches 100%, meaning nearly twice as many embedded tokens as a non-overlapping pass. The exact total differs at document boundaries, where overlap should usually reset instead of carrying text from one document into another.
More overlap may improve boundary-case recall, but it can also return near-duplicate chunks. That duplication consumes retrieval slots and generator context without adding independent evidence. Measure both outcomes:
- How often does the system retrieve at least one passage containing the required evidence?
- How many of the top results repeat substantially the same text?
- Does deduplication remove a necessary qualification?
- How much embedded and retrieved context grows for each improvement in answer quality?
Metadata can become a constraint before vector bytes do. As checked on 2026-07-23, Pinecone’s official upsert documentation lists a 2 MB maximum request, up to 1,000 vector records per upsert, a 40 KB total metadata limit per document, a 512-character record-ID limit, and a maximum dense-vector dimensionality of 20,000 (official Pinecone upsert limits). These are Pinecone-specific current limits, not portable assumptions. Storing the full chunk text and extensive permissions data in every record can therefore matter as much as vector dimensions.
For a broader provider decision after sizing the corpus, see Pinecone vs. Turbopuffer for RAG.
Evaluation protocol
A calculator narrows the search space; it does not select the winner. Use a repeatable evaluation protocol:
- Build a question set from real tasks. Include direct lookups, multi-sentence explanations, exceptions, table questions, code questions, and questions that the corpus cannot answer.
- Label the required evidence. Record the document and passage needed for each answer. If several passages are acceptable, label all of them.
- Freeze variables outside chunking. Keep the corpus snapshot, embedding model, retrieval algorithm, top-k, filters, reranker, generator, and prompt fixed.
- Test three sizes and at least two overlap settings. A useful first grid is the baseline, minus 25%, and plus 25%, each with low and moderate overlap.
- Measure retrieval before generation. Track evidence hit rate at k, ranking position, duplicate rate, and retrieved tokens. Generation scores alone can hide retrieval failures because a model may answer from prior knowledge.
- Apply an answer-quality gate. Check factual support, citation correctness, completeness, and appropriate refusal when evidence is absent. The self-hosted evaluation gate offers a framework for keeping acceptance criteria under your control.
- Inspect failures, then segment the corpus. One global size may be inferior to separate policies for code, tables, FAQs, and narrative documents.
Report the corpus version, splitter version, tokenizer, model identifiers, parameters, evaluation-set size, and date. Avoid presenting a small internal test as a universal benchmark. If output must pass automated acceptance checks, connect the retrieval evaluation to machine gates for AI output.
Choose the smallest configuration that meets the evidence and answer-quality gates with acceptable cost and latency. If two settings perform similarly, prefer the simpler one and keep the evaluation set for regression testing.
Failure modes to check before production
Counting words as tokens. Word-to-token ratios vary by language, code, formatting, and tokenizer. Use estimated words only for early capacity planning.
Splitting before cleaning. Repeated headers, navigation, OCR artifacts, and boilerplate can dominate embeddings and create misleading matches.
Ignoring structure. Fixed windows can detach headings, table labels, citations, code signatures, and policy exceptions. Parse structure where practical and enforce a token ceiling afterward.
Overlapping across documents. Never join the end of one source to the beginning of another. Reset overlap at document, permission, and version boundaries.
Changing several variables at once. A new embedding model, reranker, prompt, and chunk policy in one test makes the result difficult to attribute.
Optimizing only hit rate. High recall with heavy duplication may reduce the diversity of evidence available to the generator.
Treating vendor limits as targets. A maximum accepted record size says what the API permits, not what retrieval quality needs. Recheck official limits at deployment and keep them separate from evaluated chunk settings.
Losing traceability. Each chunk should retain a stable source identifier and enough structural metadata to reconstruct where the text came from. Without that, citation checks and re-indexing become fragile.
Frequently asked questions
What is the best chunk size for RAG?
There is no corpus-independent best size. For English prose, 400–800 tokens is a reasonable first test range; compare a smaller and larger setting against labeled retrieval questions before choosing. Preserve document boundaries even when that produces variable-length chunks.
How much chunk overlap should I use?
Start with 10%–20% when fixed windows may split useful context, and include a zero- or low-overlap variant in the evaluation. Increase overlap only when boundary misses improve enough to justify extra embedding volume and duplicate retrieval results.
Should chunk size be measured in words, characters, or tokens?
Use the embedding model’s tokenizer and measure tokens for the final limit. Words or characters are acceptable for rough planning, but they vary across languages, code, and formatting and can hide truncation at the embedding endpoint.
Is a larger chunk always better for contextual answers?
No. A larger chunk preserves more neighboring text but may dilute the matching evidence and consume more of the generator’s context budget. If an answer needs multiple sections, retrieving several focused chunks or using a reranker may work better; see the RAG reranking guide for the next stage of that decision.
When should different document types use different chunk sizes?
Use separate policies when the corpus has materially different boundaries or question patterns—for example, API endpoints, tables, contract clauses, and transcripts. Validate each policy with the same retrieval metrics, as described in evaluating RAG retrieval, and keep the selected splitter settings versioned.
The calculator’s output is a capacity model. The final chunk policy is an evaluated, versioned configuration tied to a particular corpus and workload.