Chunk size for dense retrieval

Dense retrieval embeds text into vectors. Long documents have to be split into chunks, which raises three questions:

1. Is there an optimal chunk size for a given model and vector size?
2. If a document is split into several chunks, how do you rank the document?
3. When is one vector per document enough?

There is no formula for the first. But there are bounds from both sides, and the experiments mostly agree with them. Only result quality is discussed here, not speed or cost.

Context: Text search options, where chunking is described as dense retrieval’s new analysis chain. Metrics (nDCG, recall, MRR) are explained there too.

Units: sizes are given in tokens unless a study used words or characters. In English, one word is roughly 1.3 tokens: OpenAI’s rule of thumb is 100 tokens ≈ 75 words ↗. The exact ratio depends on the tokenizer. The ratio is much higher for code and for morphologically rich languages, so a token target is not language-neutral.

Upper bound: chunks shouldn’t be too long

The context window is a hard limit: text past it is silently cut off. It is 512 tokens for BERT-era models and 8k+ for long-context ones.

The training length is the practical limit, and usually the tighter one. A model embeds best text of about the length it was trained on.

• Most embedding models are fine-tuned on short texts. LongEmbed ↗: “traditional training data for embedding models are short queries and passages not exceeding 512 tokens”.
• Typical training passages are 56–100 words. MS MARCO averages about 56 words (BEIR ↗, Table 1), and DPR split Wikipedia into “disjoint text blocks of 100 words” (DPR ↗).
• A long context window doesn’t mean long texts embed well. jina-embeddings-v2 serves 8,192 tokens but was fine-tuned “solely on texts not exceeding 512 tokens” (paper ↗).
• Some models are trained longer: nomic-embed at 2,048 tokens (paper ↗), and BGE-M3 with long documents up to 8k (paper ↗). For those, the limit moves up.

Quality degrades before that limit:

• Dilution. Length-Induced Embedding Collapse ↗ (ACL 2025): embeddings of long texts drift together and become less distinguishable, because self-attention smooths the representation more as the text gets longer.
  • For example, BGE’s IMDB accuracy falls from 75.6% for 0–100-token texts to 59.0% for 400–500-token texts.
  • This is measured on classification, not retrieval, but the mechanism is the same.
• Margin. The margin is the score gap between a relevant and an irrelevant document. Luan, Eisenstein, Toutanova & Collins ↗ (TACL 2021) show that, for a simple family of encoders (random projections), the dimension sufficient to keep that gap grows linearly with document length. Learned encoders may do better, but longer texts make the job harder.

Upper bound: at most the model’s training length (≈512 tokens for most BERT-era models), with quality degrading before that.

Raw information capacity is not the limit. Kuratov et al. (2025) ↗ fit up to 1,568 tokens into one 4096-d vector and decoded them back with Llama-3.1-8B. But that is reconstruction, with the vector optimized for each text separately. Retrieval needs something different: the relevant chunk must score clearly above every competitor for a short query.

Rerankers can add another upper bound. If a cross-encoder reranks the retrieved chunks, chunks longer than its input limit get cut off. The limits vary widely. Classic BERT-based cross-encoders such as ms-marco-MiniLM ↗, still by far the most downloaded, take 512 tokens. LLM-based rerankers such as Qwen3-Reranker ↗ (0.6B–8B, 2025) take 32k, so for them this bound effectively disappears.

Lower bound: chunks must make sense on their own

The real lower bound is semantic. A chunk has to contain enough context to be matched on its own: a sentence like “It was raised to 40% in 2021” means nothing without knowing what “it” is.

The strongest evidence: small chunks work if the context is put back.

• Dense X Retrieval ↗ (EMNLP 2024) splits text into propositions: single facts that an LLM rewrites to be self-contained, averaging 11 words. They beat 58-word passages for unsupervised retrievers (Contriever average Recall@5: 43.0 → 52.7).
  • They succeed because the rewriting supplies the missing context, not because they are small.
  • For supervised retrievers on their own training domains (DPR, GTR), propositions made almost no difference.
• Two other ways to add context back:
  • Late chunking ↗ runs the whole document through a long-context model once, then pools the token vectors per chunk. Each chunk vector carries context from the full document. Its gain is largest at small chunk sizes.
  • Contextual retrieval ↗ has an LLM prepend a sentence of document context to each chunk before embedding.

Aside: the dimension ceiling is not a chunk-length bound

Weller et al. ↗ (ICLR 2026) show that a vector of dimension d can only produce so many distinct top-k result sets. In their best-case experiment, vectors are optimized directly on the test data, and every pair of documents must be returnable as a top-2 result. The largest corpus that works fits n ≈ 0.0037·d³ (about 500k for d=512, 4M for d=1024).

It is tempting to turn this into a chunk-length bound. A corpus of T tokens in chunks of L tokens gives T/L vectors, so L ≥ T / (0.0037·d³), which is about 250 tokens for a billion tokens at d=1024. The conversion doesn’t hold:

• The query assumption is too pessimistic. It requires every pair of chunks to be retrievable together. Chunks of the same document never need to be told apart when documents are ranked by their best chunk. Real query sets need only a tiny share of all pairs, e.g. 3.7×10⁻⁵ of document pairs share a relevant query in HotpotQA (paper, App. F).
• The model assumption is too optimistic. Real encoders hit the ceiling far earlier than vectors optimized on the test set.
• The numbers are extrapolated. The curve was fitted only up to d = 45, so the values for d = 768–1024 are about 20× beyond the data.
• The paper’s own theorem gives tiny limits. With margin γ = 0.1, 10¹¹ documents need at least d ≥ 21. The authors call it “an extreme lower bound”.

What does hold: dense retrieval degrades faster than sparse retrieval as the index grows, and more so at low dimensions (Reimers & Gurevych, 2021 ↗). Very small chunks over a very large corpus at a low dimension is the risky corner.

The tension, and how it is resolved

• Smaller chunks match more precisely, but lose context.
• Larger chunks keep context, but get diluted.

There are two ways out of the trade-off:

1. Put the context back into small chunks: propositions, late chunking, contextual retrieval (above).
2. Retrieve small, return big. Match on small chunks, then hand the user or the LLM the parent section or page.
   • The retrieval unit is optimized for matching, and the returned unit for answering. These are separate choices.
   • HiChunk ↗ (ACL 2026) does this at retrieval time (“auto-merge”: retrieved chunks are merged up into their parent section). On a set where each answer needs a lot of evidence, evidence recall rises from 77.7 to 81.0.

What the experiments found

Study	Sizes tested	Best	Notes
Callan 1994 ↗	25–10,000-word windows	150–300 words	Lexical retrieval, before BM25. 200–250 words was a safe single choice.
Chroma 2024 ↗	200–800 tokens	Recall at ~250–400; precision at ~200	text-embedding-3-large. Overlap didn’t help this model (400/0 recall 89.5 vs 800/400 85.4). The weaker all-MiniLM-L6-v2 needed overlap (82.4 vs 77.1).
Bhat et al. 2025 ↗	64–1,024 tokens	Depends on dataset	SQuAD best at 64. TechQA 16.5% → 61.3% from 128 to 512. NarrativeQA best at 1,024.
NVIDIA 2025 ↗	128–2,048 tokens, page, section	Page-level overall	End-to-end answer accuracy. Factoid sets best at 256–512, analytical ones at 1,024 or page-level.
Late chunking ↗, Table 5	128 vs 512 tokens vs whole 8k document	Chunks	Naive chunking. Chunks beat one whole-document vector on 3 of 4 long-document tasks.

Patterns:

• Query type matters most.
  • Factoid and entity questions peak at 64–256 tokens.
  • Analytical, multi-hop and long-document questions want 512–1,024 tokens or a page, or small chunks with “retrieve small, return big”.
  • About 200–400 tokens is a robust default. Chroma’s dense results support it, and Callan’s lexical 150–300 words lands in the same range.
• Overlap reduces damage from cuts at bad places. The evidence is mixed: it helped a weak model and not a strong one (Chroma).
• The model shifts the optimum. Supervised models on their own training domain are less sensitive to chunk size. Weaker models are more sensitive to how text is split.
• No evidence ties the optimum to vector dimension. Shaukat et al. (2026) ↗ tested five models with 64–1024 dimensions and found |r| < 0.2 between dimension and quality, though dimension is confounded with the model.

Ranking a document from its chunks

In IR this is called passage-level evidence, and it is older than embeddings (Callan, 1994). Common ways to turn chunk scores into a document score:

Method	Document score
MaxP	Best chunk’s score
FirstP	First chunk’s score
SumP	Sum of chunk scores
Document + passage	Whole-document score combined with the best chunk’s score
Learned	A model combines the chunks

Dai & Callan (2019) ↗, 150-word windows, nDCG@20. Title queries are a few keywords; description queries are a full sentence.

Collection, query type	FirstP	MaxP	SumP
Robust04, title	0.444	0.469	0.467
Robust04, description	0.491	0.529	0.524
ClueWeb09-B, title	0.286	0.293	0.289
ClueWeb09-B, description	0.272	0.262	0.261

MaxP is at least as good as SumP everywhere. FirstP wins only on ClueWeb description queries.

These results come from BERT rerankers (cross-encoders), which score each query–chunk pair directly. That MaxP also wins for chunk vectors is plausible, and it’s what engines do, but these studies don’t test it directly.

Combining scores:

• Document + passage. Callan found it “was always best to combine document-level evidence and passage-level evidence”. The gain over the document score alone was +5.5% to +23.5% across collections.
• Learned aggregation pays off when relevant content is spread across a document. PARADE ↗ (2020) beats a MaxP baseline on GOV2 (nDCG@20 0.527 → 0.609), where 40% of relevant documents contain 3 or more relevant passages.
• When relevance sits in one passage, as in MS MARCO (99% of relevant documents have exactly one relevant passage), PARADE’s simple max-pooling variant beat its learned one on TREC DL 2019/2020.

Chunk size matters less when you rank documents rather than chunks:

• Zhou et al. (2026) ↗ found chunk size correlates only weakly with document-level retrieval (r ≈ 0.07–0.18). They suggest MaxP as “one possible explanation”.
• Qu et al. (2025) ↗ found document recall steady across chunk sizes. Precision rose with smaller chunks.

Engines default to MaxP:

• Elasticsearch: “Nested kNN queries only support score_mode = max” (docs ↗).
• Qdrant: grouped results are “ordered by the score of the top point in the group” (docs ↗).
• Vespa: for multi-vector fields, closeness uses the closest vector (docs ↗).

Pitfall: if you retrieve the top N chunks and then group them by document, you get fewer than N documents. Retrieve several times more chunks than the number of documents you need.

One vector or many

Queries about the document as a whole (its topic, similar documents, classification, deduplication) want one vector for the whole document.

• Long-context embedder. The whole document goes through the model in one pass, and pooling collapses the per-token vectors into one: mean over tokens, a special CLS token, or the last token for LLM-based embedders. The upper-bound limits apply.
• Late chunking. The same pass gives both the per-chunk vectors and, pooled over all tokens, the whole-document vector.

Factoid queries want many vectors per document, used to find the relevant passages within a document, or to rank documents by their best chunk (MaxP). In the late chunking paper (Table 5), an 8k model’s chunk vectors (128 or 512 tokens) beat one whole-document vector on 3 of 4 long-document tasks, +24.5% relative at 512 tokens. The exception was summary-style queries (SummScreenFD), where the gist of the whole document is the answer.

Averaging chunk vectors is a fallback, not a real option. If you already have chunk vectors, you can always average them into a document vector, but it is a poor proxy for the document:

• With dot product, q · mean(v₁…vₙ) = mean(q · v₁ … q · vₙ). The averaged vector scores the document by its average chunk score.
• With cosine, it’s the average score multiplied by 1/‖mean v‖. That is a per-document constant, which boosts documents whose chunks point in different directions.
• Either way, it can’t recover the best chunk’s score. It’s close to SumP, which trails MaxP above, and it never sees the document in one pass.

Idea: markup-aware chunking

This is an idea, not a proven improvement. There is no clear evidence that it beats a well-tuned fixed chunk size.

Fixed-size chunks can cut anywhere: mid-sentence, mid-table, mid-code-block. Documents with markup (Markdown, HTML, PDF converted to Markdown with tools like marker or docling, LaTeX) already mark the natural boundaries. So instead of a fixed size, fit chunks of varying size to the markup boundaries.

The proposed rules:

1. Sizes: target 200–400 tokens, minimum about 100, maximum the model’s training length.
2. Procedure: split by the highest structural level (section, then subsection, then paragraph, then sentence) until every piece is under the maximum. Then merge neighbouring pieces under the minimum, within the same section. Never split mid-sentence.
3. Tables and code blocks are never merged with prose, and never split mid-row or mid-block. Each keeps its caption or lead-in sentence, which supplies the context it needs. A table over the maximum is split by rows, repeating the header row in each piece.
4. Optionally prepend the heading path (e.g. “Pricing > Enterprise > Limits”) to each chunk.

Related evidence

Study	What was compared	Result
MDKeyChunker ↗ (2026, preregistered)	Markdown chunks split at headings, with tables and code kept whole, vs 512-character windows; hybrid BM25 + dense retrieval	+23.0 points on Qasper, +5.1 on FreshStack (developer docs)
HiChunk ↗ (ACL 2026)	Chunks following an inferred section structure (~200-token target) vs fixed 200 tokens	Evidence recall 77.7 vs 74.1 on questions whose evidence is dense
Jimeno Yepes et al. ↗ (2024)	Chunks following document elements in financial reports (new chunk at each title, tables whole) vs fixed 512 tokens	QA accuracy 53.2 vs 48.2, but lower page-retrieval accuracy. Confounded: the element chunks also had LLM-generated summaries and keywords.

Findings that bear on specific rules:

• Rule 2, merging small pieces. In HtmlRAG ↗ (WWW 2025), using raw HTML elements as units cost up to 9 points of end-to-end QA accuracy, because they were “so fragmented that the embedding model fails”. The authors merged them into blocks of up to 256 words.
• Rule 3, tables.
  • TARGET ↗ (2025): embedding the whole table (column names plus rows, as Markdown) beats embedding an LLM summary of it. On FeTaQA, Recall@10 was 0.741 vs 0.436; on TabFact 0.824 vs 0.469. Column names alone scored 0.208 on FeTaQA. One row per chunk scored about the same as the whole table (0.711), so splitting big tables by rows looks safe.
  • How the table is written out matters, and it depends on the model (Bhandari et al., 2026 ↗). On WTQ, MPNet’s Recall@1 was 0.09 with HTML vs 0.25 with TSV. BGE-M3 was less sensitive but still varied (0.22–0.31). Avoid heavy HTML; plain row formats are safer.

Evidence against

• The advantage shrinks against better baselines.
  • Against 256-token overlapping windows, MDKeyChunker’s gain falls to +12.7 on Qasper and −2.6 (not significant) on FreshStack.
  • HiChunk’s gain almost disappears on datasets where each question needs little evidence.
• On plain text, where you cut barely matters. In the late chunking experiments on BEIR, fixed 256 tokens scored 52.2 nDCG@10, 5-sentence chunks 52.4, and semantic boundaries 52.4.
• Heading-path prefixes (rule 4) are inconsistent. Contextual retrieval gains from LLM-written context. But in MDKeyChunker the section-path prefix changed results by only −1.5 to +1.3 points.

Gaps

No controlled study was found for:

• mixing code and prose in one chunk;
• embedding a table alone vs inside a prose chunk;
• leaving raw Markdown syntax in prose chunks vs stripping it.

Most studies above are small (under 150 questions) or 2025–26 preprints.

The practical way to judge it is to compare markup-aware chunks against a 256-token overlapping baseline on a labelled query set from your own documents.

Summary

Parameter	Value
Hard limit	The model’s context window, and the reranker’s input limit (512 tokens for classic cross-encoders, 32k for LLM-based ones)
Upper bound	The model’s training length (≈512 tokens for most BERT-era models), with quality degrading before that
Lower bound	Semantic: the chunk must make sense on its own. Small chunks work if context is added back.
Default	200–400 tokens
By query type	Factoid: 64–256. Analytical: 512–1,024 or a page, or retrieve small and return big.
Overlap	Helps weaker models, mixed evidence otherwise
Document score	MaxP. Combining it with a whole-document score or learned aggregation helps when relevant content is spread within a document.
One vector per document	Only for queries about the document as a whole: a long-context embedder or late chunking, not an average of chunk vectors
Final choice	Tune on a small labelled query set from your own data