Text search options

There are four main families of text search. In rough historical order:

Substring and regular-expression search, e.g. grep, ripgrep, ugrep.
The TF-IDF family, with BM25 as its pinnacle, over an inverted index. Almost every classic search engine uses BM25: Lucene, Elasticsearch, OpenSearch, Solr, Tantivy, SQLite FTS5, Postgres extensions and so on.
Learned sparse retrieval, like SPLADE.
Dense vectors: embeddings plus nearest-neighbour search.

A fifth, newer step is not a new family: agentic search, where an LLM agent drives the search and uses any of the four as tools.

Each step moves from matching strings towards matching meaning. Each step also changes what you trade. You gain understanding of language and give up some exactness. At the dense end you also give up control and the ability to explain why something matched.

Out of scope:

Non-alphabetic languages (CJK etc.).
Typo-tolerant (fuzzy) search.
Specialised search like code search or math formula search.
Speed and cost. Only the quality of results is discussed.

Quality metrics used below: MAP/AP, MRR@10 and nDCG@10. For an introduction see Introduction to Information Retrieval, ch. 8 ↗ (Manning, Raghavan & Schütze; covers MAP and nDCG). MRR is covered in the Stanford CS276 evaluation lecture ↗.

1. Regex: exact matching

Regex search is not really comparable to the others. It isn’t a relevance tool at all: a line either matches or it doesn’t, and results come back in file order. The user writes the pattern, so the user is doing the “understanding” part.

It is still in wide use, for different reasons:

Exactness. It finds exactly what you wrote and nothing else. Text is searched as is, so nothing gets lost to tokenization or stemming. That’s what you want for logs, identifiers, error messages, config keys and code.
Predictability. It is easy to reason about why something matched or didn’t.

The only quality levers are a thin normalization layer:

Case-insensitive matching (-i), and smart case (rg -S): case-insensitive unless the pattern contains an uppercase letter.
Unicode-aware case folding instead of ASCII-only. For example, MICRO SIGN µ matches GREEK SMALL LETTER MU μ.
Unicode normalization. é can be one codepoint (NFC) or e plus a combining accent (NFD). A byte- or codepoint-based grep treats these as different strings.
Diacritic folding: cafe matches café. This is rare in grep-like tools. SQLite’s trigram tokenizer ↗ has remove_diacritics, and ICU StringSearch ↗ can ignore accents and case by matching on collation.
Word boundaries (-w, \b). Here too there is more than one correct definition. ripgrep’s -w matches (-2) for -2, where \b-2\b does not (guide ↗).

Nothing beyond that: no stemming, no synonyms, no ranking. Once you need those, you’re in the next family.

2. TF-IDF family and BM25: approximating how people search

This family is the first to approximate how people actually use search:

People type a few words, not a pattern. The query is treated as a bag of words.
People expect the best results first. Documents are ranked:
- Rare terms count more (IDF).
- Repeated terms count more, with diminishing returns (TF saturation).
- Long documents don’t win just by being long (length normalization).

The history in brief:

Boolean retrieval over inverted indexes came first. It already worked on words rather than raw strings, but it returned unranked sets.
Ranking:
- IDF (Spärck Jones, 1972 ↗).
- The vector space model with TF-IDF weights, from Salton’s SMART system (1960s; Salton, Wong & Yang, 1975 ↗).
- The probabilistic model (Robertson & Spärck Jones, 1976 ↗).
BM25 came out of that probabilistic line at Okapi at TREC-3 ↗ (1994). It is the version that survived.

Compared with regex, this gives up exactness. After tokenization, case folding and stemming, the query no longer has to appear in the text verbatim, and punctuation and substrings inside words can’t be searched at all.

The formula is saturated

Kamphuis, de Vries, Boytsov, Lin, “Which BM25 Do You Mean?” ↗ (ECIR 2020) compared 8 BM25 variants (Robertson, Lucene, ATIRE, BM25L, BM25+, BM25-adpt and others) with the same analysis and parameters:

No pair of variants differs significantly on any of the three test collections. On Robust04, AP ranges from .2516 to .2571.
In the authors’ words: “differences due to more mundane settings (such as the choice of stopwords) are often larger than the differences we observe here.”
Trotman et al., 2014 ↗ found the same for 9 ranking functions, each tuned.

Formula-level refinements are minor:

Tuning k1 and b.
BM25F for multiple fields (2004).
Term proximity: rank documents where query terms appear close together higher (Rasolofo & Savoy, 2003; BM25TP, Büttcher, Clarke & Lushman, 2006). In Lucene-family engines this is usually approximated with phrase or slop queries.

Additions: getting closer to how people use language

The bag of words is a crude model of language. Engines patch it with an analysis chain: a sequence of steps that turns raw text into index terms, applied the same way to documents and queries. Every step closes one more gap between the words in the query and the words in the document. Each step is a hand-written approximation, and each has its own ceiling.

Language identification. Stemmers, stopword lists and decompounding are language-specific, so the engine first needs to know the language.
- The classic method compares character n-gram frequency profiles (Cavnar & Trenkle, 1994 ↗), reporting 99.8% accuracy on Usenet articles. It is implemented in TextCat and libtextcat.
- Later tools: Google’s CLD2/CLD3, and fastText’s lid.176 ↗ (176 languages, 2016).
- The usual setup detects the language of each document and indexes it with that language’s analyzer. Queries are short, which makes detection less reliable, so they are often searched across all language fields.
Tokenization: what counts as a word.
- There is no correct way to do it. The popular approximation is Unicode UAX #29 ↗ word segmentation, implemented by ICU and by Lucene’s StandardTokenizer. The annex itself calls it a “workable default” that may need tailoring.
- Hyphens, apostrophes, French elision (l'homme), URLs and emails all need decisions.
Case folding and diacritics stripping. People don’t type capitals or accents consistently.
- Lowercasing is not the same as Unicode case folding: ß → ss, and Turkish dotted and dotless i need special handling.
Stemming. running should match run. The stemmers (Lovins 1968, Porter 1980, Snowball ↗ 2001, Savoy’s “light” stemmers 2006) are algorithmic and have an upper bound. Snowball is roughly the pinnacle.
- Stemming is not a straightforward improvement: it helps some queries and hurts others. Conflating organization with organ, or university with universe (Porter stems both to univers), adds false matches.
  - Harman (1991) ↗ found no significant improvement on small early test collections (Cranfield, Medlars, CACM). Query by query, the number of queries that improved almost equalled the number that got worse.
  - Hull (1996) found that stemming makes a difference for many individual queries, but the average gain in search quality was only 1–3%. Small, but statistically significant for every stemmer.
  - Both are as summarized by Fautsch & Savoy (below).
- Fautsch & Savoy, 2009 ↗ measured about +7% MAP on English, and Porter, a plural-only stemmer and lemmatization were not significantly different from each other.
- LLMs don’t break the ceiling either. In Wang, Zhuang & Zuccon (SIGIR 2024) ↗, LLM-based vocabulary and contextual stemming “fail to achieve higher effectiveness than traditional stemmers”. Only entity-based contextual stemming beat Porter, and only “under specific conditions”.
- The gains grow with morphological richness. Tomlinson’s per-language gains, as quoted by Fautsch & Savoy: Dutch +4%, French +9%, German +19%, Swedish +29%, Finnish +40%.
Lemmatization, the obvious next step, gives a questionable win.
- Tomlinson (CLEF 2003, 9 languages) ↗ found the only significant differences vs Snowball in languages with many compound words (German, Dutch, Finnish). Those differences came from decompounding, not from lemmatization as such.
- Airio (2006) ↗, on English, Finnish, German and Swedish, found “no remarkable differences” between stemming and lemmatization.
Decompounding. In German, Fußballweltmeisterschaft should match Weltmeisterschaft. Braschler & Ripplinger, 2004 ↗:
- Decompounding helps more than stemming: +16–34% MAP for short queries.
- Combined with stemming: +33–60%.
Stopwords. This went the other way. Modern engines keep them, because IDF already down-weights them and removing them breaks phrases like “to be or not to be”.
- Elasticsearch has had no stopwords by default since 1.0 (2014).
- Lucene dropped them from StandardAnalyzer in 8.0.
Synonyms and acronyms. car should match automobile, and NYC should match New York City. The lists are hand-made and domain-specific.
Transliteration. A query in one script should match text in another: Serbian is written in both Cyrillic and Latin, so Beograd should match Београд (ICU transforms).
Markup stripping. Tag names and attributes shouldn’t become searchable words (html_strip char filter).

The upper bound

All of this is still lexical matching. If the document doesn’t contain the query’s words, or their stems, or a synonym someone wrote down, it can’t be found. This is called vocabulary mismatch.

Proximity variants and phrase queries capture how close words are, but BM25 still has no notion of negation or of what the query means: “flights not via London” boosts documents about London.

The last improvements to BM25 already borrow from outside it:

Relevance feedback. Rocchio (1971) expanded the query with terms from documents the user marked relevant. Pseudo-relevance feedback (RM3, 2001–2004) assumes the top results are relevant and does the same automatically. Tuned BM25+RM3 was still competitive with most neural models on Robust04 in 2019 ↗.
Document expansion with a neural model: doc2query / docTTTTTquery ↗ (2019) generates likely queries for each document and appends them. On MS MARCO, MRR@10 goes from 0.184 to 0.277.
Query expansion with an LLM: query2doc ↗ (2023) gives BM25 +3–15%.

Once a language model generates the extra words, the next question is why not let the model decide the words directly.

3. SPLADE: learning the additions

Learned sparse retrieval is the bridge between BM25 and dense vectors. It keeps BM25’s shape: an inverted index, with terms and weights. But a transformer replaces the hand-made analysis chain:

Learned term weights instead of TF-IDF: DeepCT (2019), DeepImpact (2021), uniCOIL (2021).
Learned expansion instead of synonym lists and stemmers: the model adds related terms that are not in the text.
SPLADE ↗ (2021) does both in one model. It outputs a weight for every term in BERT’s ~30k WordPiece vocabulary.

Quality now depends on training rather than on rules:

Distillation from a stronger model, and training on hard negatives: documents that look relevant but aren’t (SPLADE++ ↗, 2022; SPLADE-v3 ↗, 2024).
It generalizes well. On 13 BEIR datasets (zero-shot, outside the training domain), SPLADE++ averages 50.5 nDCG@10, against 43.7 for BM25 and the dense models of its time (TAS-B 43.7, Contriever 47.5).

The trade-offs:

Kept from BM25: you can still see which terms matched and with what weight. Term matching is mostly kept, but the model may down-weight query terms. Rare names and IDs get split into word pieces, so exact matching is weaker than in BM25.
Lost: control. The expansion comes from the model, and you can’t fix it by editing a list.
New limitation: the tokenizer comes back.
- Dense models also use WordPiece, but for SPLADE the vocabulary is the output space. The model can only produce terms that exist in it, and English BERT has almost no room for other languages.
- Multilingual versions have been hard (BLADE, SIGIR 2023 ↗).

4. Dense vectors: understanding instead of matching

Vectors for meaning are an old idea. Latent Semantic Indexing ↗ (Deerwester et al., 1990) mapped documents into a low-dimensional “concept” space, but it never displaced lexical ranking. What changed is that the representations are now learned by large neural models trained on huge amounts of text.

Dense models don’t need the linguistic additions. There is no stemming, no synonym list, no stopword list, no decompounding and no case folding. The model has seen enough text that:

running is close to run, and car is close to automobile;
a paraphrase is close to the original;
with a multilingual model, a question in German is close to an answer in English.

Text cleaning (markup stripping, Unicode normalization) still applies. Vocabulary mismatch doesn’t disappear either. It changes shape and comes back for rare names, new jargon and IDs (see below).

Quality now depends almost entirely on the model:

Model size is one of the strongest levers.
- “Scaling Laws for Dense Retrieval ↗” (SIGIR 2024): measured on contrastive loss, quality follows a power law in model size and the amount of training data.
- GTR ↗ (2021) scaled the encoder up while keeping the vector size fixed at 768, and it still generalized better.
Training data and recipe matter about as much as size.
- Hard negatives: DPR ↗ and ANCE ↗, 2020.
- Contrastive pretraining at scale, learning that matching text pairs belong together: Contriever (2021) and E5 ↗ (2022).
- LLM backbones and instructions: E5-mistral ↗ (ACL 2024).
Domain fit. GPL ↗ (2021) adapts a model to a new domain with synthetic queries. It gains up to 9.3 nDCG@10 points over the same model unadapted.
Multi-vector / late interaction. ColBERT ↗ (2020) keeps one vector per token and scores by matching each query token to its best document token. It is dense retrieval’s counterpart of SPLADE’s bridge role: closer to term matching, while keeping learned representations.

Different trade-offs and limitations

You gain understanding, and you lose these:

Exact matching. Rare names, IDs, product codes and numbers become “something similar”. The nearest vector is not the same as the exact term.
- Sciavolino et al., 2021 ↗: dense retrievers fail on simple questions about less common entities, where BM25 does fine.
- Out of domain, early dense models fell below BM25 on BEIR ↗ (2021). Newer, larger models beat it on average (see the comparison table below).
Negation. Understanding language doesn’t mean understanding negation.
- In NevIR ↗ (EACL 2024), pairs of documents differ only by a negation, and random ranking scores 25%.
- “No bi-encoder architecture scores higher than 12%”, and SPLADE scores about 8%.
- Only cross-encoders do better than random (MonoT5-3B: 50.6%).
Explainability and control. There is no term to point at to explain why a document matched, and nothing to edit when it shouldn’t have. The only fixes are to change the model or the data.
Long documents. One vector per long document loses detail, so documents get split into chunks. Chunking is the new analysis chain: the extra work moves from processing language to preparing documents. See Chunk size for dense retrieval for how big chunks should be and how to rank a document from its chunks.
- Late chunking ↗ (2024): embed the whole document, then pool per chunk, so each chunk vector keeps context from the rest of the document.
- Contextual retrieval ↗ (2024): an LLM prepends document context to each chunk. This cut top-20 retrieval failures by 35%, and by 49% together with BM25 over the same contextualized chunks.
Dimension ceiling. Weller et al. ↗ (ICLR 2026) prove that a vector of dimension d can only produce so many different top-k result sets across all possible queries.
- This is not about long documents. Their LIMIT benchmark uses tiny documents like “Jon likes apples and quokkas”, and each query asks for one thing (“who likes quokkas?”).
- The difficulty is the combinations. Each query has two relevant documents, and together the queries cover every pair from a set of 46 documents.
- Recall@100 on the 50k-document version: BM25 93.6, a multi-vector model (ColBERT) 54.8, the best single-vector model 18.9.
- The limit is about dimension, not model size. Larger dimensions help. In their best-case experiment, with vectors optimized directly on the test data, the largest corpus that works grows roughly as d³. Real models hit the limit much earlier.
- Multi-vector and sparse representations fare better. The authors say none of the alternatives provides “a clear path to solving this problem as-is”.

5. Agentic search: the model drives the search

The latest step is not a new way of matching but a new driver. An LLM agent writes the queries, reads the results and searches again, using any of the methods above as tools. This brings regex back: an agent reads through many results and keeps adjusting the query, and a strong model’s knowledge of language and domain does the job that the analysis chain or the embedding model did before. The evidence so far is young and mixed. Iteration narrows the gap between exact and semantic tools without closing it, and the weak point is the agent’s own queries. Details in Agentic search.

Where the trade-offs ended up

No approach wins outright, because they fail on different queries. Current practice combines them:

Hybrid search. Run BM25 (or SPLADE) and dense side by side and merge the two result lists. That hybrid is the default is itself evidence that dense vectors didn’t make BM25 irrelevant.
- Rank fusion uses only positions. Reciprocal Rank Fusion ↗ (2009) scores each document 1/(60 + rank), summed over the lists. It needs no tuning and is the common default. It descends from older meta-search methods like Borda and Condorcet fuse.
- Score fusion normalizes the scores and takes a weighted sum, α·dense + (1−α)·lexical.
- Bruch, Gai & Ingber ↗ (ACM TOIS, 2024) found that a tuned weighted sum beats RRF both in-domain and out of domain, by a few nDCG points. Tuning needed less than 5% of the training queries. So RRF is the safe choice without labelled queries; with even a few, use a weighted sum.
Reranking. A cross-encoder (monoBERT ↗, 2019) or an LLM (RankGPT ↗, 2023) reads the query and each candidate together.
- This is the quality ceiling of the whole field, and the only stage that handles negation reasonably.
- It removes the dimension ceiling from scoring, but not from recall. A document the first stage missed never reaches the reranker.
Regex alongside: for when the user knows exactly what string they’re looking for.

Approach	Matches on	Who handles language	Gained vs the previous step	Lost vs the previous step
Regex	The exact pattern	The user	—	—
TF-IDF / BM25	Terms, after the analysis chain	Hand-made rules (stemmers, synonyms) with a ceiling	Ranking; matching how people type queries	Exactness (verbatim strings, punctuation, substrings)
SPLADE	Terms, with learned weights and expansion	A trained model, bound to its vocabulary	Expansion without hand-made lists	Direct control; some exactness for rare terms; language coverage
Dense	Meaning (vector similarity)	The model	Paraphrase, morphology, cross-lingual	Exactness for rare terms; explainability; a dimension ceiling
Agentic	Whatever its tools match on, including exact patterns again	An LLM writing the queries	Iteration, query rewriting, using several methods together	Predictability: results depend on the agent’s queries, which often fail

Comparison

Each approach is shown at its bottom (early or weak) and top (strong) end, on two benchmarks:

MS MARCO dev, MRR@10: in-domain. Models were trained on MS MARCO.
BEIR-13, nDCG@10: the average over 13 public BEIR datasets. Zero-shot: models were not trained on these domains.

Numbers come from different papers, so treat differences under about 0.5 as noise.

Approach	System	MS MARCO MRR@10	BEIR-13 nDCG@10
Regex	—	not ranked	not ranked
BM25	BM25	18.4	43.7
BM25 + expansion	docT5query	27.7	45.3¹
Learned sparse, bottom	SPLADE (2021)	32.2	—
Learned sparse, top	SPLADE-v3 (2024)	40.2	51.7
Dense, bottom	DPR (2020)	31.1	26.3¹
Dense, top	RepLLaMA, 7B (2023)	41.2	55.1
Multi-vector, bottom	ColBERT (2020)	36.0	45.1¹
Multi-vector, top	ColBERTv2 (2021)	39.7	49.7
Hybrid	SPLADE++ + BM25	—	52.1
Rerank, bottom	BM25 + monoBERT / cross-encoder	36.5	49.5¹
Rerank, top	RepLLaMA + RankLLaMA	45.2²	56.6²

¹ Averaged over the same 13 datasets from the per-dataset scores in the BEIR paper ↗, Table 2. DPR was trained on Natural Questions, not MS MARCO. ² RankLLaMA-13B for MS MARCO and RankLLaMA-7B for BEIR.

Sources:

SPLADE ↗, Table 1.
SPLADE++ ↗, Tables 1–3: BM25 BEIR, hybrid.
SPLADE-v3 ↗, Tables 1–2.
ColBERTv2 ↗, Table 4: DPR, ColBERT and ColBERTv2 MS MARCO.
monoBERT ↗, Table 1.
RepLLaMA / RankLLaMA ↗, Tables 1–2.

The in-domain gap is large: from 18.4 for BM25 to 45.2 with an LLM reranker. Out of domain it is much smaller (43.7 to 56.6), and early dense models fall well below BM25 there. BM25 needs no training, so it has no training domain to fall out of.