Distillation
BM25
How does a search engine decide which document is most relevant — using nothing but word counts?
Three ordinary ideas, stacked.
Imagine you search for “vector database”. You have five documents. Document A says “vector” five times; document B mentions it once. Which one is more likely to be about vectors? The one that keeps talking about them.
So: how often a word appears matters. That is term frequency.
But there is a complication. If every document contains the word “database”, then seeing it in a document tells you almost nothing. Rare words discriminate; common words are background noise. That is how rare a word is: inverse document frequency.
And one more: a long document has more chances to repeat any word at all. Raw counts quietly favour the longest page. So the count should be corrected for how long the document is.
Three ordinary ideas — frequency, rarity, length — and BM25 is simply the way they are combined.
Deeper
Why do repeats stop mattering so quickly? Because the second mention of a word adds less evidence than the first, and the tenth adds almost nothing. Relevance saturates. BM25 models this with a curve whose steepness is controlled by a knob called k₁.
The equation looks intimidating. The idea isn’t.
The formula below is the three ideas stacked: rarity multiplies frequency, and frequency is curved and length-corrected.
Ranked documents
- 01 D1 len 12 · norm 1.01 0.68
Vector databases store vectors. A database indexes vectors for fast vector search.
- 02 D2 len 15 · norm 1.20 0.63
A vector database stores embeddings, and searching embeddings with a vector index returns similar items.
- 03 D5 len 12 · norm 1.01 0.57
A database can be a vector store if it indexes embeddings efficiently.
- 04 D4 len 8 · norm 0.76 0.33
Vector search finds similar embeddings. Embeddings encode meaning.
- 05 D3 len 12 · norm 1.01 0.29
Relational databases use tree indexes; choosing a database depends on the workload.
Query terms
| term | df | idf |
|---|---|---|
| vector | 4 | 0.29 |
| database | 4 | 0.29 |
Saturation — the tf part, for the top document
Score breakdown — D1
| term | tf | idf | tf part | contribution |
|---|---|---|---|---|
| vector | 2 | 0.29 | 1.37 | 0.39 |
| database | 1 | 0.29 | 0.99 | 0.29 |
Read the breakdown table as a sentence: this term appears this often, it is this rare, the document is this long — so it contributes this much.
Deeper
k₁ controls saturation. At k₁ = 0 a word counts once, whether it appears once or fifty times. At high k₁, extra occurrences keep adding. Most tuned systems land somewhere around 1.2 — strong preference for frequency, with diminishing returns.
b controls length correction. At b = 0, document length is ignored. At b = 1, length is fully compensated. Around 0.75 is a common default. Neither default is truth; both are starting points for your data.
Math
With a query of terms q and document D:
score(D, Q) = Σ IDF(qᵢ) · f(qᵢ, D)·(k₁ + 1) / ( f(qᵢ, D) + k₁·(1 − b + b·|D|/avgdl) )
f(qᵢ, D)— how many times termqᵢoccurs in the document. The(k₁ + 1)in the numerator normalises the curve so k₁ only changes its shape, not its scale.IDF(qᵢ) = ln(1 + (N − df + 0.5) / (df + 0.5))— withNdocuments anddfdocuments containing the term. The+0.5smoothing keeps very common terms from hitting zero or negative scores.|D| / avgdl— the document’s length relative to the corpus average.bdecides how much that matters.
Only the ranking matters, not the absolute scores. Two documents with the same score are, as far as BM25 is concerned, equally relevant.
Old, unglamorous, everywhere.
BM25 was published in the 1990s and is still inside most search you touch: Elasticsearch, Postgres full-text search, SQLite’s FTS5 — and small local tools, like the MCP server I use to search my own Claude Code history.
The query is tokenised by the same analysis chain that processed the documents — lowercasing, stemming, maybe stop words. A mismatch here is invisible and expensive.
Deeper
The reason BM25 survives every wave of new technology: it needs no training, no GPU, no network, and it explains itself. When a document ranks first, you can point at the exact term frequencies and IDF values that put it there. That explainability is worth a lot when search results look wrong.
What word counts cannot see.
- Vocabulary mismatch. You search “puppy”; the document says “dog”. To BM25 these are two unrelated strings. No amount of tuning fixes this — it is the ceiling of the method.
- Morphology. “indexes” and “index” are different tokens unless you stem them. Stemming helps, and occasionally conflates things it should not — a trade, not a fix.
- No word order, no negation. “not a vector database” scores like “vector database”. BM25 sees a bag of words, not a sentence.
- Tuning is dataset-specific. k₁ and b are not universal constants. The defaults are a place to start, not an answer.
- Repetition without meaning. Keyword stuffing works on BM25 in the same sad way it once worked on the web.
Which is why the honest answer is: BM25 and embeddings belong together. One is precise about words, the other is flexible about meaning. Neither alone is a search engine.
If you remember one thing
BM25 ranks a document by how often your words appear in it, weighted by how rare those words are, corrected for document length.