BM25: the ranking function that refuses to die

2026-07-02 · 7 min · retrieval · search · ranking · algorithms

Why read this

BM25 built term by term from TF-IDF, with a live scorer you can query, sliders for k1 and b, and 30 lines of Python that reproduce it.

Runs on a laptop CPU
Interactive explanations
Evergreen reference

Developer tools & infraIntro guide

How this was scored

Is it new?: 0 of 3: Repackaging or news of a known thing
Can I trust it?: 1 of 3: Spot-checks a few numbers
Can I run it?: 3 of 3: Open, permissive, runs on reader hardware with instructions
Will I understand it?: 3 of 3: Mechanism carried by interactives built from real code or data
Can I act on it?: 2 of 3: A concrete recipe, numbers or comparison
Will it last?: 3 of 3: Evergreen fundamentals
Does it affect many?: 2 of 3: A widely used model, tool or lab release
Only here?: 1 of 3: Some original analysis

Score 61 of 100, ranked 314 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Here's a fact that should be more embarrassing for machine learning than it is: a ranking function designed in the early 1990s, with no learned parameters and no notion of meaning, is still the default text scorer in Lucene, Elasticsearch, and OpenSearch — and still a baseline that dense neural retrievers regularly fail to beat out of domain. It's called BM25, and if you do search, retrieval, or RAG, it's worth understanding exactly, because you're almost certainly running it.

"BM25" is Best Matching 25 — roughly the 25th matching function tried in the Okapi information-retrieval project at City University London (Robertson, Spärck Jones, and colleagues), grounded in the probabilistic relevance framework. Strip away the theory and it's a small, honest idea: score a document by summing, over the query terms it contains, how rare each term is times how emphatically the document uses it — with two corrections that make it work in practice. Let's build it.

Start with TF-IDF, and its two flaws

The classic bag-of-words score is TF-IDF: for each query term, multiply its frequency in the document (term frequency, TF — "this document uses the word a lot") by its rarity across the corpus (inverse document frequency, IDF — "and the word is distinctive"). Sum over query terms. It's a reasonable instinct, and it has two clear problems:

TF grows linearly and forever. A document that says "jaguar" 40 times scores 40× one that says it once. But the difference between 1 and 2 mentions is meaningful; the difference between 39 and 40 is noise. Relevance saturates.
Length is unaccounted for. A 10,000-word document will rack up term counts just by being long, drowning out a tight 100-word document that's genuinely more on-topic.

BM25 is, almost exactly, TF-IDF with those two flaws fixed. Here's the whole thing:

\text{score}(D, Q) = \sum_{q \in Q} \text{IDF}(q)\cdot\frac{f(q, D)\,(k_1 + 1)}{f(q, D) + k_1\left(1 - b + b\,\dfrac{|D|}{\text{avgdl}}\right)}

Three pieces do all the work: the IDF weight, the saturating term-frequency factor (the $k_1$ part), and the length normalization (the $b$ part). Here they are, color-coded — the map for the rest of the piece:

anatomy of BM25 · three factors, one score

score(D,Q) =Σq ∈ QIDF(q)×f · (k1 + 1)f + k1 · (1 − b + b · |D|/avgdl)

rarity · IDF

Rare terms dominate; a word in every document scores ≈ 0, so there's no stopword list.

saturation · k1 = 1.2

Term frequency saturates — the first mention is strong evidence, the tenth is nearly noise.

length · b = 0.75

Longer documents are discounted, so they don't rank highly just for having more words.

Take them one at a time.

Rarity: the IDF term

$f(q,D)$ is just the count of term $q$ in document $D$ . The multiplier in front is inverse document frequency — how much a term's presence should count, based on how rare it is:

\text{IDF}(q) = \ln\!\left(1 + \frac{N - n(q) + 0.5}{n(q) + 0.5}\right)

$N$ is the number of documents, $n(q)$ the number containing $q$ . A term in every document (like "the") gets an IDF near zero — matching it tells you nothing. A term in one document out of a million gets a large IDF — matching it is almost the whole story. This is why BM25 needs no stopword list: common words are down-weighted automatically because their IDF collapses. (The $\ln(1 + \cdots)$ form is Lucene's; it keeps IDF non-negative. The original Robertson–Spärck-Jones IDF drops the $+1$ and can go slightly negative for terms in more than half the corpus.)

inverse document frequency · rarity is (almost) the whole score

n — documents containing the term7 of 1000 (0.7%)

A term in one document out of 1000 is almost the whole story (IDF ≈ 6.5); a term in every document tells you nothing (IDF → 0). Matching a rare identifier — a name, an error code, a ticker — outweighs matching a hundred common words. And “the”, sitting in nearly every document on the right, down-weights itself: BM25 needs no stopword list because rarity is baked into the score.

Saturation: the k1 term

Now the first fix. Instead of using the raw count $f(q,D)$ , BM25 passes it through a saturating function $\frac{f\,(k_1+1)}{f + k_1\cdot(\dots)}$ that rises fast for the first few occurrences and then flattens toward an asymptote. The parameter $k_1$ controls how fast:

term-frequency saturation · why the 10th occurrence barely counts

k1 — saturation rate (Lucene 1.2)1.2

tf — occurrences (marker)1.57 · 71% of max

The curve flattens toward k1+1: extra repetitions of a term give ever-smaller returns. Small k1 saturates almost instantly (any occurrence counts about the same); large k1 keeps rewarding repetition, approaching the straight TF-IDF line. Real relevance behaves like the curve, not the line — which is exactly what BM25 encodes.

The first mention of a term is strong evidence; each additional mention adds less. That curve — not the straight line of TF-IDF — is how relevance actually behaves. Set $k_1 = 0$ and it becomes binary (any occurrence counts the same); crank $k_1$ up and it straightens back toward linear TF. Lucene's default is $k_1 = 1.2$ .

Length: the b term

The second fix lives in the denominator: the $k_1$ term is scaled by $\left(1 - b + b\,\frac{|D|}{\text{avgdl}}\right)$ , where $|D|$ is the document's length and $\text{avgdl}$ the average across the corpus. A longer-than-average document gets a bigger denominator, so its term-frequency factor is discounted — the same two mentions count for less when they're diluted across more text:

document-length normalization · same term count, different lengths

b — length normalization (Lucene default 0.75)0.75

As b rises, the longer-than-average document is penalized: the same two mentions are diluted across more text, so they're weaker evidence that the document is about the term. At b = 1 the normalization is full; b = 0.75 is the usual compromise.

The knob $b \in [0,1]$ sets how aggressively. At $b = 0$ length is ignored entirely; at $b = 1$ it's fully normalized. The default $b = 0.75$ is a compromise that's proven hard to beat.

Put it together: a live BM25 engine

That's the entire algorithm. Here it is running on a tiny corpus — edit the query, drag $k_1$ and $b$ , and every document is re-scored with the exact formula above. Expand a document to see each query term's contribution (its IDF times the saturated, length-normalized factor):

live BM25 · scores the 6-document corpus against your query

query

k11.2b0.75

doc 1the quick brown fox jumps over the lazy dog

per-term contribution · |D| = 9 tokens, avgdl = 12.0

quickidf 0.69 · tf 10.77

lazyidf 0.69 · tf 10.77

foxidf 1.03 · tf 11.15

Each score is a sum over query terms of rarity (IDF) times a term-frequency factor that saturates (controlled by k1) and is normalized by document length (controlled by b). Rare query terms dominate; repeating a common word barely helps; and long documents don’t win just for being long.

Notice the behaviors fall out on their own: rare query terms dominate the ranking, repeating a common word barely moves the score, and a long document doesn't win just for being long. No training, no embeddings — just term statistics arranged sensibly.

Thirty lines of Python

There's no magic hiding in a library. The whole thing is a couple of counters and the formula:

import math
from collections import Counter
 
class BM25:
    def __init__(self, corpus, k1=1.2, b=0.75):
        self.k1, self.b = k1, b
        self.docs = [doc.lower().split() for doc in corpus]
        self.N = len(self.docs)
        self.avgdl = sum(len(d) for d in self.docs) / self.N
        self.tf = [Counter(d) for d in self.docs]          # term counts per doc
        self.df = Counter()                                 # docs containing each term
        for d in self.docs:
            for term in set(d):
                self.df[term] += 1
 
    def idf(self, term):
        n = self.df.get(term, 0)
        return math.log(1 + (self.N - n + 0.5) / (n + 0.5))
 
    def score(self, query, i):
        d, tf = self.docs[i], self.tf[i]
        norm = self.k1 * (1 - self.b + self.b * len(d) / self.avgdl)
        s = 0.0
        for term in query.lower().split():
            f = tf.get(term, 0)
            if f:
                s += self.idf(term) * f * (self.k1 + 1) / (f + norm)
        return s
 
    def rank(self, query):
        return sorted(((i, self.score(query, i)) for i in range(self.N)),
                      key=lambda x: -x[1])

In production you don't loop over every document — you keep an inverted index (term → postings list of documents that contain it) and only score documents that share a term with the query. That's what makes BM25 fast enough to serve web-scale corpora on commodity hardware, and it's the same index that's been powering Lucene since 2011.

inverted index · a query only touches the postings it hits

query

term → postings list

quickdf 3→124

lazydf 3→134

foxdf 2→12

∪union→

documents scored

123456

scored 4 of 6 · 2 never touched

The index maps each term to its postings list — the documents that contain it — built once at index time. A query walks only its own terms’ postings and scores their union; the other 2 documents are never opened. At web scale that’s the difference between scoring a handful of postings and scanning a billion documents — and it’s the same index Lucene has shipped since 2011.

The variants you'll meet

The core formula spawned a small family, mostly patching edge cases:

BM25+ adds a small constant $\delta$ (default 1.0) to the term-frequency factor, fixing a subtle bug where very long documents can be over-penalized to the point that a document containing a rare term scores below one that doesn't.
BM25F ("fielded") scores structured documents — title, body, anchor text — by combining per-field term frequencies before saturation, with a weight per field, so a title match counts more than a body match. It's what real search engines actually run.
BM25L re-weights to stop long documents from being unfairly buried.
Lucene's implementation is BM25 with the non-negative IDF above and per-field length norms quantized into a single byte — the pragmatic engineering version of the equation.

Why it won't die

Neural retrieval was supposed to make this obsolete years ago. It hasn't, for reasons worth naming:

It's a brutal baseline. On out-of-domain benchmarks (the BEIR suite made this famous), BM25 beats or ties many dense retrievers — because it can match any term, including names, codes, and jargon a fixed-vocabulary embedding never saw in training. It never has an "out-of-distribution" moment.
It's the sparse half of hybrid search. The current default in serious systems is to run BM25 and a dense retriever and fuse the results (often with reciprocal-rank fusion). Lexical precision plus semantic recall beats either alone, which is why "BM25 is obsolete" quietly became "BM25 is one of your two retrievers."
It's cheap and interpretable. No GPU, no training, no embedding drift. When it ranks a document highly you can point at exactly which rare terms did it — which matters when a ranking has to be debugged or defended.

The lesson I take from BM25 is that a model doesn't have to learn anything to encode real knowledge about a problem. Every piece of it is a hypothesis about relevance — rarity matters, repetition saturates, length dilutes — written as arithmetic instead of learned from data. Three good hypotheses, two tunable knobs, and thirty-five years later it's still the thing your search bar is probably running.

BM25 originates in the Okapi project (Stephen Robertson, Karen Spärck Jones, et al.) and the probabilistic relevance framework; the formulation and non-negative IDF here follow Lucene's BM25Similarity (defaults $k_1 = 1.2$ , $b = 0.75$ ). BM25+ / BM25L are from Lv & Zhai (2011).