{"title":"Giving my own search the Contextual BM25 treatment","description":"This site's search was a substring indexOf — first match wins, no ranking, and any multi-word query that isn't a verbatim phrase returns nothing. I replaced it with the two things I'd just written up: BM25 for ranking, and Contextual Retrieval for chunks — except the context is deterministic frontmatter instead of a Claude call, so it runs at request time with no model. Here's the rebuild, with the real before/after.","date":"2026-07-20","tags":["search","retrieval","bm25","rag","information-retrieval"],"draft":false,"kind":"blog","slug":"site-search-contextual-bm25","body":"I recently wrote two pieces back to back: one on [BM25](/articles/bm25), the ranking\nfunction that refuses to die, and one on [Contextual Retrieval](/blog/contextual-retrieval),\nAnthropic's fix for chunks that forget where they came from. Then I opened `lib/search.ts` —\nthe thing powering this site's own `/search` and the `/api/search` endpoint agents hit — and\nfound this:\n\n```ts\n// the old lib/search.ts, abridged\nfor (const item of getAllContent()) {\n  const fields = [item.title, item.description, item.tags, item.body]\n  for (const text of fields) {\n    if (text.toLowerCase().includes(q)) {   // q = the whole lowercased query\n      results.push({ /* … a ±120-char window around the hit */ })\n      break\n    }\n  }\n}\n```\n\nA substring `indexOf`. It has three problems, and the third is the one that actually bites:\n\n1. **No ranking.** The first document that contains the substring wins, in date order. A tight,\n   on-topic match and an incidental mention are indistinguishable.\n2. **No notion of rarity or length.** Matching \"the\" counts the same as matching \"kalman\".\n3. **Multi-word queries fall off a cliff.** `q` is the *entire* query string, so\n   `includes(q)` needs that **exact phrase** somewhere in the text. Nobody searches in verbatim\n   phrases. `\"bm25 length normalization\"` returns **zero results** — those three words are all\n   in the BM25 article, just never contiguous.\n\nSo I ate my own cooking. Here's the rebuild.\n\n## BM25, the same formula I just wrote up\n\nThe [BM25 walkthrough](/articles/bm25) has the whole derivation; the code is a direct\ntranscription of it. Lucene's non-negative IDF, term-frequency saturation at `k1 = 1.2`, length\nnormalization at `b = 0.75`:\n\n```ts\n// lib/search.ts — the ranking core, straight from the article's formula\nconst K1 = 1.2\nconst B = 0.75\n\nfunction idf(term: string, ix: Index): number {\n  const n = ix.df.get(term) ?? 0\n  return Math.log(1 + (ix.n - n + 0.5) / (n + 0.5))\n}\n\n// per query term present in a chunk:\nconst f = chunk.tf.get(term) ?? 0\nconst norm = K1 * (1 - B + (B * chunk.len) / ix.avgdl)\nscore += idf(term, ix) * (f * (K1 + 1)) / (f + norm)\n```\n\nTwo things fall out for free. The query is **tokenized** into terms and each is scored\nindependently, so multi-word queries just work — no phrase has to exist. And rare terms\ndominate: `kalman` carries far more weight than `filter`, because `idf` collapses for common\nwords. No stopword list, same as the article promised.\n\nI keep a **postings map** (`term → chunk indices`) so a query only touches chunks that actually\nshare a term, instead of scanning the whole corpus. The corpus is one person's writing, so this\nis overkill — but it's the same inverted-index shape a real engine uses, and it's three lines.\n\n## Contextual chunks, minus the Claude call\n\nBM25 alone still has the chunk-amnesia problem from the [Contextual Retrieval\npost](/blog/contextual-retrieval): if I split an article into passages and index each one alone,\na passage that reads \"it drops 41% at parity\" has no idea it's *about the Harness Effect, in the\nsection on the controlled swap*. A query for either misses it.\n\nAnthropic's fix is to have Claude write a one-line context for every chunk before indexing.\nMine is cheaper and dumber: the context a chunk lost is sitting right there in the document's\nfrontmatter and headings. So before indexing, every chunk inherits its document's **title,\ndescription, tags, and nearest heading** — deterministically, at request time, no model, no\nbuild step:\n\n```ts\n// each chunk is packed with weighted terms: its own body, its heading, and the\n// document context it would otherwise have lost when the body was split.\nconst contextTokens = tokenize([title, description, tags.join(\" \")].join(\" \"))\nfor (const t of bodyTokens) add(t, 1)\nfor (const t of headingTokens) add(t, 1)          // section-local: full weight\nfor (const t of contextTokens) add(t, CONTEXT_WEIGHT) // 0.5 — present, not dominant\n```\n\nThe honest caveat: this is a **poor man's** Contextual Retrieval. Claude's per-chunk context can\nsay things the frontmatter can't (\"the previous quarter's revenue was \\$314M\"); my version can\nonly replay the structural context the document already carries. And prepending the *same* title\nto every chunk of a document inflates those terms' document-frequency, which is exactly why the\ncontext terms get `CONTEXT_WEIGHT = 0.5` instead of full weight — present enough to make the\nchunk findable by its subject, quiet enough not to drown the passage's own words. It's the 80%\nof the win for 0% of the inference cost, which is the right trade for a static personal site.\n\nEach result also reports **which section** it matched — the nearest heading rides along as the\nresult's `field`, so `/api/search` tells an agent not just *which* document but *where* in it.\n\n## Before / after\n\nSame queries, old substring search vs. the new Contextual BM25, on the live corpus:\n\n```text\nquery                          substring indexOf         contextual BM25 (top hit)\n--------------------------------------------------------------------------------------\n\"bm25 length normalization\"    0 results (no verbatim     articles/bm25\n                                phrase)                    [Start with TF-IDF, and its two flaws]\n\n\"revenue grew quarter\"         0 results                  blog/contextual-retrieval\n                                                           [The problem: chunks forget …]\n\n\"kalman filter\"                first dated doc that        articles/kalman-filter  (8.4)\n                                contains the phrase,        + fast-lio2 (8.6, cites it)\n                                unranked                    ranked by relevance, not date\n\n\"reciprocal rank fusion\"       0 results                  blog/contextual-retrieval\n                                                           [A dependency-free repro]\n```\n\nThe multi-word queries are the story. Under substring, anything that isn't a verbatim phrase —\nwhich is almost everything a person types — returned nothing. Under BM25 every term contributes,\nso the query finds the document even when its words are scattered across a paragraph, and the\ncontextual prefix pulls in matches on the *subject* of a passage, not just its literal words.\n\n## It's live\n\nTry it: [`/search?q=length+normalization`](/search?q=length+normalization). Agents get the\nranked, scored JSON with the section label:\n\n```bash\ncurl -s \"https://ai.thesatyajit.com/api/search?q=contextual%20retrieval&limit=3\" | jq '.results[] | {slug, field, score}'\n```\n\n<Callout type=\"warn\">\nWhat this still isn't: it's the **lexical half** only. The [contextual retrieval\npost](/blog/contextual-retrieval) makes the case that the real win is *fusing* BM25 with a dense\nretriever, and that reranking the shortlist is what takes the failure rate the last mile. There\nare no embeddings here yet — a query has to share actual terms with a chunk, so a pure paraphrase\nwith no lexical overlap can still miss. For a corpus this size, lexical BM25 over contextualized\nchunks is the honest 90% solution; the semantic half is the next commit, not this one.\n</Callout>\n\n## The takeaway\n\nThe whole change is `lib/search.ts` — a few hundred lines, no dependencies, no index server, no\nmodel at request time. It's the two ideas I'd just written about, applied to the smallest\npossible target: my own search bar. BM25 for ranking, structural context for the chunk-amnesia\nproblem, and an honest note about the half I haven't built yet. Writing about a technique is a\ngood way to understand it; running it in your own site is a better one.\n","readingTimeMins":6,"url":"https://ai.thesatyajit.com/blog/site-search-contextual-bm25"}