Contextual Retrieval, with a runnable repro and a browser playground

Anthropic's Contextual Retrieval post has been on my to-read list for a while. It targets the oldest bug in RAG, so I finally sat down, built a dependency-free reproduction, and wired up a browser playground. This is the write-up.

The problem: chunks forget where they came from

Standard RAG splits documents into chunks and indexes each chunk on its own. That destroys context. Anthropic's example is perfect:

The company's revenue grew by 3% over the previous quarter.

Which company? Which quarter? The chunk can't say. A query like "how did ACME's Q2 revenue change?" has no terms to match against that chunk — the words "ACME" and "Q2 2023" live in the document, not the chunk. Both lexical (BM25) and semantic (embedding) retrieval miss it.

The fix: let Claude situate each chunk

Contextual Retrieval prepends a short, chunk-specific context to each chunk before you embed it and before you build the BM25 index — Anthropic calls these Contextual Embeddings and Contextual BM25. The context is generated by Claude, given the whole document. The same chunk becomes:

This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter.

Now the owner and the period are in the chunk, so retrieval can find it.

Watch it happen

Here is the whole pipeline in your browser — the document, its chunks, the chunks with context prepended, then retrieval. The scoring is a real BM25 and a TF-IDF cosine (a stand-in for an embedding model); the contextual prefixes stand in for Claude's output. Step to 4 · retrieve, pick a query, and watch the right chunk climb from the standard column to the contextual one:

Toggle between BM25, embeddings, and their fusion. The pattern holds across all three: the answer chunk is buried on standard chunks and near the top once each chunk carries its context.

The one prompt that does the work

This is the prompt from the post, verbatim. It runs once per chunk, with the full document supplied as context:

<document>
{{WHOLE_DOCUMENT}}
</document>
Here is the chunk we want to situate within the whole document
<chunk>
{{CHUNK_CONTENT}}
</chunk>
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.

The obvious worry is cost: you re-send the whole document once per chunk. Prompt caching kills that. Cache the document once and every chunk in it reads from the cache, which is why Anthropic quotes a one-time $1.02 per million document tokens. The document is the stable prefix, so it takes the cache_control breakpoint; the chunk and instruction vary and come after it:

from anthropic import Anthropic
 
client = Anthropic()
 
CONTEXT_PROMPT = """Here is the chunk we want to situate within the whole document
<chunk>
{chunk}
</chunk>
Please give a short succinct context to situate this chunk within the overall document \
for the purposes of improving search retrieval of the chunk. Answer only with the \
succinct context and nothing else."""
 
def situate(doc: str, chunk: str) -> str:
    resp = client.messages.create(
        # One Claude call per chunk. For a large index you'd typically drop to a
        # cheaper model like claude-haiku-4-5 — which is what Anthropic's ~$1.02 /
        # 1M-doc-token estimate assumes — trading a little context quality for cost.
        model="claude-opus-4-8",
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": [
                # The whole document is the stable prefix: cache it ONCE, then every
                # chunk of this document reads from the cache instead of re-paying for it.
                {"type": "text",
                 "text": f"<document>\n{doc}\n</document>",
                 "cache_control": {"type": "ephemeral"}},
                {"type": "text", "text": CONTEXT_PROMPT.format(chunk=chunk)},
            ],
        }],
    )
    return "".join(b.text for b in resp.content if b.type == "text").strip()

A dependency-free repro

To convince myself the lift was real and not marketing, I wrote a ~180-line pure-stdlib script: naive chunking, BM25 from scratch, a TF-IDF cosine as an embedding stand-in, and reciprocal rank fusion. The retrieval core is small — here is BM25 and the fusion step:

class BM25:
    def score(self, query, i):
        dl, tf, s = len(self.docs[i]), self.tf[i], 0.0
        for t in query:
            if t not in tf:
                continue
            num = tf[t] * (self.k1 + 1)
            den = tf[t] + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
            s += self.idf(t) * num / den
        return s
 
def rrf(rankings, k=60):  # reciprocal rank fusion of BM25 + embedding rankings
    scores = Counter()
    for ranking in rankings:
        for rank, item in enumerate(ranking):
            scores[item] += 1 / (k + rank + 1)
    return [i for i, _ in scores.most_common()]

Then I index the chunks two ways — plain, and with a one-line context prepended — and measure where the correct chunk lands for a couple of "which entity, which period" queries. The actual output:

query                                                method   plain rank  ctx rank
------------------------------------------------------------------------------------
How did ACME Corp revenue change in Q2 2023?         bm25              4         2
How did ACME Corp revenue change in Q2 2023?         emb               4         1
How did ACME Corp revenue change in Q2 2023?         hybrid            4         2
 
What happened to Beta Industries revenue in Q3 2023? bm25              5         2
What happened to Beta Industries revenue in Q3 2023? emb               5         1
What happened to Beta Industries revenue in Q3 2023? hybrid            5         2
 
recall@1 (fraction of queries where the right chunk ranks #1):
  bm25    plain 0/2   contextual 0/2
  emb     plain 0/2   contextual 2/2

Contextualizing the chunks moves the answer from rank 4–5 to rank 1–2 across BM25, embeddings, and their fusion — and embeddings recall@1 goes from 0/2 to 2/2. On this toy corpus fusion lands the answer at #2 rather than #1 (a small-N artifact of RRF), which is a good reminder that the fusion win is an aggregate effect — which is exactly what Anthropic's real evaluation measures.

What the real evaluation found

On Anthropic's benchmark (top-20 retrieval failure rate, i.e. 1 - recall@20), stacking the techniques compounds:

• Contextual Embeddings alone: 5.7% → 3.7% — a 35% cut in the failure rate.
• + Contextual BM25: 5.7% → 2.9% — 49%.
• + reranking: 5.7% → 1.9% — 67%.

The lexical half matters more than you'd guess — BM25 nails exact identifiers (error codes, ticker symbols, function names) that embeddings smear together, so contextualizing both indexes and fusing them beats either alone.

Things worth copying from the post

• Retrieve top-20, not top-5/10. Anthropic found 20 the most performant cut for the final context.
• Rerank the shortlist. Retrieve ~150 candidates, then rerank down to 20 for the answer prompt — that's the step that takes the failure rate from 2.9% to 1.9%.
• Embedding model matters. Gemini and Voyage embeddings were the standouts in their tests.
• Chunking still matters. Size, boundary, and overlap all move the numbers — Contextual Retrieval sits on top of good chunking, it doesn't replace it.
• A domain-tuned context prompt beats the generic one. The template above is a floor, not a ceiling.

The takeaway

The move is almost embarrassingly simple: spend a cheap, cached Claude call per chunk to write down the context a human would need to make sense of it, then index that. It attacks the failure at its source instead of papering over it downstream, it helps lexical and semantic retrieval at the same time, and — as the little repro above shows — you can watch the right chunk climb the rankings the moment the context goes in.

Source: Introducing Contextual Retrieval (Anthropic). The prompt and the 35 / 49 / 67% numbers are theirs; the repro and the browser playground are mine, and the playground's scoring runs entirely client-side.