My site's chatbot was stuffing 273k tokens into every message. I gave it tools instead.

The assistant embedded on this site — the ⌘K console, /api/ask, and the ask_satyajit MCP tool — worked. It was also doing something faintly absurd. Every single request built its system prompt like this:

// the old lib/chat.ts, abridged
const sections = []
for (const page of ["about", "resume", "health", "now", "uses", "reading"])
  sections.push(await dataPageMarkdown(page))
for (const item of getAllContent())           // every article, blog, log, digest…
  sections.push(contentMarkdown(item.kind, item.slug))
 
return [PERSONA, RULES, "=== CONTENT CORPUS ===", sections.join("\n---\n")].join("\n")

It pasted the entire site into the prompt and let the provider's context cache eat the cost. That's a defensible move when the corpus is a page or two. Mine isn't anymore:

old full-corpus system prompt:  1,090,854 chars  ≈  273,000 tokens

The code even carried a comment — "if the corpus ever approaches ~100K tokens, switch to retrieval" — that reality had quietly sailed past nearly 3× over. Two hundred seventy-three thousand tokens is past the input window of the model serving it. The chat was running on whatever survived truncation. Time to do the thing the comment said.

The fix: retrieve, don't dump

I rebuilt the assistant as a small tool-using agent. The system prompt now carries only a compact catalog — every page's kind/slug, title, and one-line description, about 5k tokens — and the agent pulls the bodies it actually needs through tools. Three of them:

// lib/chat.ts — the whole tool set
search_content({ query, limit })  // BM25 over the site (ranked, returns section + snippet)
get_content({ kind, slug })       // fetch one page's full markdown, on demand
think({ thought })                // a reasoning scratchpad; records the thought, returns nothing

A question now flows: read the catalog → search_content (or jump straight to get_content if the catalog already names the page) → read one or two pages → optionally think to reconcile them → answer, with citations. The base prompt is fixed and small; the variable cost is the one-to-three pages it fetched, not the other sixty-five it didn't.

before:  ~273k tokens, every request, whether relevant or not
after:   ~5k catalog  +  ~2–10k per page actually fetched

That's the same dynamic loading idea from Kimi K3's tool-calling guide — their point is that a big upfront payload "eats up context and makes the model more likely to pick the wrong thing." Kimi loads tool schemas on demand; my corpus is the payload, so I load content on demand. Same principle, one layer down.

Three tools, on purpose

The tool count is a decision, not an accident. Mario Zechner's pi coding agent makes the case that "four tools are all you need" — read, write, edit, bash — and that MCP servers which "dump their entire tool descriptions into your context on every session" are the anti-pattern. A read-only site agent needs fewer still: search, fetch, think. Each description is a couple of sentences. The combined tool surface is under a thousand tokens, so keeping it declared upfront (rather than lazily loading schemas, which only pays off with dozens of tools) is the right call here — the honest version of "dynamic tools" for a small surface is don't have a big one.

The one genuinely new tool is think, from Anthropic's think-tool post. It does nothing — literally logs the thought and returns:

think: tool({
  description: "Think out loud: plan which pages to fetch, or check a draft answer against the sources you read. Records the thought and returns nothing new.",
  inputSchema: z.object({ thought: z.string() }),
  execute: async ({ thought }) => ({ ok: true, thought }),
}),

That looks pointless until you watch a multi-step tool run. Between search_content and the final answer, the model has raw tool output sitting in context and no designated place to reason over it before committing. think is that place — a scratchpad that keeps the "what did I just read, and does it actually answer the question" step from being skipped. Anthropic reports it buying a large margin on multi-step tool tasks; the appeal for a retrieval agent is exactly that it makes the model check the page it fetched instead of answering from the snippet.

The loop

The agent loop itself is one call, courtesy of the Vercel AI SDK — stopWhen bounds how many tool round-trips it may take before it has to answer:

const result = streamText({
  model: chatModel("main"),
  system,                       // persona + rules + the 5k-token catalog
  messages,
  tools: agentTools(),          // search_content, get_content, think
  stopWhen: stepCountIs(6),     // search → read → think → answer is ~4; 6 is headroom
})

The same tool set backs all three surfaces — the streaming ⌘K console (/api/chat), the one-shot /api/ask, and the ask_satyajit MCP tool — so there's one harness, not three. And search_content is the Contextual BM25 engine I'd already built for the site's /search, now doing double duty as the agent's retriever. The pieces compose: the search work made the harness possible.

The take

The old design wasn't wrong when it was written — it was wrong at 273k tokens. The rebuild is the agent-harness framing applied to my own site: the model barely changed, but the loop, the tool set, and the context policy around it changed completely. Give the agent a map and three tools and let it fetch, instead of force-feeding it the whole library and hoping the answer survives the truncation. Retrieve, don't dump.