{"title":"My site's chatbot was stuffing 273k tokens into every message. I gave it tools instead.","description":"The assistant on this site used to paste the entire content corpus — every article, every log — into the system prompt on every request: ~273k tokens, past the model's own context window. I rebuilt it as a small tool-using agent that retrieves on demand: three tools (BM25 search, fetch-a-page, and a think scratchpad), a ~5k-token catalog, and the bodies pulled only when needed. Notes on the Kimi / pi / Anthropic ideas I borrowed.","date":"2026-07-20","tags":["agents","llm","tools","retrieval","systems"],"draft":false,"kind":"blog","slug":"site-agent-dynamic-tools","body":"The assistant embedded on this site — the `⌘K` console, `/api/ask`, and the `ask_satyajit`\nMCP tool — worked. It was also doing something faintly absurd. Every single request built its\nsystem prompt like this:\n\n```ts\n// the old lib/chat.ts, abridged\nconst sections = []\nfor (const page of [\"about\", \"resume\", \"health\", \"now\", \"uses\", \"reading\"])\n  sections.push(await dataPageMarkdown(page))\nfor (const item of getAllContent())           // every article, blog, log, digest…\n  sections.push(contentMarkdown(item.kind, item.slug))\n\nreturn [PERSONA, RULES, \"=== CONTENT CORPUS ===\", sections.join(\"\\n---\\n\")].join(\"\\n\")\n```\n\nIt pasted **the entire site** into the prompt and let the provider's context cache eat the cost.\nThat's a defensible move when the corpus is a page or two. Mine isn't anymore:\n\n```text\nold full-corpus system prompt:  1,090,854 chars  ≈  273,000 tokens\n```\n\nThe code even carried a comment — *\"if the corpus ever approaches ~100K tokens, switch to\nretrieval\"* — that reality had quietly sailed past nearly 3× over. Two hundred seventy-three\nthousand tokens is past the input window of the model serving it. The chat was running on\nwhatever survived truncation. Time to do the thing the comment said.\n\n## The fix: retrieve, don't dump\n\nI rebuilt the assistant as a small **tool-using agent**. The system prompt now carries only a\ncompact **catalog** — every page's `kind/slug`, title, and one-line description, about **5k\ntokens** — and the agent pulls the bodies it actually needs through tools. Three of them:\n\n```ts\n// lib/chat.ts — the whole tool set\nsearch_content({ query, limit })  // BM25 over the site (ranked, returns section + snippet)\nget_content({ kind, slug })       // fetch one page's full markdown, on demand\nthink({ thought })                // a reasoning scratchpad; records the thought, returns nothing\n```\n\nA question now flows: read the catalog → `search_content` (or jump straight to `get_content` if\nthe catalog already names the page) → read one or two pages → optionally `think` to reconcile\nthem → answer, with citations. The base prompt is fixed and small; the variable cost is the\none-to-three pages it fetched, not the other sixty-five it didn't.\n\n```text\nbefore:  ~273k tokens, every request, whether relevant or not\nafter:   ~5k catalog  +  ~2–10k per page actually fetched\n```\n\nThat's the same **dynamic loading** idea from\n[Kimi K3's tool-calling guide](https://platform.kimi.ai/docs/guide/kimi-k3-tool-calling-best-practice) —\ntheir point is that a big upfront payload \"eats up context and makes the model more likely to\npick the wrong thing.\" Kimi loads *tool schemas* on demand; my corpus is the payload, so I load\n*content* on demand. Same principle, one layer down.\n\n## Three tools, on purpose\n\nThe tool count is a decision, not an accident. Mario Zechner's\n[pi coding agent](https://mariozechner.at/posts/2025-11-30-pi-coding-agent/) makes the case that\n\"four tools are all you need\" — `read`, `write`, `edit`, `bash` — and that MCP servers which\n\"dump their entire tool descriptions into your context on every session\" are the anti-pattern.\nA read-only site agent needs fewer still: search, fetch, think. Each description is a couple of\nsentences. The combined tool surface is under a thousand tokens, so keeping it declared upfront\n(rather than lazily loading schemas, which only pays off with dozens of tools) is the right call\nhere — the honest version of \"dynamic tools\" for a small surface is *don't have a big one*.\n\nThe one genuinely new tool is `think`, from Anthropic's\n[think-tool post](https://www.anthropic.com/engineering/claude-think-tool). It does nothing —\nliterally logs the thought and returns:\n\n```ts\nthink: tool({\n  description: \"Think out loud: plan which pages to fetch, or check a draft answer against the sources you read. Records the thought and returns nothing new.\",\n  inputSchema: z.object({ thought: z.string() }),\n  execute: async ({ thought }) => ({ ok: true, thought }),\n}),\n```\n\nThat looks pointless until you watch a multi-step tool run. Between `search_content` and the\nfinal answer, the model has raw tool output sitting in context and no designated place to reason\nover it before committing. `think` is that place — a scratchpad that keeps the \"what did I just\nread, and does it actually answer the question\" step from being skipped. Anthropic reports it\nbuying a large margin on multi-step tool tasks; the appeal for a retrieval agent is exactly that\nit makes the model *check the page it fetched* instead of answering from the snippet.\n\n## The loop\n\nThe agent loop itself is one call, courtesy of the Vercel AI SDK — `stopWhen` bounds how many\ntool round-trips it may take before it has to answer:\n\n```ts\nconst result = streamText({\n  model: chatModel(\"main\"),\n  system,                       // persona + rules + the 5k-token catalog\n  messages,\n  tools: agentTools(),          // search_content, get_content, think\n  stopWhen: stepCountIs(6),     // search → read → think → answer is ~4; 6 is headroom\n})\n```\n\nThe same tool set backs all three surfaces — the streaming `⌘K` console (`/api/chat`), the\none-shot `/api/ask`, and the `ask_satyajit` MCP tool — so there's one harness, not three. And\n`search_content` is the [Contextual BM25](/blog/site-search-contextual-bm25) engine I'd already\nbuilt for the site's `/search`, now doing double duty as the agent's retriever. The pieces\ncompose: the search work made the harness possible.\n\n<Callout type=\"warn\">\nHonest scope. This is retrieval over a small, single-author corpus, not a coding agent — the\nlessons transfer but the stakes are lower. `think`'s benefit is task-dependent (Anthropic saw\nbig gains on policy-heavy multi-step tasks, near-zero on simple ones); on a two-hop lookup it\nmostly earns its keep by stopping the model from answering off the snippet. And the retriever is\nlexical BM25 — no embeddings yet, so a pure paraphrase with no shared terms can still miss the\nright page. The catalog is the safety net there: the model can see every title even when search\ncomes up short.\n</Callout>\n\n## The take\n\nThe old design wasn't wrong when it was written — it was wrong at 273k tokens. The rebuild is the\n[agent-harness](/articles/agent-harness) framing applied to my own site: the model barely changed,\nbut the loop, the tool set, and the context policy around it changed completely. Give the agent a\nmap and three tools and let it fetch, instead of force-feeding it the whole library and hoping the\nanswer survives the truncation. Retrieve, don't dump.\n","readingTimeMins":5,"url":"https://ai.thesatyajit.com/blog/site-agent-dynamic-tools"}