# My site's chatbot was stuffing 273k tokens into every message. I gave it tools instead.

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/site-agent-dynamic-tools
> date: 2026-07-20
> tags: agents, llm, tools, retrieval, systems
The assistant embedded on this site — the `⌘K` console, `/api/ask`, and the `ask_satyajit`
MCP tool — worked. It was also doing something faintly absurd. Every single request built its
system prompt like this:

```ts
// the old lib/chat.ts, abridged
const sections = []
for (const page of ["about", "resume", "health", "now", "uses", "reading"])
  sections.push(await dataPageMarkdown(page))
for (const item of getAllContent())           // every article, blog, log, digest…
  sections.push(contentMarkdown(item.kind, item.slug))

return [PERSONA, RULES, "=== CONTENT CORPUS ===", sections.join("\n---\n")].join("\n")
```

It pasted **the entire site** into the prompt and let the provider's context cache eat the cost.
That's a defensible move when the corpus is a page or two. Mine isn't anymore:

```text
old full-corpus system prompt:  1,090,854 chars  ≈  273,000 tokens
```

The code even carried a comment — *"if the corpus ever approaches ~100K tokens, switch to
retrieval"* — that reality had quietly sailed past nearly 3× over. Two hundred seventy-three
thousand tokens is past the input window of the model serving it. The chat was running on
whatever survived truncation. Time to do the thing the comment said.

## The fix: retrieve, don't dump

I rebuilt the assistant as a small **tool-using agent**. The system prompt now carries only a
compact **catalog** — every page's `kind/slug`, title, and one-line description, about **5k
tokens** — and the agent pulls the bodies it actually needs through tools. Three of them:

```ts
// lib/chat.ts — the whole tool set
search_content({ query, limit })  // BM25 over the site (ranked, returns section + snippet)
get_content({ kind, slug })       // fetch one page's full markdown, on demand
think({ thought })                // a reasoning scratchpad; records the thought, returns nothing
```

A question now flows: read the catalog → `search_content` (or jump straight to `get_content` if
the catalog already names the page) → read one or two pages → optionally `think` to reconcile
them → answer, with citations. The base prompt is fixed and small; the variable cost is the
one-to-three pages it fetched, not the other sixty-five it didn't.

```text
before:  ~273k tokens, every request, whether relevant or not
after:   ~5k catalog  +  ~2–10k per page actually fetched
```

That's the same **dynamic loading** idea from
[Kimi K3's tool-calling guide](https://platform.kimi.ai/docs/guide/kimi-k3-tool-calling-best-practice) —
their point is that a big upfront payload "eats up context and makes the model more likely to
pick the wrong thing." Kimi loads *tool schemas* on demand; my corpus is the payload, so I load
*content* on demand. Same principle, one layer down.

## Three tools, on purpose

The tool count is a decision, not an accident. Mario Zechner's
[pi coding agent](https://mariozechner.at/posts/2025-11-30-pi-coding-agent/) makes the case that
"four tools are all you need" — `read`, `write`, `edit`, `bash` — and that MCP servers which
"dump their entire tool descriptions into your context on every session" are the anti-pattern.
A read-only site agent needs fewer still: search, fetch, think. Each description is a couple of
sentences. The combined tool surface is under a thousand tokens, so keeping it declared upfront
(rather than lazily loading schemas, which only pays off with dozens of tools) is the right call
here — the honest version of "dynamic tools" for a small surface is *don't have a big one*.

The one genuinely new tool is `think`, from Anthropic's
[think-tool post](https://www.anthropic.com/engineering/claude-think-tool). It does nothing —
literally logs the thought and returns:

```ts
think: tool({
  description: "Think out loud: plan which pages to fetch, or check a draft answer against the sources you read. Records the thought and returns nothing new.",
  inputSchema: z.object({ thought: z.string() }),
  execute: async ({ thought }) => ({ ok: true, thought }),
}),
```

That looks pointless until you watch a multi-step tool run. Between `search_content` and the
final answer, the model has raw tool output sitting in context and no designated place to reason
over it before committing. `think` is that place — a scratchpad that keeps the "what did I just
read, and does it actually answer the question" step from being skipped. Anthropic reports it
buying a large margin on multi-step tool tasks; the appeal for a retrieval agent is exactly that
it makes the model *check the page it fetched* instead of answering from the snippet.

## The loop

The agent loop itself is one call, courtesy of the Vercel AI SDK — `stopWhen` bounds how many
tool round-trips it may take before it has to answer:

```ts
const result = streamText({
  model: chatModel("main"),
  system,                       // persona + rules + the 5k-token catalog
  messages,
  tools: agentTools(),          // search_content, get_content, think
  stopWhen: stepCountIs(6),     // search → read → think → answer is ~4; 6 is headroom
})
```

The same tool set backs all three surfaces — the streaming `⌘K` console (`/api/chat`), the
one-shot `/api/ask`, and the `ask_satyajit` MCP tool — so there's one harness, not three. And
`search_content` is the [Contextual BM25](/blog/site-search-contextual-bm25) engine I'd already
built for the site's `/search`, now doing double duty as the agent's retriever. The pieces
compose: the search work made the harness possible.

<Callout type="warn">
Honest scope. This is retrieval over a small, single-author corpus, not a coding agent — the
lessons transfer but the stakes are lower. `think`'s benefit is task-dependent (Anthropic saw
big gains on policy-heavy multi-step tasks, near-zero on simple ones); on a two-hop lookup it
mostly earns its keep by stopping the model from answering off the snippet. And the retriever is
lexical BM25 — no embeddings yet, so a pure paraphrase with no shared terms can still miss the
right page. The catalog is the safety net there: the model can see every title even when search
comes up short.
</Callout>

## The take

The old design wasn't wrong when it was written — it was wrong at 273k tokens. The rebuild is the
[agent-harness](/articles/agent-harness) framing applied to my own site: the model barely changed,
but the loop, the tool set, and the context policy around it changed completely. Give the agent a
map and three tools and let it fetch, instead of force-feeding it the whole library and hoping the
answer survives the truncation. Retrieve, don't dump.