# Agent harnesses: engineering the loop around the model

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/agent-harness
> date: 2026-07-08
> tags: explainer, agents, llm, systems
A base model does one thing: given tokens, predict the next ones. Everything an agent actually *does* — read a repo, run a test, spawn a subagent, decide it is finished — happens in the code wrapped around that model. Lilian Weng calls that wrapper the **harness**, and her post [Harness Engineering for Self-Improvement](https://lilianweng.github.io/posts/2026-07-04-harness/) makes a sharp claim about it: the harness is not glue you can ignore. It is "the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results." Her thesis in one line: "the layer between the raw model and the real-world context seems to be as important as the model's raw intelligence."

That reframes a lot of agent engineering. The interesting design surface is not only the model — it is the loop, the tool set, the context policy, and the evaluator you build around it. This is a walk through her framing.

<Callout type="note">
Weng's post is framed around **recursive self-improvement (RSI)** — the idea, dating to I. J. Good (1965) and his "ultraintelligent machine," that a capable system can improve the machinery that produces it. In modern terms the model doesn't rewrite its own weights; it improves the *training pipeline* and the *deployment system* around itself. The harness is that deployment system, which is why it sits at the center of the story. This article follows her two-part structure: first what a harness *is* (design patterns), then how you *optimize* it (the self-improving outer loop).
</Callout>

## The loop

Strip an agent down and you get a loop. The model emits an action, the harness executes it, the result comes back, the model emits the next action. Weng's simplest picture of it: user input feeds model inference, which either returns a response or calls a tool; tool results feed straight back into the next inference step.

<Figure
  src="/articles/agent-harness/fig2.png"
  alt="A flow diagram: USER INPUT flows into MODEL INFERENCE, which branches to either AGENT RESPONSE or TOOL CALLS; TOOL CALLS loops back into MODEL INFERENCE."
  caption="The minimal agent loop: inference emits either a final response or a tool call, and tool results feed back into the next inference step (Lilian Weng, simplified Codex agent loop)."
/>

For a coding agent that loop takes a concrete, goal-oriented shape: **plan, execute, observe/test, improve, and execute again until the goal is achieved.** Weng draws it as a pipeline — observe the repo, plan, search and read files, edit and write patches, run tests, inspect errors — with a `Done` exit and a repeat arrow when the goal isn't met yet.

<Figure
  src="/articles/agent-harness/fig1.png"
  alt="A left-to-right pipeline of rounded boxes: Observe repo → Plan → Search/read files → Edit/write patches → Run tests → Inspect errors. A green Done box sits above Run tests; a dashed Repeat arrow returns from Inspect errors back toward Plan."
  caption="The coding-harness loop: the agent works a repo the way a developer works an IDE, iterating until the tests pass (Lilian Weng, coding harness loop)."
/>

The static figure shows the shape; stepping through it shows the mechanic. Here is that loop walked over one real task — a failing test — where the first patch is too narrow and the harness has to loop back before the suite goes green. Watch what crosses the boundary at each phase: the model emits a **tool call**, the harness runs it, and the **observation** returns into context.

<HarnessLoop />

Two things are worth pulling out of that trace. First, the model touches nothing directly — every action is a tool call the harness executes, and every result is an observation the harness chooses to feed back. Second, the harness owns the **control flow**: it decides when the loop repeats and when the goal is met and the loop exits. That decision — keep going or stop — is not the model's to make unilaterally, and getting it wrong is a failure mode we'll come back to.

## The tools are the harness

If the loop is the skeleton, the tool set is the muscle. Weng's tour of a modern coding harness is essentially a table of tool categories, and the range is the point — this is a lot more than "call a function":

| Category | Representative tools |
|---|---|
| **File system** | discovery: `glob`, `grep`, `ls` · read: `read`, `read_many` · modify: `write`, `edit`, `multi_edit`, `apply_patch` |
| **Shell execution** | `bash`, `PowerShell` |
| **IO / repo** | `lsp`, `git_status`, `git_diff`, `git_commit` |
| **External context** | MCP tools, Skills |
| **Web** | `web_search`, `web_fetch`, browser tools |
| **Artifacts** | read docs/images; generate HTML/images |
| **Backend processes** | `CronCreate`, `CronDelete`, `CronList` |
| **Agent delegation** | `spawn_agent`, `resume_agent`, `wait_agent`, `list_agents`, `close_agent` |

A design note runs through the whole list: the tools are "deliberately simple and generic to enable generalization." They lean on primitives a developer already knows — a file system, a shell, git — rather than bespoke abstractions. That matters because "learning how to read, write, and edit the file system (commonly via `bash` commands) is a foundation skill for LLMs." The model has seen a million shell sessions in training; give it a shell and it already knows the idiom. The last two rows — backend processes and agent delegation — are where a harness stops being a single loop and becomes a small operating system: it can schedule work and fork subagents, which is what makes long-horizon and parallel tasks tractable.

## Context is the scarce resource

Here is the constraint that shapes everything else. A long task produces "experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories" that "often grow much longer than the context window that the model has trained for." You cannot keep the whole trajectory in the prompt. So Weng's rule is blunt: "a harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files."

That single decision — context as a bounded working set, disk as the durable store — is what keeps a long task from strangling on its own history. Step through a run and watch the two strategies diverge:

<ContextLedger />

The naive strategy appends everything and eventually overflows the window; past that point the model is quietly losing the early details. The file-backed harness keeps context flat and spills the history to disk, where it stays retrievable with `grep` and `read` — the same file tools from the table above, now doing double duty as memory. This is why file-system fluency is the load-bearing skill: the file system *is* the agent's long-term memory.

The same principle governs parallelism. Weng's guidance is to make it "explicit and inspectable" — store subagent outputs as "files, logs, and status records" rather than transient chat contexts, so the system can "recover after interruptions and reason over its own execution history." Durable-state-in-files isn't only a context trick; it's what makes an agent restartable.

<Callout type="tip">
There's a research lineage here worth naming. **Agentic Context Engineering (ACE)** maintains a "context playbook" of itemized bullet points — each with an identifier and description — updated by a Generator / Reflector / Curator trio that appends *structured entries* instead of rewriting the whole prompt. **Meta Context Engineering (MCE)** goes one level up, separating "mechanism (how to manage context) from artifact content (what is in context)." Both are the same instinct as keeping state in files: treat context as a managed, structured store, not an ever-growing transcript.
</Callout>

## Guardrails live outside the loop

Give an agent `bash`, `edit`, and the ability to spawn more agents and you've handed it a lot of reach. Weng is direct that this breaks abstraction boundaries: when programs can edit the systems they run on, you need a "proper design of editable surface" with "permission control and security layers outside this loop." The guardrail is deliberately *not* another prompt instruction inside the model's context — it's an enforcement layer the model cannot talk its way past. That placement is the whole point: a permission check the agent can edit is not a permission check.

## Optimizing the harness

The second half of Weng's post asks the recursive question: if the harness matters this much, can the agent improve *its own* harness? That turns the inner task loop into an **outer loop** over harness designs — run the current harness, mine where it failed, propose edits, keep the ones that survive a regression test.

<Figure
  src="/articles/agent-harness/fig3.png"
  alt="A four-stage cycle. Weakness Mining: run the current harness on tasks, collect execution traces, cluster failure patterns. Harness Proposal: use the failures to propose harness edits like validate-before-conclude, a loop-breaker, a tool-policy update. Proposal Validation: run a regression test and accept or reject each edit. Accepted edits produce an Updated Harness that proceeds to the next iteration; if all are rejected, no update."
  caption="Self-Harness: an outer loop that mines failure patterns from traces, proposes harness edits, and promotes only the ones that pass a regression test (Lilian Weng, Self-Harness loop)."
/>

This is one instance of a broader family the post surveys — **ADAS**, **AFlow**, **STOP**, **AlphaEvolve**, the **Darwin Gödel Machine** — all variations on "search over the scaffolding, not the weights." The headline result is that it works: the Darwin Gödel Machine's discovered agents went from **20% → 50%** on SWE-bench Verified and **14.2% → 30.7%** on Polyglot, matching or beating handcrafted agents.

<BenchBars
  title="Darwin Gödel Machine — starting agent vs. self-discovered agent"
  unit="%"
  bars={[
    { label: "SWE-bench · start", value: 20 },
    { label: "SWE-bench · discovered", value: 50, highlight: true },
    { label: "Polyglot · start", value: 14.2 },
    { label: "Polyglot · discovered", value: 30.7, highlight: true },
  ]}
/>

But the honest caveat is the more instructive part. **STOP** improved performance when the base model was GPT-4 and *degraded* it with weaker models (GPT-3.5, Mixtral). Weng's reading: "recursive structure alone is not enough. The base model must be capable enough to improve the mechanism." Self-improvement is not free lift from the loop; it's a gain that depends on a model already good enough to reason about its own scaffolding. Below that bar, the outer loop makes things worse.

## Failure modes

The post catalogs where autonomous agents actually break, drawing on an analysis (Trehan & Chopra, 2026) of auto-research attempts. Six recur:

1. **Training-data defaults.** The model reaches for old libraries, stale commands, and standard formats instead of what the actual repo uses.
2. **Implementation drift.** When the proposed method gets complex, the model quietly slides toward a simpler solution than the one it was asked for.
3. **Memory degradation.** Long-horizon projects lose critical details — unless the logs were written out as persistent artifacts. (The file-system point again, stated as a failure when you skip it.)
4. **Over-optimism.** The model declares success on noisy or failed experiments — a pattern Weng names "p-hacking and eureka-ing."
5. **Insufficient domain intelligence.** It lacks the tacit craft knowledge to judge whether a result is even plausible.
6. **Weak scientific taste.** The experiments run fine but fail to answer the right question.

Notice how many of these the *harness* is supposed to catch rather than the model: memory degradation is a context-policy failure, over-optimism is an evaluator failure, drift is a control-flow failure. The whole post is an argument that these are engineering problems in the wrapper, not just intelligence gaps in the core.

## What's still hard

Weng closes with the bottlenecks between here and genuine self-improvement, and they read as an honest problem list rather than a roadmap:

- **Weak fuzzy evaluators.** "Many research claims don't have a fast/precise verifier." Taste and novelty are far harder to score than a passing test suite, and an outer loop is only as good as its evaluator.
- **Context and memory lifecycle.** Managing context growth over long autonomous runs is becoming "a core part of intelligence," not a plumbing detail.
- **Negative results.** LLMs are biased toward success and struggle to abandon a hypothesis, because their training data is skewed toward things that worked.
- **Diversity collapse.** Evolutionary and RL loops exploit known high-reward patterns; without pressure for diversity the population collapses into variants of one solution.
- **Reward hacking.** A self-improvement loop optimizes the signal it's given — including benchmark artifacts and vulnerabilities in a judge model.
- **Long-term success.** Short-horizon optimization ignores maintainability, ownership boundaries, migration cost, backwards compatibility, and debugging burden — the things that decide whether real systems survive.
- **The human role.** Her framing is that "humans should move up the stack, not be removed from the loop" — providing oversight "at the right time, the right abstraction level," not disappearing from it.

## The take

The useful shift in Weng's framing is where it puts the design surface. Agent quality is not only a function of the model you call; it's a function of the loop you wrap it in, the tools you expose, the context policy you enforce, and the evaluator you trust. Those are engineering decisions, and most of them are decisions about *state* — what stays in context, what spills to disk, what the permission layer refuses, what the regression test has to pass before a change ships. The honest bounds are stated plainly too: self-improving harnesses only help above a base-model capability threshold, the outer loop is only as good as a verifier we mostly don't have for fuzzy goals, and reward hacking and diversity collapse are unsolved. Which lands on a pragmatic note — the harness is where a lot of the near-term gains are, and it's ordinary systems engineering: files, permissions, control flow, and tests, applied to a model instead of a service.

---

*Built on Lilian Weng's [Harness Engineering for Self-Improvement](https://lilianweng.github.io/posts/2026-07-04-harness/) (2026). Quotations and the three figures are reproduced from that post for commentary; the interactive loop and context-budget diagrams are my own illustrations of the mechanism, not measured traces. Benchmark numbers (Darwin Gödel Machine, STOP) are as reported in her post.*