2026-07-08 · 12 min · explainer · agents · llm · systems
A base model does one thing: given tokens, predict the next ones. Everything an agent actually does — read a repo, run a test, spawn a subagent, decide it is finished — happens in the code wrapped around that model. Lilian Weng calls that wrapper the harness, and her post Harness Engineering for Self-Improvement makes a sharp claim about it: the harness is not glue you can ignore. It is "the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results." Her thesis in one line: "the layer between the raw model and the real-world context seems to be as important as the model's raw intelligence."
That reframes a lot of agent engineering. The interesting design surface is not only the model — it is the loop, the tool set, the context policy, and the evaluator you build around it. This is a walk through her framing.
The loop
Strip an agent down and you get a loop. The model emits an action, the harness executes it, the result comes back, the model emits the next action. Weng's simplest picture of it: user input feeds model inference, which either returns a response or calls a tool; tool results feed straight back into the next inference step.
For a coding agent that loop takes a concrete, goal-oriented shape: plan, execute, observe/test, improve, and execute again until the goal is achieved. Weng draws it as a pipeline — observe the repo, plan, search and read files, edit and write patches, run tests, inspect errors — with a Done exit and a repeat arrow when the goal isn't met yet.
The static figure shows the shape; stepping through it shows the mechanic. Here is that loop walked over one real task — a failing test — where the first patch is too narrow and the harness has to loop back before the suite goes green. Watch what crosses the boundary at each phase: the model emits a tool call, the harness runs it, and the observation returns into context.
New task: the suite has a failing test. Get the lay of the land before touching anything.
The model never edits the repo directly. Each phase emits a tool call the harness runs, and each result is an observation fed back into the next generation. The harness owns the control flow: it decides when to loop back and when the goal is met and the loop exits.
Two things are worth pulling out of that trace. First, the model touches nothing directly — every action is a tool call the harness executes, and every result is an observation the harness chooses to feed back. Second, the harness owns the control flow: it decides when the loop repeats and when the goal is met and the loop exits. That decision — keep going or stop — is not the model's to make unilaterally, and getting it wrong is a failure mode we'll come back to.
The tools are the harness
If the loop is the skeleton, the tool set is the muscle. Weng's tour of a modern coding harness is essentially a table of tool categories, and the range is the point — this is a lot more than "call a function":
| Category | Representative tools |
|---|---|
| File system | discovery: glob, grep, ls · read: read, read_many · modify: write, edit, multi_edit, apply_patch |
| Shell execution | bash, PowerShell |
| IO / repo | lsp, git_status, git_diff, git_commit |
| External context | MCP tools, Skills |
| Web | web_search, web_fetch, browser tools |
| Artifacts | read docs/images; generate HTML/images |
| Backend processes | CronCreate, CronDelete, CronList |
| Agent delegation | spawn_agent, resume_agent, wait_agent, list_agents, close_agent |
A design note runs through the whole list: the tools are "deliberately simple and generic to enable generalization." They lean on primitives a developer already knows — a file system, a shell, git — rather than bespoke abstractions. That matters because "learning how to read, write, and edit the file system (commonly via bash commands) is a foundation skill for LLMs." The model has seen a million shell sessions in training; give it a shell and it already knows the idiom. The last two rows — backend processes and agent delegation — are where a harness stops being a single loop and becomes a small operating system: it can schedule work and fork subagents, which is what makes long-horizon and parallel tasks tractable.
Context is the scarce resource
Here is the constraint that shapes everything else. A long task produces "experiment logs, code diffs, paper summaries, error traces, and past rollout trajectories" that "often grow much longer than the context window that the model has trained for." You cannot keep the whole trajectory in the prompt. So Weng's rule is blunt: "a harness should not carry the entire workflow and all logs in context; instead, it should keep durable state in files."
That single decision — context as a bounded working set, disk as the durable store — is what keeps a long task from strangling on its own history. Step through a run and watch the two strategies diverge:
The naive prompt appends every log, diff, and trace, so it grows with the task and overflows the window by turn 7 — the model starts losing the early details. The file-backed harness keeps a bounded working set in context and spills the growing history to disk, where it stays retrievable with grep and read. Same task, flat context cost.
The naive strategy appends everything and eventually overflows the window; past that point the model is quietly losing the early details. The file-backed harness keeps context flat and spills the history to disk, where it stays retrievable with grep and read — the same file tools from the table above, now doing double duty as memory. This is why file-system fluency is the load-bearing skill: the file system is the agent's long-term memory.
The same principle governs parallelism. Weng's guidance is to make it "explicit and inspectable" — store subagent outputs as "files, logs, and status records" rather than transient chat contexts, so the system can "recover after interruptions and reason over its own execution history." Durable-state-in-files isn't only a context trick; it's what makes an agent restartable.
Guardrails live outside the loop
Give an agent bash, edit, and the ability to spawn more agents and you've handed it a lot of reach. Weng is direct that this breaks abstraction boundaries: when programs can edit the systems they run on, you need a "proper design of editable surface" with "permission control and security layers outside this loop." The guardrail is deliberately not another prompt instruction inside the model's context — it's an enforcement layer the model cannot talk its way past. That placement is the whole point: a permission check the agent can edit is not a permission check.
Optimizing the harness
The second half of Weng's post asks the recursive question: if the harness matters this much, can the agent improve its own harness? That turns the inner task loop into an outer loop over harness designs — run the current harness, mine where it failed, propose edits, keep the ones that survive a regression test.
This is one instance of a broader family the post surveys — ADAS, AFlow, STOP, AlphaEvolve, the Darwin Gödel Machine — all variations on "search over the scaffolding, not the weights." The headline result is that it works: the Darwin Gödel Machine's discovered agents went from 20% → 50% on SWE-bench Verified and 14.2% → 30.7% on Polyglot, matching or beating handcrafted agents.
But the honest caveat is the more instructive part. STOP improved performance when the base model was GPT-4 and degraded it with weaker models (GPT-3.5, Mixtral). Weng's reading: "recursive structure alone is not enough. The base model must be capable enough to improve the mechanism." Self-improvement is not free lift from the loop; it's a gain that depends on a model already good enough to reason about its own scaffolding. Below that bar, the outer loop makes things worse.
Failure modes
The post catalogs where autonomous agents actually break, drawing on an analysis (Trehan & Chopra, 2026) of auto-research attempts. Six recur:
- Training-data defaults. The model reaches for old libraries, stale commands, and standard formats instead of what the actual repo uses.
- Implementation drift. When the proposed method gets complex, the model quietly slides toward a simpler solution than the one it was asked for.
- Memory degradation. Long-horizon projects lose critical details — unless the logs were written out as persistent artifacts. (The file-system point again, stated as a failure when you skip it.)
- Over-optimism. The model declares success on noisy or failed experiments — a pattern Weng names "p-hacking and eureka-ing."
- Insufficient domain intelligence. It lacks the tacit craft knowledge to judge whether a result is even plausible.
- Weak scientific taste. The experiments run fine but fail to answer the right question.
Notice how many of these the harness is supposed to catch rather than the model: memory degradation is a context-policy failure, over-optimism is an evaluator failure, drift is a control-flow failure. The whole post is an argument that these are engineering problems in the wrapper, not just intelligence gaps in the core.
What's still hard
Weng closes with the bottlenecks between here and genuine self-improvement, and they read as an honest problem list rather than a roadmap:
- Weak fuzzy evaluators. "Many research claims don't have a fast/precise verifier." Taste and novelty are far harder to score than a passing test suite, and an outer loop is only as good as its evaluator.
- Context and memory lifecycle. Managing context growth over long autonomous runs is becoming "a core part of intelligence," not a plumbing detail.
- Negative results. LLMs are biased toward success and struggle to abandon a hypothesis, because their training data is skewed toward things that worked.
- Diversity collapse. Evolutionary and RL loops exploit known high-reward patterns; without pressure for diversity the population collapses into variants of one solution.
- Reward hacking. A self-improvement loop optimizes the signal it's given — including benchmark artifacts and vulnerabilities in a judge model.
- Long-term success. Short-horizon optimization ignores maintainability, ownership boundaries, migration cost, backwards compatibility, and debugging burden — the things that decide whether real systems survive.
- The human role. Her framing is that "humans should move up the stack, not be removed from the loop" — providing oversight "at the right time, the right abstraction level," not disappearing from it.
The take
The useful shift in Weng's framing is where it puts the design surface. Agent quality is not only a function of the model you call; it's a function of the loop you wrap it in, the tools you expose, the context policy you enforce, and the evaluator you trust. Those are engineering decisions, and most of them are decisions about state — what stays in context, what spills to disk, what the permission layer refuses, what the regression test has to pass before a change ships. The honest bounds are stated plainly too: self-improving harnesses only help above a base-model capability threshold, the outer loop is only as good as a verifier we mostly don't have for fuzzy goals, and reward hacking and diversity collapse are unsolved. Which lands on a pragmatic note — the harness is where a lot of the near-term gains are, and it's ordinary systems engineering: files, permissions, control flow, and tests, applied to a model instead of a service.
Built on Lilian Weng's Harness Engineering for Self-Improvement (2026). Quotations and the three figures are reproduced from that post for commentary; the interactive loop and context-budget diagrams are my own illustrations of the mechanism, not measured traces. Benchmark numbers (Darwin Gödel Machine, STOP) are as reported in her post.