# The harness is the generalizer: how a scaffold learns to solve longer, unseen tasks

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/harness-compositional-generalization
> date: 2026-07-21
> tags: agents, harness, llm, systems, explainer
Compositional generalization is the thing humans do without thinking: given a novel problem, break it
into familiar sub-problems, solve each, recombine. Transformers, famously, are *unreliable* at it — a
model that aces 32k-token tasks does not simply keep working at 2M tokens, and one trained to classify
Jeopardy questions does not automatically transfer to spam detection. The usual answer is *scale*:
more data, more parameters, and the ragged edges smooth out. [Alex Zhang and Omar
Khattab](https://alexzhang13.github.io/blog/2026/harness/) make a different argument, and it's a sharp
one: **the capacity for compositional generalization can live in the harness** — the program that wraps
the model — rather than in the weights.

Their one-line thesis: *"the primary job of the harness should be to carry a higher-level inductive
bias that can reduce unfamiliar and complex problems to compositions of simpler ones."* Scaling data
still matters most, they're careful to say — but *"the machinery that we feed that data into and its
inductive biases are what will determine the coefficients of that scaling."*

## The trick: keep every call locally in-distribution

Here is the mechanism that makes it work, and it's worth sitting with. A task can be wildly
out-of-distribution *as a whole* — no model trained on 32k-token inputs has ever seen a 2M-token one —
while every *individual* model call inside it stays comfortably in-distribution. Zhang calls this
property **locally in-distribution (LID)**. Drag the length multiplier and watch what it buys you:

<LidBand />

A base Transformer reads the whole long task in one context window. Past the length regime it trained
on, that context is unfamiliar — the model degrades, the phenomenon people now call *context rot*. The
harness that stays LID never puts the model in that position: it decomposes the task so each call sees
only a short, familiar slice, and accuracy holds as the task grows.

<Figure
  src="/articles/harness-compositional-generalization/fig1.png"
  alt="Left: a task prompt plus reasoning fans out into sub-queries and tool calls, each marked with an eye icon meaning an individual LM call sees only that in-distribution slice. Right: the same content laid out as one long flat sequence that an individual LM would have to read whole, labelled out-of-distribution / unseen."
  caption="Locally in-distribution: decomposed, each call sees a short in-distribution slice (left); flattened into one sequence, the whole thing is unseen (right) (paper, Figure 3)."
/>

## How the harness does it: RLM

The concrete harness they study is a **Recursive Language Model (RLM)**, and it earns LID with two
moves. The first is **context offloading**. Instead of appending each raw observation — a tool output,
a retrieved document, a sub-agent's answer — to the running context, the RLM stores it in a REPL
variable and passes only a tiny symbolic *handle*. The root LM's view stays a short, task-agnostic
prefix; the bulk data sits in the environment, peeked at through small probes. Drag the step count and
watch the two context sizes diverge:

<ContextOffload />

The second move is **programmatic sub-agent calling**. Sub-agents behave like functions: they run,
and their output lands in a REPL variable rather than being spliced back into the caller's context.
Zhang stresses these are equal partners — *"programmatic sub-calling is equally as important as context
offloading"* — because together they're what keep the root context from bloating step over step, which
is exactly what would drag it out of distribution.

<Figure
  src="/articles/harness-compositional-generalization/fig2.png"
  alt="Two side-by-side comparisons. Left: with context offloading the root LM sees a short REPL-based prefix; without it, it sees a large raw context block. Right: with programmatic sub-calling, sub-agent outputs stay out of the root context; without it, every tool and sub-agent output is appended to what the root LM sees."
  caption="The two mechanisms. Offloading keeps a short task-agnostic prefix; programmatic sub-calls keep sub-agent outputs out of the root context entirely — what the root LM sees stays small (paper, Figure 5)."
/>

Standard agent patterns — ReAct, CodeAct, and by extension most coding agents — fail LID precisely
because they append everything to a growing history. The RLM is the same idea run in reverse: the
harness works to *keep the model's window small and familiar*, and lets the environment hold the state.

<Callout type="note">
This is a different claim from the two harness pieces already on this site. [Agent
harnesses](/articles/agent-harness) is about *engineering the loop* — tools, context policy, the
self-improving outer loop. [The harness effect](/articles/harness-effect) is about *token economics* —
same model, cheaper orchestration. This one is about *generalization*: the harness as an inductive bias
you can train, so the scaffold itself learns to solve tasks it never saw.
</Callout>

## It generalizes — and the base model doesn't

The payoff is measured, not asserted. Zhang RL-trains the RLM on **short** tasks and evaluates on long
held-out ones, across six long-context benchmarks. The training signal is short-task reward; the
interesting question is whether the eval reward on much longer tasks *tracks* it.

<Figure
  src="/articles/harness-compositional-generalization/fig3.png"
  alt="Six training-curve panels — MRCRv2, GraphWalks, LongBench-Pro, OOLONG, OOLONG-Pairs, Ada-LEval. In each, the RLM's held-out long-task eval reward climbs with training and approaches the dotted RLM(GPT-5.5) reference line, while the base Transformer with YaRN stays flat and low."
  caption="Train short, evaluate 8–32× longer. The RLM's long-task eval reward (blue) rises with training and approaches frontier RLM(GPT-5.5); the base Transformer + YaRN (orange) flatlines (paper, Figure 6)."
/>

It does. Training only on short tasks — 150 steps on `Qwen3-30B-A3B-Instruct` — the RLM generalizes to
tasks **8–32× longer**, with eval reward that *"more closely matches the train reward on shorter
tasks,"* while the base Transformer's eval stays flat even as its train reward rises. On MRCRv2,
GraphWalks, and OOLONG the trained 30B RLM approaches or exceeds a frontier `GPT-5.5` RLM. Zhang reports
roughly **10× the eval lift for the same train lift** versus a vanilla Transformer.

<BenchBars
  title="Trained on short tasks, evaluated this much longer"
  unit="×"
  bars={[
    { label: "MRCRv2 (64k → 2M)", value: 32, highlight: true },
    { label: "Ada-LEval (8k → 128k)", value: 16 },
    { label: "GraphWalks (128k → 1M)", value: 8 },
    { label: "LongBench-Pro (32k → 256k)", value: 8 },
    { label: "OOLONG (32k → 256k)", value: 8 },
    { label: "OOLONG-Pairs (8k → 32k)", value: 4 },
  ]}
/>

And it isn't only length. In a separate **strategy generalization** test, the RLM trained on one domain
transfers to a completely different one — Jeopardy-style TREC classification to spam/ham; essay
similarity to *math-problem* similarity; Twitter stance detection to error-detection in chat logs.
Again the RLM's train reward tracks its eval reward across the domain gap, and again the base
Transformer plateaus. The decomposition strategy the harness learns is the thing that transfers, not
the surface task.

<Callout type="warning">
It isn't free. The RLM runs **1.5–3× slower** than the base Transformer per sample — multiple LM calls
per step, sub-call latency — and for a couple of benchmarks (MRCRv2) it needed a light *"nudge to
decompose"* to converge on a generalizing strategy rather than a brittle one. Zhang's read: at scale no
supervision should be necessary, but a hint buys sample efficiency.
</Callout>

## The take

The reflex in this field is to push every capability into the weights and let scale sort it out. This
work is a reminder that *where* an inductive bias lives is a design choice. A harness that holds each
call locally in-distribution turns "solve a 2M-token task" into "solve a sequence of 32k-token tasks,"
and that reframing is learnable — you can RL-train it on cheap short tasks and watch it generalize to
long, unseen, even cross-domain ones. It fits the pattern the other harness pieces on this site keep
circling: the layer *around* the model is not glue. Here it's the part that generalizes.

---

*Source: [Language model harnesses are compositional generalizers](https://alexzhang13.github.io/blog/2026/harness/)
(Alex L. Zhang, with Omar Khattab), July 2026. Figures are the post's; the two interactives are mine.*