2026-07-21 · 7 min · agents · harness · llm · systems · explainer
Compositional generalization is the thing humans do without thinking: given a novel problem, break it into familiar sub-problems, solve each, recombine. Transformers, famously, are unreliable at it — a model that aces 32k-token tasks does not simply keep working at 2M tokens, and one trained to classify Jeopardy questions does not automatically transfer to spam detection. The usual answer is scale: more data, more parameters, and the ragged edges smooth out. Alex Zhang and Omar Khattab make a different argument, and it's a sharp one: the capacity for compositional generalization can live in the harness — the program that wraps the model — rather than in the weights.
Their one-line thesis: "the primary job of the harness should be to carry a higher-level inductive bias that can reduce unfamiliar and complex problems to compositions of simpler ones." Scaling data still matters most, they're careful to say — but "the machinery that we feed that data into and its inductive biases are what will determine the coefficients of that scaling."
The trick: keep every call locally in-distribution
Here is the mechanism that makes it work, and it's worth sitting with. A task can be wildly out-of-distribution as a whole — no model trained on 32k-token inputs has ever seen a 2M-token one — while every individual model call inside it stays comfortably in-distribution. Zhang calls this property locally in-distribution (LID). Drag the length multiplier and watch what it buys you:
The full trajectory is unseen — no model trained on 32k-token tasks has been asked to handle a 2M-token one. But every individual call the RLM makes stays inside the length regime it was trained on, so the pieces keep working even as the whole gets far longer. That is locally in-distribution: the harness, not the weights, carries the generalization.
A base Transformer reads the whole long task in one context window. Past the length regime it trained on, that context is unfamiliar — the model degrades, the phenomenon people now call context rot. The harness that stays LID never puts the model in that position: it decomposes the task so each call sees only a short, familiar slice, and accuracy holds as the task grows.
How the harness does it: RLM
The concrete harness they study is a Recursive Language Model (RLM), and it earns LID with two moves. The first is context offloading. Instead of appending each raw observation — a tool output, a retrieved document, a sub-agent's answer — to the running context, the RLM stores it in a REPL variable and passes only a tiny symbolic handle. The root LM's view stays a short, task-agnostic prefix; the bulk data sits in the environment, peeked at through small probes. Drag the step count and watch the two context sizes diverge:
Appending raw observations makes the root context grow with the task and cross into the regime where models degrade (context rot, past the dashed line). Offloading keeps each observation in a variable and passes only a tiny handle, so the root LM's view stays a short, task-agnostic prefix no matter how big the task gets — the same short prefix it was trained on.
The second move is programmatic sub-agent calling. Sub-agents behave like functions: they run, and their output lands in a REPL variable rather than being spliced back into the caller's context. Zhang stresses these are equal partners — "programmatic sub-calling is equally as important as context offloading" — because together they're what keep the root context from bloating step over step, which is exactly what would drag it out of distribution.
Standard agent patterns — ReAct, CodeAct, and by extension most coding agents — fail LID precisely because they append everything to a growing history. The RLM is the same idea run in reverse: the harness works to keep the model's window small and familiar, and lets the environment hold the state.
It generalizes — and the base model doesn't
The payoff is measured, not asserted. Zhang RL-trains the RLM on short tasks and evaluates on long held-out ones, across six long-context benchmarks. The training signal is short-task reward; the interesting question is whether the eval reward on much longer tasks tracks it.
It does. Training only on short tasks — 150 steps on Qwen3-30B-A3B-Instruct — the RLM generalizes to
tasks 8–32× longer, with eval reward that "more closely matches the train reward on shorter
tasks," while the base Transformer's eval stays flat even as its train reward rises. On MRCRv2,
GraphWalks, and OOLONG the trained 30B RLM approaches or exceeds a frontier GPT-5.5 RLM. Zhang reports
roughly 10× the eval lift for the same train lift versus a vanilla Transformer.
And it isn't only length. In a separate strategy generalization test, the RLM trained on one domain transfers to a completely different one — Jeopardy-style TREC classification to spam/ham; essay similarity to math-problem similarity; Twitter stance detection to error-detection in chat logs. Again the RLM's train reward tracks its eval reward across the domain gap, and again the base Transformer plateaus. The decomposition strategy the harness learns is the thing that transfers, not the surface task.
The take
The reflex in this field is to push every capability into the weights and let scale sort it out. This work is a reminder that where an inductive bias lives is a design choice. A harness that holds each call locally in-distribution turns "solve a 2M-token task" into "solve a sequence of 32k-token tasks," and that reframing is learnable — you can RL-train it on cheap short tasks and watch it generalize to long, unseen, even cross-domain ones. It fits the pattern the other harness pieces on this site keep circling: the layer around the model is not glue. Here it's the part that generalizes.
Source: Language model harnesses are compositional generalizers (Alex L. Zhang, with Omar Khattab), July 2026. Figures are the post's; the two interactives are mine.