MemHarness: agent memory should be reconstructed, not replayed

2026-08-03 · 16 min · agents · memory · reinforcement-learning · llm · explainer

Most memory-augmented agents treat a retrieved experience the way a tape recorder treats a cassette: press play, get back exactly what was stored. MemHarness, from a Zhejiang University / Shanghai AI Lab team, argues that's the wrong model of memory entirely — and points at cognitive science to say so. Human recall isn't playback; it's reconstruction, rebuilt each time from fragments and reshaped to fit the moment. Their agent does the same: before acting on a retrieved memory, it first critiques that memory against what it's looking at right now, and rewrites it if the two don't match.

The failure mode: negative transfer from a memory that no longer fits

Retrieval-augmented agents work like this: finish a task, distill what happened into a short natural-language "experience," store it, and next time a similar task comes up, pull the closest matches back into context. The problem is closest is doing a lot of work. An experience learned from one kitchen layout, one inventory state, one shopping page, gets pasted into a context where the fridge is already full or the shelf has something else on it — and the agent, having no reason to doubt its own memory, follows the stale instruction anyway. MemHarness calls this the "replay" paradigm, and its central claim is that replay's failures are systematic, not occasional: the retrieved experience is abstract and general by construction, while the state at decision time is concrete and constantly changing, and nothing in a replay pipeline reconciles the two.

Three stacked panels. Top, 'Previous: Replay Retrieved Memory' — a memory store retrieves a memory and a history state, both injected unchanged into an input context alongside the task, with a warning that the memory may not align with the current state. Middle, 'Human Memory: Not just Replay, But Reconstruction' — a brain retrieves fragmented past experience as puzzle pieces and reconstructs them, informed by current state and knowledge, into an informed action. Bottom, 'MemHarness: Reconstruct Memory for the Current State' — the same retrieval happens, but the retrieved memory and current state pass through the agent, which critiques, aligns, and adapts the memory before it enters the final input context as reconstructed, aligned memory. — The three paradigms MemHarness is contrasting: naive verbatim replay, human reconstructive memory, and MemHarness's learned reconstruction step in between retrieval and action (Wu et al., 2026, Figure 1).

This isn't a new observation for this site — Agent harnesses: engineering the loop around the model already named "context and memory lifecycle" as one of the open problems in agent engineering, and pointed out that a file-backed harness effectively turns the file system into the agent's long-term memory. MemHarness is a concrete answer to a sharper version of that problem: it's not enough to store memory durably, the harness also has to decide, at read time, whether a piece of stored memory still applies — and if not, what to do about it. MemHarness's answer is to make that decision itself a trained, learned skill rather than a fixed retrieval-and-paste rule.

Three stages: retrieve, reconstruct, act

Formally, the agent holds a memory bank $\mathcal{B} = \{m_i\}_{i=1}^N$ of entries $m_i = (e_i, o_i^{src})$ — an abstracted experience $e_i$ paired with the source observation $o_i^{src}$ it was distilled from. At each step $t$ , retrieval returns the top- $k$ closest entries, $\mathcal{E}_t = \mathcal{R}(q_t, \mathcal{B})$ . A pure replay policy conditions the next action directly on whatever comes back:

a_t \sim \pi_\theta(\cdot \mid \mathcal{T}, h_t, \mathcal{E}_t)

MemHarness inserts one step in between. The same policy first produces guidance $g_t$ by critiquing the retrieved experiences against the current history $h_t$ , then maps that into final guidance $\tilde{g}_t$ , and only then generates the action:

g_t \sim \pi_\theta(\cdot \mid \mathcal{T}, h_t, \mathcal{E}_t), \qquad \tilde{g}_t = f(g_t), \qquad a_t \sim \pi_\theta(\cdot \mid \mathcal{T}, h_t, \tilde{g}_t)

The reconstruction input concatenates the task, the recent history, and every retrieved (experience, source-state) pair:

x_\text{recon} = \mathcal{T} \oplus h_t \oplus \bigcup_{i=1}^{k} (e_{t,i},\, o_{t,i}^{src})

and $f$ is a simple conditional: if the policy decides nothing retrieved applies, it emits the literal token <EMPTY>, and $\tilde{g}_t$ falls back to a fixed self-reasoning prompt $p_\text{self}$ instead of forcing a bad match into the action context.

Three-panel pipeline diagram. Stage 1, Memory Retrieval: task, history, and current state feed a policy model that issues a query, which retrieves the top-k memories (each a strategy paired with its source observation) from the memory store. Stage 2, Contextual Memory Reconstruction: the policy model compares each memory's source observation against the current observation, then a conditional mapping either passes through the guidance or, if it is EMPTY, substitutes a self-reasoning prompt, producing final guidance. Stage 3, Action Generation: the policy model conditions on task, history, and the final guidance to produce an action, which is executed in the environment and scored by an outcome-plus-format reward, which updates the policy model via GRPO with a group-relative advantage. — The three-stage inference pipeline — retrieval, contextual reconstruction, action — trained end-to-end with GRPO on outcome-plus-format reward (Wu et al., 2026, Figure 2).

The mechanics of that middle stage — comparing a retrieved memory's source state against the live one, and deciding pass-through / adapt / reject — are easiest to see with a toy example. Toggle through the three cases below:

reconstruct vs. replay · one decision stepillustrative

task: put a clean apple in the fridge · retrieved memory (source state: fridge was empty): “open the fridge, place the apple on the middle shelf”

current: fridge is empty (same as the memory's source state)

stage 1

retrieve

stage 2

critique + compare o_t vs oᵢˢʳᶜ

stage 3

act on g̃ₜ

REPLAY (verbatim)unchanged

“open the fridge, place the apple on the middle shelf”

RECONSTRUCT (MemHarness)pass-through

“open the fridge, place the apple on the middle shelf”

Replay injects whatever the memory bank returns and never looks at whether it still fits. When the state matches the memory’s source, that costs nothing — both paths agree. Once the state has drifted, replay keeps repeating the stale instruction while reconstruction rewrites the target to the shelf that is actually free. And when nothing retrieved applies, reconstruction can say so explicitly — <EMPTY>— and drop back to the agent’s own reasoning, instead of forcing a bad match into the context.

Note what each branch does differently from plain replay. When the state genuinely matches the memory's source, reconstruction is a no-op and replay would have been fine anyway — the interesting cases are the other two. When the state has drifted, reconstruction rewrites the target, not just the wording, while replay keeps repeating an instruction the environment has already invalidated. And when retrieval turns up nothing usable, MemHarness can say so explicitly and fall back to the agent's own reasoning — a replay pipeline has no equivalent move; it either injects a weak match or injects nothing silently.

Training: one policy, three roles, GRPO end to end

The same weights play all three parts — retriever-decider, reconstructor, actor — and the whole thing is trained with GRPO on a sparse outcome reward plus a small format bonus:

R(\tau_i) = R_\text{outcome} + 0.1 \cdot R_\text{format}

$R_\text{outcome}$ is 10 for a successful episode and 0 otherwise; $R_\text{format}$ checks that every step emits exactly one valid <think> block, one valid <action> block, that memory is retrieved through valid <retrieve_memory> blocks (one to five times per episode), and that everything is in English. Rewards are group-normalized — sample $G=8$ rollouts per prompt and standardize against the group:

A_i = \frac{R(\tau_i) - \text{mean}(\{R(\tau_k)\}_{k=1}^{G})}{\text{std}(\{R(\tau_k)\}_{k=1}^{G})}

and the policy update is the standard clipped-surrogate-plus-KL objective, with clip range $\varepsilon = 0.2$ and KL coefficient $\beta = 0.01$ :

\mathcal{J}(\theta) = \mathbb{E}\!\left[\frac{1}{\sum_i |\tau_i|}\sum_{i=1}^{G}\sum_{j=1}^{|\tau_i|} \Big(\mathcal{L}^{\text{CLIP}}_{i,j}(\theta) - \beta\, \mathbb{D}_{\text{KL}}[\pi_\theta \| \pi_\text{ref}]\Big)\right]

None of this is a new RL recipe — it's the same token-level, group-relative machinery covered in Token-level RL is a first-order approximation to the reward you actually want. What's specific to MemHarness is that the reconstruction step itself is inside the RL loop and gets credit for the same sparse outcome reward as the final action, rather than being a fixed prompt template bolted on the side. That's also why an ablation later in this piece — replacing the trained reconstruction with a generic, untrained LLM doing the same rewriting job — measurably underperforms: rewriting text is not the same skill as rewriting text so that it wins the episode.

Before RL, there's a short cold-start SFT stage — 200 trajectories with GPT-5.1-generated retrieval and reconstruction turns, plus 200 trajectory-to-memory summarization examples per benchmark — whose only job is to teach the interaction protocol (when to emit <retrieve_memory>, how to format guidance). The paper is explicit that this stage is about "protocol and format alignment rather than task-skill acquisition," and the numbers back that up: the cold-start model alone scores a worse 7.6% on ALFWorld than the untrained base model's 14.5%, because it has learned to follow a longer protocol without yet having learned to solve the task.

The memory bank itself lives in Milvus, embedded with BGE-M3, retrieved by cosine similarity at $k=3$ . It isn't hand-curated — during training, the policy distills roughly half of its own generated trajectories (balanced between successes and failures where possible) into new memory entries, so the bank grows out of the same policy that reads from it. Write-time deduplication skips a new entry if it's too similar (cosine $> 0.85$ ) to something already stored — enabled for WebShop, disabled for ALFWorld — and retrieval-time deduplication thins a larger candidate pool before truncating to the top- $k$ .

Does it beat the baselines

On the headline numbers: MemHarness reaches 85.2% average success on ALFWorld's six task categories and 75.6% on WebShop, ahead of every baseline the paper reports — including foundation models an order of magnitude larger:

ALFWorld · Avg. success rate across 6 task categories (Table 1)

Qwen2.5-7B (base)

14.5%

ReAct

27.9%

Mem0

33.5%

GPT-4o

49.2%

Gemini-2.5-Pro

62.1%

EvolveR (reproduced)

70.1%

GRPO (no memory)

76.4%

MemHarness

85.2%

050100

WebShop · success rate (Table 1)

Mem0

Qwen2.5-7B (base)

7.8%

ReAct

19.5%

GPT-4o

23.7%

Gemini-2.5-Pro

35.9%

GRPO (no memory)

66.1%

EvolveR (reproduced)

72.6%

MemHarness

75.6%

020406080

A few things worth being precise about here. The 16-row full table (not all shown above) mixes closed-source frontier models (GPT-4o, Gemini-2.5-Pro), prompt-only memory agents (ReAct, Reflexion, Mem0, ExpeL, MemP, SimpleMem), and RL-trained agents (RLOO, GRPO, MemRL, EvolveR, and two "+GRPO" memory hybrids) — and with one exception (EvolveR, explicitly marked "reproduced"), the paper doesn't say whether the other baseline numbers are copied from those methods' original papers or re-run by the authors under this setup. Given every RL-based and prompt-based baseline shares the same Qwen2.5-7B-Instruct backbone as MemHarness, it reads as an in-house re-implementation for a controlled, like-for-like comparison — which is the right thing to do for fairness, but it also means there's no independent number to check any of them against. Mem0 in particular scores worse than the untrained base model on WebShop (2.0% vs. 7.8%) — plausible for a general-purpose memory library not tuned to this task, but a reminder that "memory system" is not automatically an improvement.

Why raw memory can hurt: the ablation

The paper's most useful table isn't the leaderboard, it's the ablation, because it isolates why MemHarness wins rather than just that it wins. Same policy, same GRPO recipe throughout — only the memory wiring changes:

ablation explorer · same policy, different memory wiringTable 2, measured

ALFWorld · Avg. SR76.4% · reference

WebShop · Avg. SR66.1% · reference

RL Only (GRPO). GRPO training with no memory at all, at training or test time. The reference line below.

The reference tick marks RL-only, no memory at all. Raw verbatim replay (RL + Raw Memory) sits below that line on ALFWorld — memory hurt more than it helped until it was reconstructed. Withholding memory at test time after reconstruction-aware training (w/o memory) still beats RL-only on both benchmarks, which is the paper’s case that the reconstruction objective sharpens the policy’s own reasoning, not just its memory use.

Two results here are worth sitting with. First, RL + Raw Memory — verbatim replay, grafted onto the same trained policy — actually loses to having no memory at all on ALFWorld (70.1% vs. 76.4%), which is the paper's sharpest evidence that unreconstructed memory is not a free win; it can be actively confusing. Second, w/o memory — the fully-trained MemHarness policy with retrieval switched off at test time — still beats the no-memory-ever RL baseline on both benchmarks (83.0% vs. 76.4% on ALFWorld, 73.6% vs. 66.1% on WebShop). The paper reads this as evidence that training the policy to reconstruct memories also sharpens its general reasoning, independent of whether memory is available at inference — the reconstruction objective works partly as a training-time signal, not only a run-time lookup.

The training curves back this up with a second, independent kind of evidence — not an end-of-training snapshot, but what happens over the run:

Line chart titled ALFWorld, x-axis Training Steps from 0 to about 150, y-axis Success Rate percent from 0 to 100. Three lines: overall Success Rate, SR when Accepted (a reconstruction was kept), and SR when Rejected (a reconstruction was rejected), all rising together from about 8% to about 85% over training, with SR when Accepted tracking closely just under overall SR and SR when Rejected running lower and noisier throughout. — Success rate over GRPO training, split by whether the episode contained at least one accepted or rejected memory reconstruction — accepted reconstructions track the rising success rate; rejected ones lag behind it (Wu et al., 2026, Figure 4a).

Trajectories where the policy accepted a reconstructed memory track the overall success-rate curve closely; trajectories where it rejected one lag behind and stay noisier throughout training. That's a consistency check on the whole framework: if "accept vs. reject" were an arbitrary or miscalibrated signal, there'd be no reason for it to correlate with which trajectories actually succeed.

MemHarness also holds up when the environment itself is unfamiliar. On ALFWorld's out-of-distribution split — unseen room layouts and object placements — it scores 85.9%, while stripping reconstruction back out (raw memory injected, same OOD environments) drops to 76.3%, and disabling reconstruction only at test time (same trained policy) lands at 82.4%. The direction of every result here matches the in-distribution ablation: verbatim replay is the worst way to use memory precisely when the environment has changed most.

The mechanism, under a microscope

Everything so far shows that reconstruction helps. The paper also runs two controlled probes asking a narrower question: does the policy's reconstruction step actually compare the current state against the memory's recorded source state, or is it just producing plausible-sounding rewrites without really checking anything?

mechanism probe · does reconstruction compare states?Table 4, measured

correct sourceRR 8.7% · SR 85.2%

no sourceRR 7.8% · SR 80.0%

random sourceRR 13.3% · SR 84.3%

◧ rejection rate (RR)◨ success rate (SR)

On ALFWorld, removing the source state entirely leaves rejection rate almost flat but success rate drops — the agent accepts guidance it should have questioned. Swapping in a random source state spikes rejection instead, because the model is actively comparing the current observation against that source, not just reading the retrieved text.

The source-state ablation answers this directly: strip $o_i^{src}$ out of the reconstruction prompt entirely, and rejection rate barely moves — but success rate drops, because the policy now accepts guidance it has no way to judge as stale. Swap in a random memory's source state instead — a state that's guaranteed not to match — and rejection rate jumps sharply (8.7%→13.3% on ALFWorld, 56.0%→63.3% on WebShop). That asymmetry is the tell: removing the comparison signal doesn't change behavior much because the policy simply can't tell anymore, while corrupting it with a wrong-but-present signal actively triggers more rejections. The counterfactual probe — asking a strong LLM to make a minimal edit to 1,000 real states so a previously-applicable memory should no longer apply, then scoring only the reconstruction output — shows the same pattern from the other direction: minimal edits shift outputs measurably away from "unchanged" and toward "adapted" or "rejected" on both benchmarks, with WebShop rejecting far more often than ALFWorld in both the matched and edited conditions (72–79% vs. 0–6%), which the paper attributes to WebShop's longer, more heterogeneous page observations making a fuzzy accept riskier than a clean reject.

Honesty check

Self-reported, single lab, no replication. Every number above is from this one paper. I found no independent reproduction, and the community-discussion page on alphaXiv had nothing beyond the paper's own abstract and tables at the time of writing.
Baselines are the paper's own reruns, not cited published numbers, as far as the text discloses — with the single exception of EvolveR, marked "(reproduced)" in the table. That's a reasonable design for a fair, same-backbone comparison, but it also means none of the sixteen rows in Table 1 have an outside number to be checked against.
No hardware, no latency, no cost. The paper never states what GPUs it trained or evaluated on, never reports wall-clock time, tokens/sec, or dollars, and never measures the added inference cost of the reconstruction step itself. Reconstruction is a second full decode pass through the same 7B policy on every step where memory is retrieved (retrieval decision, then reconstruction, then action) — at minimum one extra generation versus a direct-replay or no-memory baseline — and that overhead is not quantified anywhere in the paper.
No variance, no seeds. Every success-rate and rejection-rate number is reported as a single figure with no standard deviation, confidence interval, or multi-seed spread disclosed for the evaluation runs.
Two benchmarks, one model scale. ALFWorld and WebShop are both well-worn, relatively short-horizon (15–50 step) simulated environments; the paper does not test a larger backbone, a real-world tool-using agent, or a benchmark with a genuinely different observation modality. The conclusion names this directly: "future work will explore scaling to larger models and open-ended environments" — which is the authors' own way of saying this hasn't been tried yet.
No explicit Limitations section. The paper has no dedicated limitations discussion; what's above is reconstructed from the ablations, the conclusion, and what the method section does and doesn't measure.
What is solid: the mechanism probes (source-state ablation, counterfactual editing) are a genuine attempt to falsify the "it's just fluent rewriting" explanation, and they point the same direction from two independent angles. That's better methodological care than a bare leaderboard table, even without outside replication.

The take

The idea underneath MemHarness is simple enough to state in a sentence — compare the memory's source state to the current one before you trust it — and the paper's real contribution is making that comparison a trained skill inside the same policy, credited by the same sparse outcome reward as the action itself, rather than a hand-written heuristic bolted onto retrieval. The ablations back the framing better than the leaderboard does: raw memory replay measurably loses to no memory at all on one benchmark, and the reconstruction-trained policy keeps a chunk of its advantage even with memory switched off entirely, which says the training signal is doing more than teaching better lookups.

What it hasn't shown yet is whether any of this survives outside two small, well-studied simulators at 7B scale, and whether the extra reconstruction pass is worth its unmeasured latency cost in a setting where that matters. "Reconstruct, don't replay" is a good design principle for any agent harness that reads back its own memory. Whether this specific recipe for teaching it — GRPO, a <EMPTY> escape hatch, a Milvus bank refreshed by the policy's own trajectories — is the way to get there past ALFWorld and WebShop is still an open question the paper itself doesn't claim to answer.

Built on MemHarness: Memory Is Reconstructed, Not Replayed (Wu et al., 2026; arXiv:2607.28272). Figures are reproduced from the paper for commentary. The interactive diagrams use the paper's measured table values except where marked illustrative; see the Honesty check above for what is and isn't independently verified.