Dream-RSI: the loop rewrites a scheduler, not the model

2026-09-18 · 13 min · agents · llm · self-improvement · evolutionary-search · benchmarks · explainer

The framing going around: "Google / DeepMind just published Dream-RSI, where an agent can learn from its past attempts, improve how it searches for better solutions, then repeat the loop" — offered as a step toward recursive self-improvement. Every clause in that sentence is checkable. The repo is github.com/zhengkid/Dream-RSI, a personal GitHub handle, not google-deepmind or google-research, and at the time of cloning it contains exactly four things: a README.md, a CITATION.cff, an assets/ folder of logos and figure PNGs, and papers/Dream-RSI.pdf. No implementation. The repo's own release-plan table marks "Full codebase" and "Reproduction scripts" as still being prepared. So "read the code" is not optional context here — there is no code to read. Everything below comes from the 36-page PDF and its appendix, which is the only artifact that exists.

Who actually wrote this

The paper's front page lists 17 authors with numbered affiliations: Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, and Yunsong Guo — Google, University of Maryland (College Park), Google DeepMind, and University of Virginia. Counting affiliations directly from the byline: 10 authors are Google-only (including first author Tong Zheng, who carries a dual Google/UMD affiliation), 4 are Google DeepMind, 2 are UMD-only, and 1 is University of Virginia. That's 14 of 17 authors at Google or Google DeepMind. Both corresponding authors — Xidong Wu and Zheng Zhang — list @google.com addresses, and the paper's own footer reads "© 2026 Google. All rights reserved." So the institutional-authorship claim in "Google / DeepMind just published" checks out; it isn't a stretched or borrowed affiliation.

What doesn't check out is the implied release maturity. The README's arXiv badge still says "coming soon" and CITATION.cff's preferred-citation block has a literal placeholder — arXiv:XXXX.XXXXX with a TODO: fill in the real arXiv id once the preprint is posted comment — even though the README's own BibTeX entry a few lines up already hardcodes arXiv:2609.14858. I checked that ID directly: it resolves, live, titled "Dream-RSI: Recursive Self-Improvement through Evolving Worlds." The paper is posted; the repo's own citation metadata just hasn't caught up with itself yet. None of this is misconduct — a lab author staging a paper and figures on a personal handle before the org repo and arXiv listing land in sync is completely normal. But "just published" is doing more work than the evidence supports: as of this writing there is a real, live, Google/DeepMind-majority paper, and zero lines of the system it describes are public.

What the loop actually rewrites

"Learns from past attempts, improves how it searches, repeats" describes a large family of systems — AlphaEvolve, OpenEvolve, ShinkaEvolve, and a dozen others the paper itself cites in related work all fit that sentence. The question that actually distinguishes them is what sits inside the loop: a policy's weights, a prompt, or a piece of orchestration code. Dream-RSI's discovery tree formalism (Sec. 3) answers this precisely. A tree is rooted at the workspace's initial state; each node is one generation–evaluation attempt with a saved score; an exploration policy $\pi$ observes the tree so far and selects a batch of leaves to extend, up to $W$ parallel workers. That's the whole action space — which nodes to extend, how many in parallel, when to stop. Two agents sit outside this: a discovery agent (Gemini-3.1-Pro or Gemini-3.7-Flash) that actually writes each candidate solution, and an evaluator that scores it. The paper states the boundary directly: "Only the exploration-policy code changes; the underlying models, evaluator, and execution interfaces remain fixed."

Concretely, the exploration policy is a Python class the paper's appendix (Listing 2) specifies down to the method signature:

# paper, Appendix B.2 (Listing 2) — minimal structure every revision must implement.
# Hard constraint: NAME = "OptimalPolicy"; class OptimalPolicy(LLMDesignedMethod)
# in {method_file} only. Everything else is read-only.
from see.policy.api import (
    LLMDesignedMethod, SimResult, _budget_done, _record_curve, finalize_result,
)
 
def solve(self, question, budget=None):
    question.reset()
    res, closed = SimResult(), set()
    while not _budget_done(question, budget):
        prefix = question.observed()
        update_closed(closed, prefix, question)
        batch = select_batch(prefix, question, closed)
        if not batch:
            break
        question.probe_batch(
            batch,
            on_reveal=lambda _: _record_curve(res, question),
        )
    return finalize_result(question, res)

That's the entire recursive artifact: a scheduler over question.probe_batch(...) calls. Each outer round, a fixed LLM — the paper calls it the policy-development agent, and it is a separate, also-frozen model from the discovery agent — reads replay trajectories and rewrites this file. The "dreaming" that gives the paper its name is what makes rewriting cheap: a completed discovery tree already has every node's outcome stored, so an alternative policy can be replayed against it — different branch subset, order, batching, stopping point — by reading cached results, with zero new discovery-agent or evaluator calls. The replay objective the policy-development agent optimizes against (Eq. 1) is

V_i^m = \underbrace{\max_{v \in \mathcal{T}_i^{m,k_i^{m,\star}}} s_v}_{\text{discovery quality}} - \underbrace{\beta_1 N_i^m}_{\text{execution cost}} + \underbrace{\beta_2 \frac{N_i^m}{\max\{1, k_i^{m,\star}\}}}_{\text{parallelism bonus}}

— best score reached, minus attempts spent, plus a bonus for batching useful probes instead of running them serially. $M \geq 1$ candidate policy versions get scored this way per round against every tree collected so far; the version with the best average replay score redeploys online.

Three-stage loop diagram: an exploration policy plus a coding agent explore online, storing traces into a growing discovery-tree history; the discovery tree is converted into a reusable replay simulator; candidate policies are evaluated by dreaming — interacting with the simulator pool — to derive an updated policy, which redeploys for the next round of online exploration. — Dream-RSI's own three-stage loop: online explore, construct replay simulator, dreaming-based policy improvement (Zheng et al. 2026, Figure 1).

what updates, what's frozen · one outer round

Online explore. The current policy version guides the (frozen) discovery agent through a batch of real attempts; the (frozen) evaluator scores each one; results append to the history tree. Nothing about the discovery agent or evaluator changes here — they run the same way every round, in round 1 and round 10.

Nothing about the discovery agent's weights, its prompt (Appendix B.1 — static across the entire run), or the evaluator changes, ever. Nothing about the policy-development agent's weights changes either. The one thing that is versioned round over round is the scheduler above, plus the history it schedules against. That is a real, useful mechanism — a genuinely reusable idea (accumulated search history as a free off-policy simulator for evaluating a meta-policy) — and it is squarely orchestration-level, not model-level.

Is "recursive self-improvement" the right name

"Recursive self-improvement" carries a specific weight in AI-safety discourse: a system that improves its own capacity to improve, compounding — not just better outputs, but a better improver, feeding back on itself without an obvious ceiling. Dream-RSI's title claims that name directly. What it demonstrates is an evolutionary-search system (the same family as AlphaEvolve and ShinkaEvolve, which the paper cites as its own nearest relatives) applied one level up: instead of evolving the target program, it evolves the scheduler that decides how a fixed base model's search budget gets spent, using accumulated history as a cheap simulator instead of expensive online rollouts. That is iterative refinement of a fixed pipeline's control code — real, and worth having — not a model getting better at making better models. The base discovery agent at round 10 has exactly the capabilities it had at round 1. What compounds is a growing cache of past attempts and an increasingly well-tuned Python scheduler reading it. Calling that "recursive self-improvement" isn't wrong about the mechanics so much as it borrows a much larger claim than the mechanics support.

Does it saturate

The paper's headline experiments don't run long: Lasso path discovery goes 5 rounds, the three math tasks go 10. Neither publishes a matching round-by-round curve. Exactly one ablation does — Figure 6, GPU-kernel engineering on the ConvDiv task — and it's worth reading closely because it's the only place the paper shows what happens as the loop keeps going rather than just where it lands.

Four scatter-and-step charts (VGG16, LayerNorm, ConvDiv, ConvMax) plotting performance in inverse milliseconds against number of generations, each comparing a solid Dream-RSI curve against a dashed Recursive Fixed Exploration curve. Dream-RSI reaches comparable performance using 2.43x and 1.79x fewer generations on VGG16 and LayerNorm, and reaches 2.09x and 1.44x higher performance under comparable budgets on ConvDiv and ConvMax. — GPU kernel engineering, Dream-RSI vs. its matched baseline, same discovery-compute axis on all four panels (Zheng et al. 2026, Figure 4).

does the loop win, and does the win come from compute?

ConvDiv (GPU kernel engineering) · round-best performance, Fig. 6(a)

evaluated attempts / round · Fig. 6(b)

drag through the 9 roundsE8 · perf 1.898 · 86 attempts

This is the only task the paper carries past 5–10 rounds, and performance keeps rising through all 9 — no plateau, no collapse. But look at the bar underneath: compute per round is not monotonic either. It drops 110 → 50 as the policy gets more selective, then climbs back to ~90 once progress stalls at E4–E5. The gain tracks a smarter schedule, not a bigger one.

Round-best performance on ConvDiv climbs for all 9 rounds shown (E0 through E8): 0.427, 0.625, 0.855, 1.403, 1.488, 1.499, 1.770, 1.880, 1.898 (1/ms, higher is better) — no plateau, no collapse. But the effort spent getting there isn't monotonic: evaluated attempts per round go 110, 110, 87, 80, 50, 92, 80, 91, 86 (Fig. 6b). The policy gets more selective as it improves (110 → 50 attempts through E0–E4), then spends more again once progress stalls (back up to ~90 by E5). That's a genuinely adaptive schedule, not a fixed one — and it's also the honest answer to "does it saturate": nine rounds isn't long enough to tell. It's the longest run in the paper, and it's still climbing.

One more thing worth flagging from the figure the paper doesn't discuss in text: on the Lasso task (Fig. 3b, not reproduced here), the two methods share an identical round 1 by construction, and Dream-RSI's own curve is not monotonic either — on the Gemini-3.1-Pro backbone it sits roughly flat, even slightly worse, through rounds 2–4 before a large jump at round 5. The offline replay objective guarantees the selected policy version is no worse than the current one in replay score on history collected so far — it does not guarantee the next online rollout improves on the last one, because replay score and true downstream performance aren't the same measurement. The paper's real per-round curves show exactly that gap; its prose doesn't mention it.

Baselines: is the win just bought with extra compute

This is the check this site runs on every self-improvement or looped-computation claim — see Looped transformers, compute-matched and Virtual logic depth, both about the same confound in a different architecture: is a reported gain the mechanism, or is it just more FLOPs that a matched baseline was never given a chance to spend? Dream-RSI's controlled baseline is Recursive Fixed Exploration — literally the same discovery agent, evaluator, initialization, and round-1 call budget, with the one difference being that its exploration policy never gets rewritten. That's a well-chosen ablation: it isolates the policy-rewriting mechanism from everything else in the pipeline.

Switch the widget above to its every reported comparison tab: nine rows, one per task/backbone where the paper reports Dream-RSI against this exact baseline. Eight favor Dream-RSI, several at lower compute — the Lasso task on Gemini-3.1-Pro uses 317 discovery-agent calls against the baseline's 550 (1.74× fewer) while also cutting average downstream runtime 1.22× (2931.0ms vs 3587.1ms); on Gemini-3.7-Flash it's 1879 vs 3200 calls (1.70× fewer) for a 1.07× runtime win. That's the opposite of the usual confound: the win isn't compute bought back as quality, it's less compute and better quality, on the paper's own controlled ablation. The GPU-kernel tasks report the same shape from the other direction — 2.09× and 1.44× higher performance at "similar" (not exactly matched) budgets on ConvDiv/ConvMax, 2.43× and 1.79× fewer generations at comparable performance on VGG16/LayerNorm.

The ninth comparison is where it doesn't hold. On the Autocorrelation Inequalities task, Dream-RSI scores 1.456375 against Recursive Fixed Exploration's 1.456001 — lower is better here, so Dream-RSI is measurably worse than its own matched baseline, by a small margin. Neither comes close to SimpleTES's 1.453675, which needs 51,200 generations to get there against Dream-RSI's fewer than one thousand. The paper states the SimpleTES gap plainly; it doesn't mention that Dream-RSI also loses to its own apples-to-apples baseline on this one task. And all three math tasks (Sum-Diff, Autocorrelation, Circle Packing) share a real gap: Table 1 reports round counts, not a compute column at all, so "equal compute" can't be checked for them the way it can for Lasso and the kernel tasks — only quality.

There's a second, more structural gap in the compute ledger, and it's specific to this paper's design rather than a copy of the FLOPs-accounting confound in the looped-transformer pieces. "Discovery cost is quantified by the total cumulative number of discovery-agent calls" — that's an honest, well-defined metric, and Recursive Fixed Exploration doesn't need it, since its policy never changes. But Dream-RSI's offline phase runs a separate LLM (the policy-development agent) for $M \geq 1$ revisions every round, reading replay trajectories and rewriting the policy file — real generation calls that this paper's own metric excludes by definition. The paper never reports $M$ , never reports how many tokens or calls the policy-development agent spends, and never gives a dollar or GPU-hour figure for either agent — only call and generation counts. So the headline "162× fewer calls than SimpleTES" and the compute-ratio rows above are all true as measured, and none of them account for the one piece of compute that exists in Dream-RSI's pipeline and doesn't exist in its baseline's.

The two things that get discovered, one level apart

It's easy to conflate the exploration policy (the scheduler above) with what the discovery agent actually produces, because the paper's appendix prints a complete example of the latter too: a Lasso regularization-path solver in C++, discovered under the Lasso task described in Sec. 4.1. One line, from its inner loop:

// paper, Appendix C (Listing 3) — the discovered artifact, one level below
// the exploration policy. Branch-free soft-thresholding, used inside the
// coordinate-descent update.
static inline double soft_thresh(double z, double gamma) {
    double abs_z = std::abs(z);
    double val = abs_z - gamma;
    return std::copysign(val > 0.0 ? val : 0.0, z);
}

This is what a discovery-agent attempt looks like — hand-tuned-looking C++ with strong-rule screening, Cauchy–Schwarz KKT pruning, and hardware-aware SIMD blocking, the same category of artifact SimpleTES, AlphaEvolve, and every other system in the paper's related work produces. Dream-RSI doesn't change how these get written or scored. It changes which of the discovery agent's attempts get more of the compute budget, and when to stop asking for more — the scheduler, not the artifact.

Where this sits

Recursive Harness Self-Improvement is the closest relative on this site: it also rewrites a piece of orchestration text (a harness prompt, there) using a fixed underlying model and feedback signal, and it also earns its efficiency claim honestly (Θ(1) cost per iteration instead of a population's Θ(m²)). The difference is the feedback source — RHI compares against exactly one prior version live; Dream-RSI replays against an entire accumulated tree offline, for free. Recursive Language Models is a useful contrast in the other direction: it turns a transcript into a data structure a program can slice, and a subagent call into a function call, without anything resembling a self-editing loop — no policy anywhere gets rewritten, recursively or otherwise, which is a reminder that "recursive" in agent-systems naming doesn't always imply "self-improving," and vice versa here.

The mechanism Dream-RSI ships is real: turning a discovery tree into a free replay simulator for evaluating meta-exploration policies is a genuinely reusable idea, not a rebrand. The evidence for it is mostly well-controlled — the matched-baseline check that this site runs by default actually passes, on eight of nine measured comparisons, at equal or better compute. What it isn't is evidence of the thing its name promises: nothing about the base model changes, the longest demonstrated run is nine rounds of one ablation, one task loses to its own controlled baseline, and the compute spent rewriting the scheduler itself is never on the ledger. Iterative, useful, honestly mostly-checked-out orchestration improvement — not a model that is getting better at getting better.