Terminal-RSI: the second cycle, and how much smaller it is

Ten days ago I wrote up Recursive Self-Rewrite, a paper in which one Qwen3.8-27B solves terminal tasks under several harnesses and then re-solves each success under a plain loop to make its own fine-tuning data. I closed on the question that paper left open: does the improved model rewrite again, and does the gain flatten after one pass?

Terminal-RSI (Yang, Li, Shi et al., Tencent Hy Frontier and NUS, arXiv 2610.22192, 15 Oct 2026) is the same group's answer. The author's post puts it as "We used an agent to help train a stronger successor. Then we used that successor to help train the next." Across two cycles, they report, a 27B model's Terminal-Bench 2.1 success rises from 51.7% to 86.5% and Terminal-Bench 4.0 from 0.0% to 19.8%, "a new SOTA". The post ends on the right question: how far this kind of hill climbing goes, and what makes it stall.

The paper answers more of that than the post lets on. Every number below is labelled. Reported is the paper's figure, not re-run. Measured is something I read or counted myself from a file or page. Reasoned is my arithmetic on the other two. There was nothing to run: the project page lists code and model as "soon", the linked GitHub repository did not clone (it is private or not yet created), and the linked Hugging Face organization has no models (all measured on 2026-10-16).

What recurses, exactly

"Recursive self-improvement" is used for at least three different things on this site. In RRSI, Recursive Harness Self-Improvement and LobeHub's week a model rewrites its own harness; in Dream-RSI, a scheduler. Terminal-RSI does none of that. The harness is fixed. What changes from cycle to cycle is the weights, and the thing the model rewrites is its own training data.

expand

There are five retained checkpoints, in order: Base (the Qwen3.8-27B base model), RSR-1, RPR-1, RSR-2, RPR-2. The last one is called Terminal-RSI-27B. RSR is Recursive Self-Rewrite, supervised and actor-only. RPR is Recursive Policy-Refine, which is PPO with a learned critic. Two cycles means the rewrite-then-refine pair runs twice, and the second rewrite is done by the model the first refine produced.

expand

Self-rewrite: the model rewrites its successes, not itself

The rewrite stage is the RSR method from the earlier paper, run by whichever actor is current. The current actor attempts tasks under a set of complementary harnesses (the RSR paper used three: plain Terminus 2, a state-machine harness, and one that runs the verifier and tells the model to keep going). A trajectory is kept only when the task's protected verifier passes it, written $\mathcal{V}_x(\tau)=1$ in the paper. Then the same actor:

1. reads the kept trajectory, with harness control messages stripped, and writes a runbook: milestones, checks, pitfalls, not the answer;
2. has the runbook screened by a reviewer that sees only the public task and the runbook, a binary gate for schema, artifacts and leakage;
3. solves the task again in a fresh sandbox under the target harness, with the runbook as private guidance;
4. keeps the re-solve only if it passes the verifier and a leakage check, then deletes the runbook and the reviewer dialogue from it.

What survives is a plain trajectory from the public instruction alone. Those become supervised data, a masked negative log-likelihood over the model's own generated tokens:

where $Q_k^{\rm clean}$ is the accepted set from cycle $k$, $y_j$ a generated token, $h_j$ its public prefix and $m_j$ masks out environment output. The paper's reference setting is $K=M=4$ (four runbooks per success, four executions per runbook), "not necessarily every run here" (reported). The critic is not touched during this stage.

That answers the question I left open last time. In cycle two the discovering, planning and re-solving are done by the retained RPR-1 actor, not by the base model. The successor is trained on data its predecessor produced and verified. What the paper does not give is how big that second dataset was: there is no trajectory count for RSR-2 anywhere in it. The RSR paper's single round turned 2,001 source successes into 11,094 rewrites (reported there); for cycle two I have no number.

The first stage has public weights. RSR-1's row in this paper (51.70 → 70.10 on TB 2.1, 31.00 → 54.00 on TB Hard, 6.10 on TB 4.0, 6.80 on TB 3.0) is digit for digit the per-run mean the RSR paper reports for its released model, so RSR-1 is, by every number I can compare, IntelligenceLab/RSR-27B (reasoned). Nothing after it is released.

Policy-refine: PPO, with a critic that survives the rewrite

The refine stage is not rejection sampling and not a critic-free group method. It is PPO with a value network, built on the group's earlier T1 terminal-RL stack (slime, Megatron, SGLang, Harbor sandboxes). The settings that matter, all reported:

Both phases use batches of 256 tasks with one rollout each, an asymmetric clip of 0.30 below and 5.00 above (ratio thresholds 0.70 and 6.00), and no KL or entropy term at all, both coefficients 0.00 (reported). Context is 262,144 tokens.

The critic handoff is the part that is new relative to the rewrite paper. The first critic is prepared by training it for one epoch of SWR tasks (the authors' scientific-workflow tasks) with the RSR-1 actor frozen. After RPR-1, that critic is kept. RSR-2 changes the actor by fine-tuning and leaves the critic alone. RPR-2 then starts from the new actor and the old critic, freezes the actor for four critic-only updates so the value estimates can catch up with a policy that has moved, and only then updates both. The paper's explained-variance trace is the evidence it works at all: the warm-up critic goes from −1.15 to 0.01, RPR-1 ends at 0.27 after a peak of 0.33, and every one of 16 RPR-2 observations stays between 0.01 and 0.52 (reported). Whether an inherited critic beats a fresh one is not tested; the limitations section lists "inherited-versus-reset critic controls" as still needed.

Where the reward comes from

Verification is the task's own protected test suite, read as a Common Test Report Format (CTRF) report. Strict success $S \in \{0,1\}$ means the whole verifier passed, not that the agent declared victory or passed a subset. On top of that, two shaped rewards use $P$, the count of passed semantic checks:

with $D_f$ a per-family divisor (the 90th percentile of check counts in that task family, default 82) and both falling back to $S$ when the report is invalid. FDR pays linearly for passed checks and has no completion bonus. SPR always pays more for a success than for any failure, because the progress term is bounded below one.

The motivation is measured on the authors' own logs. In 4,352 audited trajectories, failures took 50.09% of all FDR reward; 158 failures scored at least 1.0; 48 passed every executed check and still failed the terminal gate (reported). Rescoring the same fixed trajectories with SPR drops the failures' share to 28.72% and the count of failures scoring 1.0 or more to zero (reported). That is an offline rescoring, not a training comparison, and the paper says so.

The third named component, TB-GAE, changes the GAE clock so discount and trace decay apply only at turn boundaries, not per token, so a verbose log output does not dilute credit. Crossing $B$ turn boundaries attenuates by $(\gamma_{\rm turn}\lambda_{\rm turn})^B$ regardless of how many tokens sit between. It is a sensible fix, but by the paper's own scoping note it is not shown to be in the reported model: "The documented first phase uses standard GAE with $\gamma=\lambda=1$," and the project page says outright that "TB-GAE deployment in the reported checkpoints remains unverified." With $\gamma=\lambda=1$ inside and across turns, TB-GAE and plain GAE coincide, so for RPR-1 at least, the named estimator is not doing anything (reasoned).

Two cycles, stage by stage

The widget runs the loop. The top half is the rewrite stage, the bottom half the refine stage, and the numbers are Table 2's, one shared checkpoint per stage, Mean@3 success.

All reported (paper, Table 2). The four-suite mean goes 20.68, 34.25, 44.50, 48.75, 50.58 (reported), and I get the same endpoints from the table: (0 + 51.70 + 0 + 31.00) / 4 = 20.675 and (19.84 + 86.52 + 16.95 + 79.00) / 4 = 50.5775 (measured arithmetic).

expand

Three things are visible in that table once you split it by stage type.

Rewriting does most of the lifting on the easier suites; PPO on the hardest. On TB 2.1 the two rewrite stages add 18.40 + 8.99 = 27.39 points and the two PPO stages 7.43 + 0.00 = 7.43 (reasoned). On TB Hard it is 29.50 against 18.50. On TB 4.0 it flips: rewriting adds 6.89, PPO adds 12.95 (reasoned). Supervised re-solves move the model onto tasks it could already almost do; RL with a critic is what moves the hardest suite.

RL cannot start from zero. The paper ran direct PPO from the base model as a control. Over 20 steps it never solves a single TB 4.0 task (0.00 at steps 5, 10, 15 and 20), and TB 2.1 goes from 47.75 to 48.31 (reported, Table 4). RPR-1, started from the rewritten actor, beats it at step 20 by 29.22 points on TB 2.1 and 16.67 on TB 4.0 (reported). That is aligned on update steps, not compute (the paper says so), but the shape is the point: a sparse verifier reward gives PPO nothing to push on until something succeeds, and the rewrite stage is how the loop manufactures first successes.

The second cycle is much smaller than the first. Cycle one (Base to RPR-1) raises the four-suite mean by 23.82 points; cycle two (RPR-1 to RPR-2) by 6.08 (reasoned from the reported means). Per suite, cycle two is between about a fifth and a third of cycle one: TB 2.1 +25.83 then +8.99, TB 4.0 +16.67 then +3.17, TB 3.0 +14.29 then +2.66, TB Hard +38.50 then +9.50 (reasoned). Cycle two also changed the reward and the learning rate, so this is not a clean measure of diminishing returns, but it is the paper's best evidence of where the climb is heading.

Checking the headline

Which 51.7?

The 51.70 baseline is the untrained Qwen3.8-27B under the authors' evaluation loop: Harbor sandboxes, structured JSON actions, explicit thinking disabled, temperature 0.6, 8,192 output tokens per turn, summarization at 32,000 accumulated tokens, mean of three runs (reported). It is the same 51.7 the RSR paper reports for the base model under plain Terminus 2, so it is a Terminus 2 number (reasoned from the match; this paper never names the harness). It is the right baseline for the claim: the same loop, before and after training.

One oddity. TB 2.1 has 89 tasks, and 51.70% is not a whole number of successes over 89, 178 or 267 attempts; the nearest are 46/89 = 51.69% and 138/267 = 51.69% (measured). A task or two must have dropped out of the denominator; read the second decimal loosely.

86.52 is reached by rewriting, not by the last RL stage

TB 2.1 is 86.52 at RSR-2 and 86.52 at RPR-2, which is 77 of 89 tasks (measured: 77/89 = 86.517%). The final PPO stage adds exactly zero on the benchmark in the headline. "Improve through practice" did not do the last stretch on this suite.

0.0 to 19.84 mixes two task sets

The appendix gives the nominal task counts: 89 for TB 2.1, 70 for TB 3.0, 63 for TB 4.0, 100 for Hard, and adds that "earlier records instead use 74 Terminal-Bench 3.0 tasks and 66 Terminal-Bench 4.0 tasks" (reported). I checked every reported score for which task count and run count can produce it as a whole number (measured, script over n ∈ 63, 66, 70, 74, 89, 100 and 1 to 3 runs):

• RSR-1's 6.10 and 6.80 are the RSR paper's one-decimal means, padded to two decimals. On that paper's 66- and 74-task sets they are 4/66 = 6.06% and 5/74 = 6.76%, rounded (reasoned). They fit no whole count over 63 or 70.
• RPR-1's TB 3.0 score of 14.29 is 10/70 or 30/210; RSR-2's 17.46 on TB 4.0 is 11/63 or 33/189.
• The final 19.84 on TB 4.0 fits only 25/126: two runs of 63 tasks. Not three runs, and not 66 tasks. The final 16.95 on TB 3.0 fits no count over 70 or 74 at all.

So the TB 4.0 line in the headline starts on one task set and ends on another, and the end point is probably a two-run mean, not Mean@3. The paper half-concedes this: "Historical summaries contain heterogeneous repeat counts", and the task manifests "must therefore be reconciled" (reported). On 63 tasks, one task flipping in one run moves a three-run mean by 0.53 points and a two-run mean by 0.79 (reasoned). Keep that unit in mind for the next section.

Seeds, selection and what was held out

One training run. "Evaluation repeats measure inference variability rather than independent training replication," and "three evaluations do not replace independent training seeds" (reported). Checkpoints were retained by looking at the benchmarks: Figure 6 plots every suite at every evaluated step, including discarded branches, and the retained RPR-2 checkpoint is step 18 of a run that went on to step 20. The paper's own limitation reads "Checkpoint selection requires protected validation, not adaptation to final benchmarks," and the appendix that "the reported sequence records retained checkpoints rather than validating a prospective selection rule" (reported). That is an honest description of test-set selection.

expand

On contamination, the paper makes exactly one overlap claim: the 100 SWR100 test tasks are not in the 3,000-task training pool (2,400 semantic-verifier tasks plus 600 "supplementary terminal tasks"; reported). It says nothing about overlap between that pool and any Terminal-Bench suite, and adds that "broader repository and pretraining overlap remain incompletely audited" (reported). Two details sharpen this. TB Hard is not a public benchmark; the RSR paper calls it "our self-curated Terminal-Bench Hard", and the earlier write-up traced it to the same synthesis pipeline that supplied training tasks. And the group's T1 paper made a disjointness claim for its corpus against TB 2.1, but this paper does not say whether its 600 supplementary tasks are that corpus. I would weight TB 2.1, 3.0 and 4.0 well above TB Hard, and treat none of them as audited.

"A new SOTA"

Figure 4 is the paper's own comparison, and it does not show what the post says.

expand

On TB 4.0, five models are above Terminal-RSI-27B; on TB 2.1, four are (measured from the figure). The paper's text names only one per panel ("GPT-6 Sol is higher on Terminal-Bench 4.0, and Gemini 3.8 Flash is higher on Terminal-Bench 2.1") and flags that "external scores are reference values; their primary-source and protocol reconciliation remains incomplete" (reported). The reading under which "SOTA" survives is best open-weight model in Figure 4: it beats every DeepSeek, Kimi and GLM row in both panels (measured from the figure). And on TB 4.0 the margin over the next model down, Gemini 3.8 Flash, is 0.14 points (reasoned), about a quarter of one task in one run.

The official Terminal-Bench 4.0 leaderboard is a harder test. I read the data embedded in the leaderboard page at tbench.ai on 2026-10-16. Every entry there is 330 trials, which is 66 tasks at five trials each (measured; five trials reasoned from the pass@5 field). Sixteen distinct models have an entry above 19.84, from Opus 5.5 at 64.85 down to Grok 4.6 at 20.30, and two of them are open-weight: GLM-5.3 at 41.82 and GLM-5.3-Flash, MIT-licensed, at 35.76 (measured). Gemini 3.8 Flash is listed at 19.09, not the 19.70 Figure 4 uses (measured). Terminal-RSI-27B is not on the board, and its number comes from a different loop on, by the decimals, a different task set, so none of this is a like-for-like ranking. It is enough to say the claim does not hold as stated.

None of this makes the result small. A 27B model at 86.52 on TB 2.1, 3.38 points under the top of the paper's own panel, is strong. It is just not state of the art, and the decimals separating it from its neighbours are smaller than one task.

What makes it stall

The post asks this as an open question. The paper already shows several answers.

• Saturation by suite. SWR100 goes 3, 5, 6, 6, 6 across the five stages, and TB 2.1 stops at the rewrite step (reported). Each suite has a point past which the next stage adds nothing.
• Continuing PPO regresses. Run RPR-1 past its retained step 20 and TB 4.0 falls from 16.67 to 11.11 by step 35 while TB 2.1 rises to 78.95. The ordinary FDR branch of RPR-2 falls from 17.46 to 11.11 between steps 6 and 10. The retained SPR branch, run on to step 20, keeps TB 2.1 and loses 2.66, 2.38 and 3.00 points on TB 3.0, 4.0 and Hard (all reported). The useful window for RL is a handful of steps, and the paper finds it by watching the test set.
• The hardest tasks barely move. On the authors' fixed structural-difficulty strata, Extreme tasks go 10.98 at RPR-1, 6.10 at RSR-2, back to 10.98 at RPR-2, while Medium climbs from 23.58 to 58.96 (reported). Rewriting cannot manufacture successes on tasks no harness solves, and PPO cannot learn from successes that never happen.
• The verifier is the ceiling. The 48 failures that pass every executed check are one sign; the case study is a better one. On a Rust–C task, the final model builds a static Rust binary, runs a C-linkage smoke test, and then delegates the actual expression evaluation to Python, which passes the hidden fuzz tests (reported). "Fuzz-test success establishes tested output equivalence, not a native Rust–C AST execution path." A loop trained on verified successes learns whatever the verifier accepts; see MiMo's RL environments for the same lesson.

What I take from it

The mechanism is simple to state and worth stealing. Bootstrap with verified re-solves, then refine with RL, and let the refined model produce the next round of re-solves. The rewrite stage is what lets RL start at all on a hard suite (direct PPO stays at zero on TB 4.0), and RL is what moves the hardest suite once there are successes to reinforce. Carrying the critic through the supervised step is a cheap, plausible way to avoid re-warming a value network every cycle, though nothing here shows it beats a reset.

The claims need trimming. The second cycle is a quarter of the first, the TB 2.1 headline comes from the rewrite step, the TB 4.0 headline spans two task sets, the checkpoint was picked on the reported benchmarks from one training seed, and "SOTA" holds only among the open-weight rows of the paper's own figure. The appendices state most of those limits more carefully than the post does.

For the earlier half of the story, read Recursive Self-Rewrite. For the training side of terminal agents, Scaling agentic RL on where verifiable terminal tasks come from, Multi-harness RL on how much the loop alone moves a score, and Qwen3.8-27B variants for the base model's family.