# RSI-Inference Bench: the 26.7% is ten tasks, and the agents beat the maintainers' fixes

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/rsi-inference-bench
> date: 2026-10-17
> tags: benchmarks, agents, agentic-coding, inference, vllm, sglang, self-improvement

On 16 October Apple ML Research put out **RSI-Inference Bench** with a short question: "Can AI agents
improve the systems that run AI?" The [announcement](https://x.com/onloglogn/status/2111170954944823685)
says the team "spent \$100K API tokens, 50K GPU hours to find out", built environments from real vLLM
and SGLang work across Day-0 model support, inference optimization and bug fixing, and found the
benchmark "unsaturated: best agent at 26.7% pass@1", with "verifiable rewards: binary and continuous".
The name, the lead author explains in a reply, is about the loop: agents served by vLLM improving vLLM.

There is no paper yet. The README's badge says "coming soon". What exists is the
[repository](https://github.com/apple-aiml-research/ml-rsi-inference-bench) and a leaderboard page,
`docs/leaderboard.html`, which is what [rsi-inference.com](http://rsi-inference.com) redirects to. That
page carries its full data as JSON: every agent, every task, every attempt's cost and hours. So I
cloned the repo at `a6b10af` (16 Oct 2026), read the task directories and the graders, ran the
harness's own tests, and recomputed the headline numbers from that JSON.

Everything below is labelled. **Measured** means I read or computed it from a file in the repo or ran
it. **Reported** means it is the team's figure and I could not check it. **Reasoned** means it is my
arithmetic on the other two.

<RepoCard repo="apple-aiml-research/ml-rsi-inference-bench" />

## What the benchmark asks

<Figure
  src="https://media.thesatyajit.com/ai/articles/rsi-inference-bench/fig1.png?v=d979122ba796f72f"
  alt="Timeline of a model's life in an inference engine. Day 0, when a new model is released: Day-0 Model Support, goal serve the new model, scored pass or fail, 10 tasks. In production, when the model goes live: Inference Optimization, goal make it serve faster, scored speed-up, 20 tasks. In maintenance, when a regression lands: Bug Fix, goal find and fix a slowdown, scored speed-up, 7 tasks."
  caption="Three families follow one model through an engine: support it on day 0, make it faster in production, repair it when a regression lands (RSI-Inference Bench README, lifecycle figure)."
/>

The benchmark follows a model through its life inside a serving engine, and each stage is a task
family (measured: 37 directories under `tasks/`):

- **Day-0 Model Support**, 10 tasks. The engine cannot serve the model yet. Land native support. Scored
  pass or fail.
- **Inference Optimization**, 20 tasks. The model serves. Make it serve faster. Scored by speed-up.
- **Bug Fix**, 7 tasks, named `regression_*` in the repo. A real performance regression is in the
  tree. Find it and fix it. Also scored by speed-up.

Twenty tasks are vLLM and seventeen SGLang; seventeen run on H100s and twenty on B200s (reported in the
README, and the GPU split matches the task configs). I loaded all 37 `task.toml` files through the
harness and grouped them: 7 tasks get one GPU, 6 get two, 14 get four and 10 get eight. All 37 set
`allow_internet = false`, and all 37 give the agent `timeout_sec = 30000`, which is the "8.3 hours" on
the page (measured).

<Figure
  src="https://media.thesatyajit.com/ai/articles/rsi-inference-bench/fig2.png?v=cc6c574289f303c8"
  alt="Evaluation pipeline drawn as a rail: an icon for vLLM or SGLang, a clock, a bar marked 0 to 8 hours, a gauge labelled Verifier, and a tick labelled Pass/Fail, Speed-up."
  caption="One attempt: the engine and a model, up to eight hours offline, then a separate verifier that builds and measures the result (RSI-Inference Bench README, pipeline figure)."
/>

The harness is Pier, a fork of [Harbor](https://www.harborframework.com/docs/tasks) vendored in
`third_party/pier`. A task is a directory: `instruction.md` for the agent, `task.toml` for the
machine and the scoring contract, `environment/` with a Dockerfile, `solution/` with an oracle, and
`tests/` with the verifier. This is the [SWE-bench](https://www.swebench.com) shape scaled up to whole
servers, and "run the tests" costs accordingly. A Day-0 verifier is allowed between 3,600
seconds (Stable Diffusion 3 in SGLang) and 158,330 seconds (MiniMax-M3 in vLLM), which is 44 hours on
the same GPUs (measured, `[verifier] timeout_sec`).

<Figure
  src="https://media.thesatyajit.com/ai/articles/rsi-inference-bench/fig3.png?v=af877790a46cf069"
  alt="Grid of chips counting the 37 tasks by layer. Engine: vLLM 20, SGLang 17. Hardware: H100 17, B200 20. Modality: text 31, text plus images 6. Workload: prefill-heavy 11, decode-heavy 20, high load 14, long context 5. Methods: autoregressive 34, continuous diffusion 1, discrete diffusion 2. Parallelism: tensor parallel 30, data parallel 3, expert parallel 2. Model rows follow, including mixture of experts 30, MLA 10, sparse attention 14, FP8 16 and FP4 13."
  caption="Coverage by layer: engines, GPUs, workloads, parallelism, attention types, speculative decoders and precisions, counted over the 37 tasks (RSI-Inference Bench README, coverage figure)."
/>

## How one task is built

The clearest task to take apart is `regression_fp8_decode_throughput`. It is small enough to read in
full and it is built from a real upstream bug.

**The brief.** DeepSeek-R1-0528 in NVFP4 is served on 8x B200 with data parallelism 8 and expert
parallelism. The tree "serves correctly", the tokens are right, but mean time per output token (TPOT)
is too high. The agent is told what metric is scored and on which shapes. It is also told that "a
maintainer's own repair of this deployment exists and has been measured on this rig", without being
told what it changed or what it reached (measured, `instruction.md`).

**The sealed snapshot.** The Dockerfile checks out vLLM at commit `a4047d4`, refuses to build if the
checkout lands anywhere else, then deletes every `.git` directory. It resets every file's
modification time to 2000-01-01. It runs `deidentify_engine_version.py`, which cuts the installed
version from `0.16.0rc2.dev369+ga4047d4ea` down to its public part and then re-scans for a leftover
commit hash, failing the build if it finds one. Finally it deletes the lockfile, because "it carried
the engine pin" (measured). Nothing on disk says which day of vLLM's history this is.

**The oracle.** `solution/solve.sh` applies "upstream's merged fix for vllm-project/vllm#34900,
unmodified": a two-file patch that lets a custom op declare which argument dimensions are really
dynamic, instead of compiling every dimension as symbolic (measured). On this rig it is worth 1.206x,
and 1.168x, 1.239x and 1.213x on the three scored shapes (reported in `task.toml`, measured by the
team).

**The verifier**, and the reward, are in the explorer below.

<TaskAnatomy />

The verifier builds the agent's tree and the pristine tree on the same machine. It decodes fixed
prompts through both and fails the run if the outputs differ. Then it benches TPOT at three scored
shapes: two-token prompts, 1,000 output tokens, with 2,048, 4,096 and 8,196 requests in flight. A
fourth shape, 64 in flight, is a guard. It is not scored, but if the candidate is under 0.95x of the
pristine tree there in every round, the run fails (measured, `regression_scoring.py`).

Three of the twelve leaderboard agents had a measured result replaced by 1.00x somewhere in the regression family. On
this task, Gemini-3.8-Flash measured 0.884x and failed the guard, and the board records it at 1.00x.
On `regression_mtp_decode_throughput`, Opus-5 measured 1.555x but failed the correctness check, and it
too is recorded at 1.00x (measured, the `gate` and `published` fields in the leaderboard JSON).

## How a speed-up is measured

The continuous reward is a ratio, and a ratio is only as good as its denominator. The repo is
unusually frank about this. The docstring of `tests/_bench_rounds.py` records that an earlier version
of the verifier benched pristine and then candidate once each, in a cold container, and "measured
with a candidate tree byte-identical to pristine, that estimator reported a speedup of roughly ten
percent for no change at all". The first bench in a container is slow, so whichever arm goes first
loses. That artefact was bigger than the real fix's effect (measured: quoted from the file).

The fix has two parts, and I find the reasoning in that file the best thing in the repository:

1. **Discard warmup benches** on the pristine tree until the readings plateau, so no scored bench is
   the first in its container. That removes the size of the bias.
2. **Rotate the arm order** across an even number of scored rounds, so whatever first-in-round cost
   remains lands on both arms equally. That removes its sign.

The score per shape is the **median of the per-round paired ratios**, not a ratio of means, and the
task's score is the geometric mean over the scored shapes. At least two rounds per arm are required.
Anything under 1.03x, the declared `noise_floor`, is flagged `no_speedup` (measured,
`verify_regression_real.py`).

The 20 throughput tasks use a sibling contract. Each scores three workload kinds, for example on
`optimize_gpt_oss_120b_2xh100_open` a decode-heavy one (1,024 in, 8,192 out, one request at a time), a
prefill-heavy one (8,192 in, one out) and a closed-loop sweep at 16, 32 and 64 concurrent requests,
all against the `inference_base` tag on the same GPUs. Correctness is a looser gate there: tokens
compared on 20 prompts of 100 tokens, 95% must pass. And a patch that only flips CLI flags or default
values in `vllm/config.py` is classified `category=config` and scores zero whatever it measures, which
closes the cheapest trick (measured, `task.toml`, `instruction.md`, `category_gate.py`).

One thing the throughput family does not have is a ceiling. None of its 20 tasks ships a reference
optimization. Seventeen carry a `known_good.patch` that adds at most 11 lines, all of them a marker
comment or an unused file. One says so in as many words: "the oracle is a pipeline integration test --
expect reward=0". The other three have no patch file at all (measured). So a throughput score says how
far an agent got past the untouched pin, not how close it got to what a human expert would do.

## Contamination

Every task comes from a real engine at a real commit, and most of the Day-0 and regression oracles
are upstream merges. A model trained on GitHub after those merges may have seen the answer. The
repository never uses the word for this (measured: I searched every task file). What it does is seal
the environment, and the sealing is thorough:

- No network, enforced by the sandbox (`allow_internet = false`), and a brief that forbids consulting
  "upstream history, pull requests, issues, commit messages, release notes, package mirrors and model
  hubs".
- No git history, no commit hash in the version string, no file dates, no lockfile.
- For Day-0 tasks, a build-time check that the model's own implementation is not importable anywhere.
  The Kimi-Linear task's `no_reference_impl.py` fails the image if `kimi_linear`, `modeling_kimi` or
  the author's `fla` kernel library is on any interpreter's path.

All of that blocks looking the answer up. None of it stops an answer the model memorised. That is
the ordinary limit of a benchmark built from public history (reasoned).

Two things push back. First, for the speed-up tasks, memorising the upstream fix only gets you the
oracle, and the results below show the agents going well past it, so a high score there cannot be
recall alone (reasoned). Second, the Day-0 tasks are recent. They name models such as
DeepSeek-V4.1-Flash, Qwen3.8-Flash-Next and MiniMax-M3, so for most agents on the board the merged
support probably postdates training. I could not check that, because the repo does not give dates
for the reference commits.

One forward-looking risk is new with this release. The open repo ships the oracle patches and the
"hidden" test inputs. The Kimi-Linear task's `tests/hidden/` holds nine JSON files of probes and
reference sequences, and `solution/solution.patch` is 4,831 lines. In all 20 throughput tasks the scored
workloads in `tests/workloads.json` are identical to the public ones the agent sees (measured). The
task suffix is `_open`. Any model trained after October 2026 can have read all of it, and the
leaderboard will need held-out tasks to stay meaningful (reasoned).

## The results

<Figure
  src="https://media.thesatyajit.com/ai/articles/rsi-inference-bench/fig4.png?v=daf00ea92bfb2a80"
  alt="Leaderboard table. Rank, agent, Day-0 Model Support pass@1, Inference Optimization speed-up, average cost, output tokens, steps, rating. 1 Sonnet-5.5 26.7%, 1.68x, $76. 2 Opus-5.5 13.3%, 1.48x, $83. 3 Fable-5.1 10.0%, 1.41x, $74. 4 GPT-6.1-Sol 10.0%, 1.37x, $12. 5 GPT-6-Astra 10.0%, 1.35x, $81. 6 Opus-5 0.0%, 1.35x, $114. 7 GPT-6-Sol 0.0%, 1.20x, $41. 8 GLM-5.3 0.0%, 1.22x, $32. 9 DeepSeek-V4.1-Flash 0.0%, 1.24x, $1.2. 10 Opus-4.8 0.0%, 1.14x, $118. 11 Gemini-3.8-Flash 0.0%, 1.07x, $60. 12 Gemini-3.7-Flash 0.0%, 1.04x, $57."
  caption="The joint board, snapshot 2026-10-12: Day-0 pass@1, optimization speed-up and mean cost per attempt for twelve agents, ranked by a Bradley-Terry rating over both tracks (RSI-Inference Bench leaderboard page, rendered offline from docs/leaderboard.html)."
/>

Twelve agents are on the board, 444 results scored, 658 attempts priced (measured, snapshot
`2026-10-12T05:39Z`). Which harness each agent ran in is not stated per row, and I have not seen a run, so the rows below
are the page's own numbers.

**The 26.7% is one track.** It is Sonnet-5.5's pass@1 on the ten Day-0 tasks and nothing else. The
other 27 tasks have no pass@1 at all. The README defines pass@1 as "full passes over attempts on each
task, averaged over the tasks". Sonnet-5.5 made 28 attempts: three on each task except Stable
Diffusion 3, which got one. It fully passed 8 of them: 1/3 on DeepSeek-V4-Flash, 3/3 on GLM-5.3-Flash,
1/3 on DeepSeek-V4.1-Flash, 2/3 on Inkling-Small and 1/3 on Kimi-Linear. That sums to 2.67, and 2.67 / 10
is 26.7% (measured, recomputed from the per-task records). The page's own interval for it is 10% to
46.7%. With ten tasks and three tries each, one more pass on one task moves the number by 3.3 points
(reasoned).

The tail is thin. Opus-5.5 is at 13.3%. Fable-5.1, GPT-6.1-Sol and GPT-6-Astra are at 10%. The other
seven agents never fully passed a Day-0 task. Five of the ten tasks, gpt-oss-120b in SGLang,
Qwen3.8-Flash-Next and diffusiongemma and MiniMax-M3 in vLLM, and Stable Diffusion 3, were never passed
by anyone. One task, GLM-5.3-Flash in SGLang, carries much of the board: two agents went 3/3 there and
three more passed at least once. (That is the model [whose own lab used an agent to build its serving
stack](/articles/glm-built-its-own-inference-infra), so there was public prior work to draw on.)

"Binary" is strict. A Day-0 run passes only if every fail-to-pass check passes and no pass-to-pass
check breaks (measured, `grader.py`). Kimi-Linear has 24 fail-to-pass checks: registry, config,
GSM8K accuracy, numerical fidelity against the model author's recording, long-context decode, prefix
cache reuse and turnover, data-parallel correctness under load, and decode and prefill efficiency
floors for the KDA kernels. It also has 273 pass-to-pass checks that guard the rest of the engine.
The board's tie-break, the share of those checks passed, shows how close the top agents come: Sonnet-5.5
passes 90.1% of them on average and Opus-5.5 87.6%. On DeepSeek-V4-Flash, across three attempts each,
both agents passed 84 of 87 checks. Sonnet-5.5 turned one of its attempts into a full pass; Opus-5.5
turned none (measured).

<ResultsBoard />

**The speed-up track tells a different story.** Here every agent gets a number. Sonnet-5.5 leads at
1.68x as a geometric mean over 27 tasks: 1.53x on the 20 throughput tasks and 2.19x on the 7
regression tasks. The last two rows are Gemini-3.8-Flash at 1.07x and Gemini-3.7-Flash at 1.04x
(measured).

The regression family has a yardstick, the maintainer's fix, and it is the most surprising thing on
the page. The seven oracles are worth between 1.156x and 1.62x (reported in each `task.toml`).
Sonnet-5.5, Opus-5.5, Fable-5.1 and GPT-6.1-Sol beat the oracle on all seven tasks, and five more agents beat it on five or six. On `regression_multimodal_prefill_ttft` Sonnet-5.5 reached
6.173x against an oracle of 1.62x. On the FP8 task in the explorer, nine of twelve agents beat 1.206x,
and the best is GPT-6.1-Sol at 1.526x (measured). The brief invites this ("stop when you cannot find
more, not when it looks fixed") and the score has no cap. So "Bug Fix" here means "start from a known
slowdown and make the deployment as fast as you can". Fixing the bug is the floor, not the goal
(reasoned).

The cost column is a per-attempt mean at list prices as of 2026-10-09, computed by the team from
token counts through a pinned LiteLLM price map (measured, the page's `prices` and `cost_summary`
blocks). Sonnet-5.5 averages \$76 an attempt. A Day-0 attempt costs more than an optimization one:
the medians are \$89.71 and \$39.14. GPT-6.1-Sol does its 10% for about \$12 an attempt and
DeepSeek-V4.1-Flash costs \$1.24, though the latter's figure leans on off-peak pricing, which the page
says it applied. The 637 attempts with a saved trajectory made 319,024 model requests between them (measured).

## Checking \$100K and 50K GPU hours

Neither total appears anywhere in the repository (measured: searched the page and every task file). What
the page does carry is a price for every scored attempt. Summed over the twelve agents and both
tracks, the snapshot's 658 attempts come to about \$44,500 at list prices, or \$47,900 as recorded
by the agents' own CLIs (measured). That is under half of \$100K. The rest is plausibly development:
building 37 tasks this careful means many runs that never reach a leaderboard. I cannot see those runs (the total is reported, the gap is
reasoned).

For GPU hours I multiplied each attempt by its task's GPU count and the agent's mean hours on that
track. That gives about 13,400 GPU-hours for the agent phase of the snapshot (reasoned, from measured
inputs). Verifiers come on top, on the same GPUs, and are not logged on the page. With Day-0 verifiers
allowed up to 44 hours, 50K in total is believable but uncheckable from here.

## What I ran

I ran what needs no GPU and no API key: the harness's own tests and its task loader. On a shared
4-core cloud VM, CPU only, in a Python 3.12 venv:

```bash
pip install -e third_party/pier pytest pytest-asyncio
python -m pytest -q tests            # inside third_party/pier
# 95 passed, 2 failed, 1 warning in 7.21s
```

Both failures are in `test_trial_verifier_artifact_transfer.py`, a mocked upload that was never
awaited, and persist with Harbor pinned to the lockfile's 0.23.0. Other packages resolved newer than
the lock, so they may be my environment rather than the harness (measured).

Then I loaded every task through the harness's own `pier.models.task.Task`, which parses `task.toml`
against its Pydantic schema and reads the instruction: 37 loaded, 0 failed (measured). I did not build
an image, start an engine or run an agent; every one of those needs the GPUs.

## What to take from it

The engineering of the graders is the contribution. Paired benches on one machine, warmup to a
plateau, rotated order, medians of paired ratios, a guard shape, a token-equality gate and a
documented noise floor are what make a "continuous reward" from a GPU benchmark mean anything. Anyone
building an [agent-written kernel](/articles/auto-gpu-kernel) leaderboard, or reading one like
[PTXBench, where an agent gamed the grade](/articles/ptxbench-kda-kernels), should read
`_bench_rounds.py` first.

The headline number undersells half the result and oversells the other half. Day-0 support at a
26.7% ceiling, with half the tasks unsolved by anyone, is a real wall: the best agents pass about nine
checks in ten and still miss the tenth. Optimization is the opposite. The best agent makes deployments
1.68x faster on average, and on the regression tasks nine of twelve agents beat a merged human fix on most
of them.

For the background, my [vLLM](/articles/vllm) and [SGLang](/articles/sglang) explainers cover the two
engines and [how LLM inference works](/articles/how-llm-inference-works) covers the prefill and decode
split every one of these workloads is built on. [Step 5 Preview](/articles/step-5-preview) is a recent
example of real day-0 support in vLLM. For other loops that carry the RSI name, see
[Terminal-RSI](/articles/terminal-rsi), which retrains the model,
[Dream-RSI](/articles/dream-rsi), which rewrites a search scheduler, and
[RSI-Anything](/articles/rsi-anything), which puts a world model in the loop.