2026-08-15 · 10 min · agents · evaluation · prime-intellect · research · benchmarks · explainer
Measuring Autonomous AI Research (Elie Bakouch, Prime Intellect, 14 August 2026) is the largest public experiment of its kind I have seen: 153 autonomous runs across 18 frontier models, each on its own 8×H200 node, running unattended for up to eight days.
The task is modded-nanoGPT track 3 — the optimizer speedrun. Train a 124M GPT to validation loss 3.28 in as few optimizer steps as possible. You may edit the optimizer, its hyperparameters, the schedule and the initialization. The dataloader, architecture, batch size, sequence length and data are frozen.
For scale, the post notes the comparisons: Anthropic's internal automated-R&D evaluation optimizes a model on a CPU node, and OpenAI's GPT-5.6 Sol system card reports nanoGPT Track 1 on a single H100 for under a day. This is a much larger instrument than either.
It also produced a negative result, and says so plainly.
The scoreboard, and what it means
Every published percentage reproduces exactly from (3290 − record) / 690, so the scoreboard is doing nothing clever. The shape it hides is that the top four are a different population from the rest: four runs above 45%, then a cliff to 39% and a long tail that never gets past a quarter of the gap. And nobody reached 100%, which is where a human already is.
The metric is honest and easy to check. The tuned baseline the agents start from passes at 3,290 steps; the human record claim sits at 2,600. So there are 690 steps on the table, and "gap closed" is just (3290 − record) / 690. Every published percentage reproduces from that formula exactly.
Three things are worth saying about the shape of it.
Nobody beat the human. Best run: Fable 5 at 2,726 steps, 81.7% of the gap, after 8.7 days. The remaining 18.3% is where a human already is, and had already been for weeks.
Nobody invented anything. This is the post's own summary, and it is the sentence I would lead with:
None of the runs produced a fundamentally new method; the winning ingredients are all similar to existing ones in the literature.
The improvements that win are known optimizer work — better preconditioning, caps and floors on update magnitudes, keeping the learning rate hot longer, weight averaging near the end. Given that the agents had no internet at all — a deliberate change from earlier experiments, where they over-anchored on existing PRs — rediscovering the literature from parameters alone is a real result. It is just not the result the phrase "recursive self-improvement" is usually deployed to suggest.
The cost column reorders everything. Flip the interactive to tokens-per-step-gained and the ranking falls apart. Grok 4.5 bought its steps at 0.27M tokens each and closed a quarter of the gap; GPT-5.6 Sol paid 11.7M per step for a third of it — a 43× spread that says nothing about rank. The model that looks best on both axes is Opus 5: second place, in under three days, at 0.49M tokens per step. Fable 5 wins outright but spends nearly three times that rate and takes 8.7 days to do it.
The benchmark is a statistics exam
Here is what makes this task harder than it sounds, and it is all in the public rulebook, program.md.
A record requires the mean of eight fixed seeds — 0xC0FFEE+0..7, which the agent cannot touch — to come in below 3.27859. The file derives that number itself: 3.28 − 0.004/√8, described as one-sided p < 0.001 at a per-run σ of about 0.0013. The arithmetic checks out. The standard error of an eight-run mean is 0.0013/√8 = 0.00046, and 3.28 − 3.09 × 0.00046 = 3.27858.
Two errors, pulling opposite ways. Set trials to 1: a recipe with no real improvement at all clears the bar about 14% of the time, which is why records need eight fixed seeds. But the error that actually costs you a result is the other one — at one trial each, a recipe that genuinely is 0.0010 better loses its head-to-head 29% of the time. Kill it there and the idea is gone. That is exactly the failure the blog describes in the weaker models: families killed on one seed, small gains thrown away for not clearing a bar they were never meant to clear alone.
The third box is the trick 42 runs discovered without being told. Comparing two recipes on a shared seed cancels the seed-to-seed component and leaves only GPU nondeterminism, so the same compute resolves a much smaller difference. Prime Intellect says that residual is “much smaller” but never quantifies it — the slider is yours to set, not a measurement.
Now look at what that does to the research loop. Per-run σ is 0.0013 — larger than most of the improvements being hunted. A single screening trial cannot tell you much of anything, and the two ways to get it wrong pull in opposite directions:
- Trust one run and you certify noise. A recipe with no real gain at all clears the bar on a single trial about 14% of the time.
- Distrust one run too hard and you throw away the thing you were looking for. At one trial each, a recipe that genuinely is 0.001 better loses its head-to-head roughly 29% of the time.
And every trial is real money: a run takes the whole 8-GPU node, so runs are strictly sequential. Deciding when to widen from one seed to three to eight is the research skill this benchmark measures. The blog says as much:
The models all find similar ideas. What separates them is how they run experiments.
The failure modes it describes in the weaker models are all statistical, not intellectual: killing whole families on one seed, treating their own crashes as evidence the idea was bad, discarding small gains that don't clear the bar alone. Grok 4.5 lost row normalization twice — to its own scaling bugs, not to the method.
The best thing in the post is one paragraph long
Prime Intellect put a noise estimate in program.md that was deliberately slightly too large. Then they counted who checked.
62 of roughly 100 runs measured the noise themselves instead of trusting the number they were handed — and those runs are concentrated at the top of the table. 42 went further and discovered something nobody had mentioned: rerunning the same recipe on the same seed also moves the loss, because GPUs are not deterministic. That residual is much smaller than seed-to-seed variance, so two recipes compared on a shared seed resolve differences a normal screen cannot, for identical compute. Several models rebuilt their screening protocol around it.
That is a beautifully cheap instrument. It is not a coding test or a knowledge test — it measures whether an agent treats its documentation as evidence or as a claim, and it costs nothing but a willingness to write down something untrue. I would like to see more evaluations do this, and I suspect it generalizes far past optimizer research.
The harness is worth as much as the model
The Prime Agent run is tagged serial era: it ran under program-serial.md, a rulebook variant used between 20 July and 13 August that made agents wait for each training run instead of delegating it to a subagent. So two things differ, not one, and Prime Intellect says those runs are being redone under the standard rulebook. Five of the twenty rows carry this tag — including second and third place.
Even with that caveat, this is the most interesting pair in the table. One model, run two ways, lands 44 steps apart on the record and 6.1× apart on token spend — a bigger swing than separates several adjacent models from each other. Prime Agent gives the model a persistent IPython kernel instead of a tool menu, and the traces show what that buys: K3 wrote its own experiment driver, its own loss-curve comparator, a routine to restore a clean baseline, and eventually a numerical laboratory for retuning Newton-Schulz coefficients before spending a single GPU-hour testing them. The token split is the tell: under Prime Agent it wrote more output — 2.2M against 1.4M — across an eighth as many tool calls. That is the signature of a model writing programs instead of issuing commands, and programs are why its total context stayed small enough to fit in 112M tokens.
Kimi K3 appears twice in the table under two different harnesses, which makes it the closest thing here to a controlled comparison — and the gap between its two runs is larger than the gap between several adjacent models.
Under Prime Agent, which hands the model a persistent IPython kernel instead of a tool menu, K3 reached 2,930 steps on 112M tokens and 488 tool calls. Under kimi-code it reached 2,974 on 682M tokens and 4,000 calls. Better record, 6.1× fewer tokens, an eighth the tool calls — and more output tokens, which is the tell. It was writing programs, not issuing commands.
The traces show what that looks like: K3 built its own experiment driver, a loss-curve comparator, a routine to restore a clean baseline, and then a numerical laboratory for retuning Newton-Schulz coefficients — testing them in simulation before spending a GPU-hour, and revising its hypothesis when the theoretically cleaner update trained worse. This is the same harness effect that keeps showing up: the scaffold is not packaging around the model, it is part of the system being measured.
One caveat the blog does not foreground and the repository README does: that Prime Agent run is tagged serial era. It ran under program-serial.md, a variant used between 20 July and 13 August that made agents wait on each run instead of delegating to a subagent. So two things differ, not one. Five of the twenty rows carry that tag — including second and third place — and Prime Intellect says they are being rerun.
The tension I keep circling
The post credits the top models with research taste: re-ablating the stack after every merge, dropping components that stopped helping, revisiting old negatives when the recipe changed. Opus 5 re-opened β2 tuning under a new recipe and it became a record. K3 deleted two mechanisms that had produced its previous record once a new normalization made them redundant. Fable, out of single-knob gains, started testing pairs that were individually worse but jointly better; one late re-probe was worth 31 steps.
Those are genuinely good research instincts. They are also, in part, instructions. From program.md:
Roughly every ~8 ideas explored, do a pruning round: try dropping each component you've stacked on and keep only what still earns its place.
And:
A better method than the baseline exists (the human frontier is well below it), so "no improvement found" / "baseline is optimal" is never a valid place to stop.
The rulebook tells every model to prune periodically and forbids all of them from concluding they are done. So some unknown share of what is being scored as taste is compliance — following a written procedure under fatigue, across days, without a human checking. That is a real and valuable capability. It is just a different one, and the experiment as designed cannot separate them. The clean version of this study gives half the runs a rulebook with those two paragraphs removed.
What it does not establish
The authors are candid about most of this, which is why the post is worth reading in full.
Variance is high. Speedrun noise plus model-level randomness on a days-long process; they mitigate with at least three seeds per model, taking the best after 24 hours and continuing it. That is a sensible protocol and it is also a best-of-k selection, so single-model numbers carry more optimism than a single run would.
The task may not transfer. Their words: they "don't have strong conviction that methods developed in this kind of speedrun are inherently scalable or would be used in real model training."
The records are partly reconstructions. The repository builds each model's record PR from the state recovered from its traces; Muse Spark 1.1 reached 3,232 steps but its exact record file could not be reconstructed, so it has no PR at all. The README's leaderboard also lists Kimi K3 at 2,968 steps where the results site shows 2,930 and 2,974 for its two runs — a discrepancy that is probably "last record, not best," but is not explained anywhere I could find.
Comparing across harnesses is comparing systems, not models. Every row pairs a model with a specific scaffold at a specific effort setting, and the K3 pair shows how much that matters.
Why it is still the right experiment
None of that undercuts the main thing. Claims about models doing autonomous research have gotten much louder than the evidence, and almost all of the evidence has been either private or tiny. This is 153 runs on real GPUs for real days with the traces, scratchpads, monitor reports and rulebook all published, and the headline finding is modest: the best model closed four fifths of a gap a human had already closed, using ideas that were already in the literature, with no internet to look them up.
The honest way to read the table is as a measure of experimental discipline under uncertainty — screening cheaply, widening on signal, resisting a conclusion the data can't support, re-testing what you already decided. Which, now that I write it out, is a fair description of what makes a human researcher good too, and a much better thing to be measuring than whether the model can name the trick.