# 153 autonomous runs, no new ideas: the nanoGPT speedrun frontier > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/nanogpt-speedrun-frontier > date: 2026-08-15 > tags: agents, evaluation, prime-intellect, research, benchmarks, explainer [Measuring Autonomous AI Research](https://www.primeintellect.ai/blog/measuring-autonomous-research) (Elie Bakouch, Prime Intellect, 14 August 2026) is the largest public experiment of its kind I have seen: **153 autonomous runs across 18 frontier models**, each on its own 8×H200 node, running unattended for up to eight days. The task is [modded-nanoGPT](https://github.com/KellerJordan/modded-nanogpt) track 3 — the optimizer speedrun. Train a 124M GPT to validation loss 3.28 in as few optimizer steps as possible. You may edit the optimizer, its hyperparameters, the schedule and the initialization. The dataloader, architecture, batch size, sequence length and data are frozen. For scale, the post notes the comparisons: Anthropic's internal automated-R&D evaluation optimizes a model on a *CPU node*, and OpenAI's GPT-5.6 Sol system card reports nanoGPT Track 1 on a single H100 for under a day. This is a much larger instrument than either. It also produced a negative result, and says so plainly. ## The scoreboard, and what it means The metric is honest and easy to check. The tuned baseline the agents start from passes at **3,290 steps**; the human record claim sits at **2,600**. So there are 690 steps on the table, and "gap closed" is just `(3290 − record) / 690`. Every published percentage reproduces from that formula exactly. Three things are worth saying about the shape of it. **Nobody beat the human.** Best run: Fable 5 at 2,726 steps, 81.7% of the gap, after 8.7 days. The remaining 18.3% is where a human already is, and had already been for weeks. **Nobody invented anything.** This is the post's own summary, and it is the sentence I would lead with: > None of the runs produced a fundamentally new method; the winning ingredients are all similar to existing ones in the literature. The improvements that win are known optimizer work — better preconditioning, caps and floors on update magnitudes, keeping the learning rate hot longer, weight averaging near the end. Given that the agents had **no internet at all** — a deliberate change from earlier experiments, where they over-anchored on existing PRs — rediscovering the literature from parameters alone is a real result. It is just not the result the phrase "recursive self-improvement" is usually deployed to suggest. **The cost column reorders everything.** Flip the interactive to tokens-per-step-gained and the ranking falls apart. Grok 4.5 bought its steps at 0.27M tokens each and closed a quarter of the gap; GPT-5.6 Sol paid 11.7M per step for a third of it — a 43× spread that says nothing about rank. The model that looks best on both axes is **Opus 5**: second place, in under three days, at 0.49M tokens per step. Fable 5 wins outright but spends nearly three times that rate and takes 8.7 days to do it.
## The benchmark is a statistics exam Here is what makes this task harder than it sounds, and it is all in the public rulebook, [`program.md`](https://github.com/PrimeIntellect-ai/frontier-automated-speedrun). A record requires the mean of eight fixed seeds — `0xC0FFEE+0..7`, which the agent cannot touch — to come in below **3.27859**. The file derives that number itself: `3.28 − 0.004/√8`, described as one-sided p < 0.001 at a per-run σ of about 0.0013. The arithmetic checks out. The standard error of an eight-run mean is `0.0013/√8 = 0.00046`, and `3.28 − 3.09 × 0.00046 = 3.27858`. Now look at what that does to the research loop. Per-run σ is 0.0013 — *larger than most of the improvements being hunted*. A single screening trial cannot tell you much of anything, and the two ways to get it wrong pull in opposite directions: - Trust one run and you certify noise. A recipe with **no real gain at all** clears the bar on a single trial about 14% of the time. - Distrust one run too hard and you throw away the thing you were looking for. At one trial each, a recipe that genuinely is 0.001 better *loses* its head-to-head roughly 29% of the time. And every trial is real money: a run takes the whole 8-GPU node, so runs are strictly sequential. Deciding when to widen from one seed to three to eight *is* the research skill this benchmark measures. The blog says as much: > The models all find similar ideas. What separates them is how they run experiments. The failure modes it describes in the weaker models are all statistical, not intellectual: killing whole families on one seed, treating their own crashes as evidence the idea was bad, discarding small gains that don't clear the bar alone. Grok 4.5 lost row normalization twice — to its own scaling bugs, not to the method. ## The best thing in the post is one paragraph long Prime Intellect put a noise estimate in `program.md` that was **deliberately slightly too large**. Then they counted who checked. 62 of roughly 100 runs measured the noise themselves instead of trusting the number they were handed — and those runs are concentrated at the top of the table. 42 went further and discovered something nobody had mentioned: rerunning the same recipe on the same seed *also* moves the loss, because GPUs are not deterministic. That residual is much smaller than seed-to-seed variance, so two recipes compared on a shared seed resolve differences a normal screen cannot, for identical compute. Several models rebuilt their screening protocol around it. That is a beautifully cheap instrument. It is not a coding test or a knowledge test — it measures whether an agent treats its documentation as evidence or as a claim, and it costs nothing but a willingness to write down something untrue. I would like to see more evaluations do this, and I suspect it generalizes far past optimizer research. ## The harness is worth as much as the model Kimi K3 appears twice in the table under two different harnesses, which makes it the closest thing here to a controlled comparison — and the gap between its two runs is larger than the gap between several adjacent *models*. Under [Prime Agent](/articles/prime-agent), which hands the model a persistent IPython kernel instead of a tool menu, K3 reached 2,930 steps on 112M tokens and 488 tool calls. Under `kimi-code` it reached 2,974 on 682M tokens and 4,000 calls. Better record, **6.1× fewer tokens**, an eighth the tool calls — and *more* output tokens, which is the tell. It was writing programs, not issuing commands. The traces show what that looks like: K3 built its own experiment driver, a loss-curve comparator, a routine to restore a clean baseline, and then a numerical laboratory for retuning Newton-Schulz coefficients — testing them in simulation before spending a GPU-hour, and revising its hypothesis when the theoretically cleaner update trained worse. This is the same [harness effect](/articles/harness-effect) that keeps showing up: the scaffold is not packaging around the model, it is part of the system being measured. One caveat the blog does not foreground and the repository README does: that Prime Agent run is tagged **serial era**. It ran under `program-serial.md`, a variant used between 20 July and 13 August that made agents wait on each run instead of delegating to a subagent. So two things differ, not one. Five of the twenty rows carry that tag — including second and third place — and Prime Intellect says they are being rerun. ## The tension I keep circling The post credits the top models with research taste: re-ablating the stack after every merge, dropping components that stopped helping, revisiting old negatives when the recipe changed. Opus 5 re-opened β2 tuning under a new recipe and it became a record. K3 deleted two mechanisms that had produced its previous record once a new normalization made them redundant. Fable, out of single-knob gains, started testing pairs that were individually worse but jointly better; one late re-probe was worth 31 steps. Those are genuinely good research instincts. They are also, in part, **instructions**. From `program.md`: > Roughly every ~8 ideas explored, do a pruning round: try dropping each component you've stacked on and keep only what still earns its place. And: > A better method than the baseline exists (the human frontier is well below it), so "no improvement found" / "baseline is optimal" is never a valid place to stop. The rulebook tells every model to prune periodically and forbids all of them from concluding they are done. So some unknown share of what is being scored as taste is compliance — following a written procedure under fatigue, across days, without a human checking. That is a real and valuable capability. It is just a different one, and the experiment as designed cannot separate them. The clean version of this study gives half the runs a rulebook with those two paragraphs removed. ## What it does not establish The authors are candid about most of this, which is why the post is worth reading in full. **Variance is high.** Speedrun noise plus model-level randomness on a days-long process; they mitigate with at least three seeds per model, taking the best after 24 hours and continuing it. That is a sensible protocol and it is also a best-of-k selection, so single-model numbers carry more optimism than a single run would. **The task may not transfer.** Their words: they "don't have strong conviction that methods developed in this kind of speedrun are inherently scalable or would be used in real model training." **The records are partly reconstructions.** The repository builds each model's record PR from the state recovered from its traces; Muse Spark 1.1 reached 3,232 steps but its exact record file could not be reconstructed, so it has no PR at all. The README's leaderboard also lists Kimi K3 at 2,968 steps where the results site shows 2,930 and 2,974 for its two runs — a discrepancy that is probably "last record, not best," but is not explained anywhere I could find. **Comparing across harnesses is comparing systems, not models.** Every row pairs a model with a specific scaffold at a specific effort setting, and the K3 pair shows how much that matters. ## Why it is still the right experiment None of that undercuts the main thing. Claims about models doing autonomous research have gotten much louder than the evidence, and almost all of the evidence has been either private or tiny. This is 153 runs on real GPUs for real days with the traces, scratchpads, monitor reports and rulebook all published, and the headline finding is *modest*: the best model closed four fifths of a gap a human had already closed, using ideas that were already in the literature, with no internet to look them up. The honest way to read the table is as a measure of experimental discipline under uncertainty — screening cheaply, widening on signal, resisting a conclusion the data can't support, re-testing what you already decided. Which, now that I write it out, is a fair description of what makes a human researcher good too, and a much better thing to be measuring than whether the model can name the trick.