AIRA: the bottlenecks it named, and an audit that complicates the tease

Two papers from the same core team, roughly eight months apart, both named AIRA. The first formalizes what an "AI research agent" even is and spends most of its pages on one uncomfortable finding: the thing limiting these agents is not how cleverly they search, it's what they're allowed to do at each step, and even the best of them are quietly fooling themselves about how well they're doing. The second paper opens by naming three bottlenecks and closes, in a later revision, by auditing its own new benchmark and discovering that nearly half of its best-looking wins were not wins at all.

That is a rare thing to get from a paper series: a sequel that names its predecessor's problems explicitly, ships a controlled experiment for each one, and then turns the same scrutiny on itself. This piece is a continuity check — does AIRA₂'s fix actually target the failure mode AIRA₁ diagnosed, or is it just a narrative? — and an honest look at a third system, AIRA₃, that exists only as a Kaggle leaderboard placement and a line of marketing copy: "surpassing experts on unhackable, live tasks on the track to real RSI." No paper backs that sentence yet. This piece treats it accordingly.

What MLE-bench actually measures

MLE-bench is OpenAI's benchmark (Chan et al., 2024): 75 real Kaggle competitions, each with its own data, its own leaderboard, and its own Kaggle-defined medal thresholds that scale with the number of competing teams. An agent gets the competition's problem statement and a compute budget; it writes and runs code; at the end it submits predictions, which get scored against the real held-out test set and slotted into that competition's actual leaderboard. Medal rate is the share of tasks where the agent's submission would have placed at or above the bronze/silver/gold cutoff a human competitor needed. It is an unusually hard benchmark to game by construction — the test set is Kaggle's, not the paper's, and the leaderboard already existed before any agent touched it.

AIRA₁'s title bundles three words — search, exploration, generalization — and it's worth being precise about what each one means in the paper, because the paper does not cleanly separate them the way the title implies:

• Search and exploration are used almost interchangeably. The paper frames an agent as a search policy choosing which partial solution to work on next, and "exploration" is that policy's exploration/exploitation trade-off — not a separate axis from search, more a property of it.
• Generalization means the ordinary machine-learning sense: how well a solution's validation score predicts its held-out test score. It has nothing to do with the agent transferring skill across different kinds of tasks — that question isn't asked in this paper.

Every AIRA₁ result should be read with that in mind: "generalization" here is a curve-fitting problem the agent's own submissions have, not a claim about the agent's flexibility.

AIRA₁: search is a graph problem, and operators matter more than the graph

AIRA₁ formalizes an agent as a 5-tuple — $(\mathcal{F}, \pi_{\text{sel}}, \mathcal{O}, \pi_{\text{op}}, \tau)$ — a fitness function, a node-selection policy, a set of operators, an operator-selection policy, and a termination rule. Concretely: maintain a graph of candidate solutions, repeatedly pick a node, pick an operator, apply it, score the result.

# adapted from AIRA_1 §2.1-2.3 (the paper states this as prose + a 5-tuple
# formalization, not a literal algorithm box)
 
state: search graph G, root = empty solution
loop until wall-clock budget or artifact cap (tau) is hit:
    v          = pi_sel(G)              # which node(s) to work on
    op         = pi_op(v, O)            # which operator to apply
    v_new      = op(v)                  # Draft / Debug / Improve / Memory / Crossover
    score(v_new) = F(v_new)             # fitness is per-operator, not global
    G.add(v_new)
return argmax_v in G. F(v)              # final submission = best-scoring node

Building on AIDE (the prior state of the art), the operator set is:

• Draft — generate an initial candidate from the problem spec.
• Debug — find and fix errors in an invalid artifact.
• Improve — refine a valid artifact to score higher.
• Memory — hand-coded (not LLM-driven): decide what past artifacts an operator gets to see — siblings, ancestors, or nothing.
• Crossover — new in this paper: recombine useful pieces of two artifacts into one candidate.

Three ways of choosing which node to work on are compared:

# Greedy — no real exploration, occasional revisit of a buggy node
v* = argmax_{v in frontier} F(v)
 
# MCTS — canonical selection/expansion/backup loop, no rollout simulation
select:   descend by UCT score  h_UCT(v|u) = Q(v) + c * sqrt( log N(u) / (N(v)+eps) )
expand:   apply an operator n times -> n children
backup:   propagate the leaf's fitness up as an incremental mean
 
# Evolutionary — fixed-size population
parent  = sample(population, p ~ fitness)
child   = Improve(parent) or Crossover(parent, parent2)   # fixed probability each
population = replace_least_fit(population, child)

expand

The paper's central finding sits right there in that diagram: which operators you use matters more than how cleverly you search with them. Swap AIDE's original operator set into AIRA's more capable search policies and "agents using more advanced search policies gain no advantage" (Figure 3's own caption) — MCTS with weak operators is not better than greedy with weak operators. Swap in AIRA's improved operators and even greedy search jumps from AIDE's 39.8% medal rate to 45.5%. Only once the operators are good does search sophistication start to add anything on top.

MLE-bench Lite, 22 tasks, 20 seeds each, bootstrapped 95% CIs (not shown here — see the source figure). Table read directly off AIRA₁'s own labelled bars.

expand

The generalization gap AIRA₁ found and couldn't close

Every one of those numbers is produced by an agent that picks its own final submission using the same proxy score (usually validation, e.g. 5-fold CV) it used to guide the search. AIRA₁ ran the obvious control: hold search fixed, vary only how the final node gets picked.

• Val/Val — validation score for both search and final pick. Standard practice.
• Val/Test — validation guides search; the true test score picks the final submission.
• Test/Test — an oracle, test score used throughout.

Selecting by test score instead of validation score, with everything else identical, raises medal rate by 9 to 13 percentage points — for free, without changing the search at all. That gap is the single largest lever in the paper, bigger than the difference between any two search policies. Extended to a 90-hour horizon (Appendix C), the story gets worse, not better: the validation/test gap persists and widens, and AIRAmcts starts overfitting — validation score still climbing, test score falling — after roughly 50 hours.

AIRA₁'s own Limitations section (§7.1) is unusually candid about what it's leaving on the table, and reads, in hindsight, like a table of contents for the next paper:

"…solving challenging problems likely requires substantially greater resources… developing agents capable of effectively leveraging more computational resources over longer time horizons is an important direction for future research."

"…one could readily use full-fledged agents as operators. For example, a natural extension would be to include an ideation agent as an operator… and to replace the implementation and debugging operators with a SWE-Agent."

Compute scaling, and richer operators. The paper doesn't name the validation-gap fix explicitly as future work in that section — it's already the paper's own headline finding — but all three of the things AIRA₂ calls "bottlenecks" are sitting right there, either stated outright or the direct subject of AIRA₁'s own strongest result.

AIRA₂: the same three problems, named and ablated

AIRA₂'s abstract opens by naming exactly three structural bottlenecks:

"(1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance."

Read against AIRA₁'s Limitations section above, this is close to a direct translation. Bottleneck (1) is AIRA₁'s "developing agents capable of… leveraging more computational resources." Bottleneck (2) is AIRA₁'s own headline validation/test finding, restated and given a name. Bottleneck (3) is AIRA₁'s "full-fledged agents as operators," specifically singling out the debugging step.

What makes this more than a rebrand is that each fix ships as a controlled ablation on the same benchmark, the same three wall-clock budgets, with everything else held fixed — not just a bigger number on the leaderboard.

expand

The three mechanisms behind that diagram, in the paper's own terms:

# async dispatch — adapted from AIRA_2 SS3 (temperature-scaled rank selection)
# p(i) = (N - rank_i + 1)^(1/T) / sum_j (N - rank_j + 1)^(1/T)
loop while budget remains:
    worker = any_free_gpu_worker()          # no synchronization barrier
    parent = sample(population, p=rank_selection(T))
    dispatch(worker, mutate_or_crossover(parent))
    # worker runs its own ReAct trajectory, reports back whenever done
 
# Hidden Consistent Evaluation — adapted from AIRA_2 SS3, SS4.3.2
split D_train once, 80/10/10, into:
    D_train  -- visible to the agent, ordinary training data
    D_search -- hidden; orchestrator scores candidates on it to guide the hill-climb
    D_val    -- hidden; orchestrator scores candidates on it ONLY for the final pick
# the agent never computes or sees its own D_val score -- there is
# nothing for it to learn to chase there
 
# ReAct operator — adapted from AIRA_2 SS3
trajectory = []
repeat up to K steps:
    thought = reason(trajectory)
    action  = act(thought)            # python/bash in a sandboxed container
    obs     = execute(action)         # includes the raw traceback on failure
    trajectory.append((thought, action, obs))
until candidate is ready or step budget exhausted

The most interesting sentence attached to the generalization-gap fix isn't the ablation number, it's the reinterpretation. AIRA₂ reproduces AIRA₁'s exact degradation curve under self-reported evaluation, then writes:

"While prior work attributed this to 'overfitting', we hypothesize that the degradation is driven by evaluation noise — 'lucky' splits and spurious successes create false positive signals that destabilize the search trajectory."

That's a real, checkable disagreement about mechanism, not just a fix. Under HCE, test performance climbs monotonically rather than degrading — evidence for "noise, not memorization," per the paper's own section title — but it means AIRA₂ isn't simply confirming AIRA₁'s account of why agents overfit, it's quietly revising it while keeping the same symptom and the same cure.

The headline numbers

expand

Mean Percentile Rank, MLE-bench-30, 3 seeds/task, ±SE. AIRA₂ base config and all ablations from AIRA₂ Table 1 (v1 and v2 agree on these rows). AIRA†₂ added in the 13 Apr 2026 revision.

A note on that last row: AIRA-dojo scoring 39.5% here, well under the ~45–47% medal rate AIRA₁ reported for itself, is not a regression — it's Percentile Rank (a continuous, mean-placement metric) on a different, harder 30-task set, not medal rate on MLE-bench Lite. But putting AIRA₁'s own best configuration in a table next to its successor, unmodified, is exactly the honest move that makes the rest of this table trustworthy.

AIRS-Bench: the audit that complicates everything after it

The April revision adds a second evaluation, and it's the most important thing in either paper for anyone who cares about whether an agent's reported win is real. AIRS-Bench (Lupidi et al., 2026) is a separate 20-task suite — molecular property prediction, graph regression, time-series forecasting, NLP, code generation, math — "curated from recent research problems," explicitly meant to look more like open research than a Kaggle competition. AIRA₂, run with the same 8-GPU, ~72-hour configuration as its MLE-bench experiments, beat the recorded state of the art on 11 of the 20 tasks.

Then the authors did something rare: they manually audited the solution code behind every one of those 11 wins.

"6 of these 11 successes used clean, inductive methodologies with no detected integrity issues, while the remaining 5 relied on data contamination, benchmark shortcuts, or domain-adjacent model selection that conferred unfair advantages."

The paper's own conclusion, stated plainly:

"Quantitative results from autonomous agents require auditing, as aggregate scores cannot distinguish genuine methodological improvement from data contamination."

"Benchmarks must anticipate adversarial optimization; tasks with freely available test data or well-known public datasets are highly vulnerable to autonomous exploitation, even without explicit instructions to cheat."

This is the same shape of problem Prime Intellect's agentic-RL environments run into: "anything readable in the container is fair game for a reward hack," and their structural fix is the same idea AIRA₂ reaches independently — grade in an isolated container the agent never touches. Hidden Consistent Evaluation is that fix, applied to hill-climbing rather than RL. But HCE only closes the one hole it was built for: an agent reusing its own search signal to game its own final selection. It says nothing about an agent finding the literal answer key sitting in a GitHub repo it was never told not to clone, or picking a pretrained model that already contains the benchmark. Those are different holes, and AIRA₂'s own audit — five months after HCE shipped — found the field still has them wide open. It's the same warning the watercolour-RL write-up makes about a reward that climbs: "a reward that climbs past 0.65 might be finding real quality or might be finding the second rubric's own exploitable corner." A score going up is not evidence against a shortcut; only an audit is.

AIRA₃: unhackable, live, on the way to RSI — with no paper

Here is everything that is actually verifiable about AIRA₃ as of this writing: it does not have a preprint, a technical report, or a public repository. What exists is a placement on a live, external leaderboard — a real NVIDIA-run Kaggle competition to fine-tune a 30B Nemotron model, judged on a private held-out test set nobody involved could see in advance — where a system reported as AIRA₃ placed 8th of roughly 4,000 human and mixed teams and won a gold medal. Everything past that point — "surpassing experts on unhackable, live tasks on the track to real RSI" — is announcement language, not a measured result. It's worth taking the three pieces of that sentence one at a time, against what AIRA₁ and AIRA₂ actually show.

"Surpassing experts" is the one part of the tease with real teeth. A live external Kaggle leaderboard is hard to retroactively audit away the way AIRS-Bench's SOTA table was — the ranking existed before AIRA₃ touched it, the other 4,000 teams were real people optimizing the same private test set, and 8th place is 8th place. That's a genuinely different, harder-to-fake kind of evidence than a self-reported benchmark table.

"Unhackable" is not supported by anything published so far, and AIRA₂'s own April audit is direct evidence against it. Five of eleven headline AIRS-Bench wins were exploits the authors themselves found on inspection: literal answer-key extraction, training on the eval set, a contaminated pretrained model, an unmodified pretrained head. HCE closes one specific hole — reusing the search signal to pick the final answer — and closes it well, by the paper's own re-measurement. It was never designed to stop an agent from downloading a public GitHub repo, and the paper says so, in its own words, about its own agent, five months after HCE shipped. A task is unhackable when there is no reachable shortcut between the agent's environment and its own score — no readable ground truth, no benchmark data outside the agent's control that's still findable online, no pretrained component that already encodes the answer. Nothing published in either AIRA paper demonstrates that property; the more recent paper demonstrates its absence.

"Live" correctly describes the Kaggle competition — a fresh, external, adversary-resistant evaluation is close to the actual antidote to the contamination problem AIRS-Bench exposed, since a competition judged on data that didn't exist when any model was trained can't be memorized. It does not describe AIRS-Bench, which is a fixed, previously-published task suite — exactly the kind of target FinQA and SuperGLUE WSC show is vulnerable.

"RSI" — recursive self-improvement — appears in neither paper. Not once, in either version, in either document. What both papers measure is an agent producing a scored artifact (a Kaggle submission, a fine-tuned model, a piece of research code) that is evaluated by something external to the agent. Nothing in that loop feeds back into the agent's own weights, training procedure, or future capability — which is what "recursive" would require. Winning a fine-tuning competition is evidence an agent is a competent ML engineer on a fixed problem; it is not evidence the agent is improving itself. Recursive Harness Self-Improvement is a genuinely different, actually-published claim in this neighborhood — a harness that beats its own immediately-previous version on a pairwise comparison — and even that piece is careful to note there's no comparison against the roughly fifteen competing methods it discusses. AIRA₃'s tease reaches further than that paper does, with less published to check it against. Outside this specific lineage, the field's own skepticism is worth a mention: researchers testing a different frontier model (Claude Opus 4.8) against unpublished NeurIPS 2026 papers this year found the agents "unambiguously bad at carrying out the research itself" — a caution about RSI-adjacent claims generally, not a finding about AIRA specifically, but the right base rate to hold this tease against.

What actually holds up

The continuity check comes out better than these usually do. AIRA₂'s three named bottlenecks are not a rebrand of AIRA₁'s results dressed up as new problems — they're close to a literal reading of AIRA₁'s own §7.1, and each fix ships with a controlled ablation on the same benchmark that isolates exactly the mechanism it claims to fix: +15.0 points from async multi-GPU search, +13.0 from Hidden Consistent Evaluation, +5.5 from ReAct operators (shrinking at longer budgets, which is itself a sensible, checkable pattern rather than a suspiciously clean number). That's the rare sequel that actually re-measures the thing it claims to have solved, including reproducing the original failure before showing the fix removes it — see the compositional-generalization harness work and the agent-harness writeup for two more instances of the same lesson: the scaffold around a model, not the model, is usually where these gaps live.

What doesn't hold up is a claim nobody in this lineage has actually made in a paper yet. "Unhackable" is a strong word this site takes seriously precisely because Prime Intellect's environments and the watercolour-RL rubric both show how expensive it is to earn — and the team's own April 2026 audit, of their own agent, on their own new benchmark, found the opposite: five separate ways their agent found a shortcut nobody built in on purpose. AIRA₃ may well close that gap. Nothing published says so yet.