~/satyajit

Ornith-1.5: the model writes the exam, builds the marking scheme, then sits it

mdjsonmcp

2026-08-23 · 8 min · open-weights · rl · agents · self-improvement · explainer

Most RL post-training pipelines have three fixed inputs: a set of human-curated tasks, a hand-designed harness that scores them, and a policy that learns against both. Ornith-1.5 makes all three learned, in the same loop, with one reward signal propagated across them.

The model proposes new tasks. For each task it generates or refines a scaffold — the instructions, tools, decomposition strategy and orchestration used to attack the problem. Conditioned on both, the policy produces solution rollouts. Reward from the rollout flows back through all three stages, so the system learns to produce better solutions and to generate more useful training tasks and to construct more effective scaffolds. All three are optimized with GRPO.

ReleaseOrnith-1.5 · HF collection · Aug 2026
Models397B MoE, 35B-A3B MoE, 9B dense — plus a quantized 9B-Mobile for phones
LicenseMIT, with FP8, GGUF, MLX and NVFP4 builds
Lineageextends Ornith-1.0, itself built on Qwen3.5 and Gemma 4 with CPT, mid-training and post-training
FlagshipTerminal-Bench 2.1 86.1 · SWE-bench Verified 86.0 · HLE 44.6 · DeepSWE 56.0
ReferenceClaude Opus 4.8 scores 85.0 and 85.8 on the first two
Protocolevery number averaged over five independent runs, with anti-hacking safeguards

The loop

Each training cycle is three stages. Given an environment or codebase, high-level instructions about the task type, and the model's own history of what it has already solved, the system proposes progressively harder tasks that go beyond that history — exposing capability gaps and pushing the training frontier outward.

Then for each task it builds a scaffold. Then the policy attempts it. Reward propagates back through all three.

task generation, scaffold construction and rollouts — all three optimized with GRPOstage 1: propose a task
A three-stage loop: propose a task, build a scaffold for it, attempt it. Reward from the attempt flows back to all three stages, and the solved-task history and novelty buffer feed forward into proposing the next one.environment / codebasetask-type instructionssolve history1propose a taskproduces a question qR_task = V · D · N2build a scaffoldproduces a harness hR_harness = C · F · H3attempt itproduces rollouts τR_rollout = h(q, τ)reward propagatesto all three stagestask buffer ℬalready generated / trained onclick a stage
R_task = V · D · N
V(q,s)is it valid and verifiable? — a hard gate: V = 0 zeroes everything
D(q,s,{τ})is it at the right difficulty? — a Gaussian on the model's own success rate, peaked at p★ = 0.2
N(q)is it sufficiently novel? — 1 minus the max similarity against the task buffer
Given an environment or codebase, high-level instructions about the task type, and the model's own history of what it has already solved, the system proposes progressively harder tasks that go beyond that history. Difficulty is estimated from the current policy's rollouts, so the curriculum moves as the policy improves.

Read the two arrows on the left as the actual claim. Reward from a rollout does not just update the policy — it updates the thing that proposed the task and the thing that graded it, so a task that produced useful learning signal makes the proposer more likely to produce tasks like it, and a harness that scored honestly makes the scaffold generator more likely to build harnesses like that.

Both rewards are products of three terms rather than sums, and in both cases the third term is the adversarial one. On the task side, novelty stops the proposer farming variations of a single good problem. On the harness side, hack resistance is the only thing standing between “the model designs its own grader” and the obvious failure mode — and it is the term defined in a sentence and never measured.

The interesting engineering is in how those rewards are shaped, because a proposer that is rewarded for difficulty will happily generate nonsense.

Rtask = V(q,s) · D(q,s,{τ}) · N(q)reward 0.850
The frontier-difficulty term as a Gaussian bump peaking at a twenty-per-cent rollout success rate, falling away toward both trivial and impossible tasks0.00.51.0p★ = 0.2D = 1.000impossibletrivial0.00.20.40.60.81.0p — the model’s own rollout success rate on the task it just proposed
success rate p20%
σ (unpublished)0.15
novelty N0.85
V 1.00×D 1.000×N 0.85=0.850near the frontier

Drag p from 0.2 rightwards and watch the reward collapse. That is the self-curricular mechanism in one motion: as the policy gets better at a task, the reward for proposing that task falls, so the generator is pushed toward harder problems without anyone writing a schedule. The curriculum evolves because difficulty is measured against the current model’s own rollouts rather than against a fixed rubric.

The target is 0.2, not 0.5, and the reason is stated plainly: challenging but still yielding enough successful trajectories for reinforcement learning to have signal. A task the model solves half the time is not at its frontier; a task it solves never produces no gradient at all.

And the multiplication matters more than any of the three terms. Under a sum, a proposer could farm novelty by generating exotic nonsense, or farm difficulty by generating unsolvable tasks. The product, with validity as a hard zero, means a proposed task has to be valid and frontier-difficulty and new, simultaneously, or it is worth nothing.

The task reward is a product, not a sum:

Rtask=V(q,s)⏟valid and verifiable?×D(q,s,{τi})⏟right difficulty?×N(q)⏟novel enough?R_{\text{task}} = \underbrace{V(q,s)}_{\text{valid and verifiable?}} \times \underbrace{D(q,s,\{\tau_i\})}_{\text{right difficulty?}} \times \underbrace{N(q)}_{\text{novel enough?}}

V checks that the generated task and scaffold form a well-defined learning environment: does the scaffold run, do high-confidence solutions pass, do clearly incorrect ones fail, does the evaluation match the specification. It is a hard gate — V = 0 zeroes the whole reward — which is what stops a malformed task from collecting reward simply by appearing difficult.

D estimates difficulty from the model's own rollouts: sample N attempts, compute the empirical success rate p, and reward tasks near a target frontier p* = 0.2 with a Gaussian. Twenty per cent, not fifty, and the reasoning is stated: challenging but still yielding enough successful trajectories for RL to have signal.

N subtracts the maximum similarity against a buffer of previously generated or trained-on tasks.

The self-curricular property falls out of D alone. As the model gets better and starts solving a task more reliably, p climbs past p* and the reward for proposing that task falls — so the generator is pushed toward harder problems without anyone maintaining a schedule. The curriculum evolves because difficulty is measured against the current model rather than against a fixed rubric.

The harness gets its own three-factor product:

Rharness=C(q,h)⏟task alignment×F(h,{τi})⏟reward fidelity×H(h)⏟hack resistanceR_{\text{harness}} = \underbrace{C(q,h)}_{\text{task alignment}} \times \underbrace{F(h,\{\tau_i\})}_{\text{reward fidelity}} \times \underbrace{H(h)}_{\text{hack resistance}}

H — resistance to evaluator failures, shortcuts and reward-hacking behaviours — is the term I would want to see ablated most, because it is the only thing standing between "the model designs its own grader" and the obvious failure mode.

What it moved

Both generations are reported side by side on the same suite at 397B and 35B, which makes the delta an unusually clean read on what extending the loop to task generation actually bought.

Ornith-1.0 → Ornith-1.5 · same benchmark suite · five runs averagedbiggest gain +48.0 points
·
the flagship · compared against GLM-5.2, DeepSeek-V4-Flash, Claude Opus 4.8 and Kimi K3
DeepSWE
8.056.0+48.0
Toolathlon-Verified
43.271.2+28.0
HLE (no tools)
30.244.6+14.4
SWE Atlas – QnA
41.255.6+14.4
NL2Repo
48.259.5+11.3
Frontier-Bench v0.1
2.713.5+10.8
HLE (with tools)
47.556.1+8.6
Terminal Bench 2.1
77.586.1+8.6
BrowseComp
79.786.6+6.9
WideSearch
75.280.8+5.6
GPQA Diamond
88.192.8+4.7
ClawEval
77.181.4+4.3
MCP-Atlas
76.480.0+3.6
SWE-bench Verified
82.486.0+3.6
SWE-bench Pro
62.265.1+2.9
SWE-bench Multilingual
78.979.6+0.7
codingreasoningagenticfaded bar = Ornith-1.0 · solid segment = what 1.5 added

The gains are wildly uneven, and the top and bottom of this list say different things. GPQA Diamond, a knowledge benchmark, moves 4.7 points. SWE-bench Multilingual moves 0.7. DeepSWE moves from 8 to 56, and at 35B from a flat zero to 22.

That is exactly the shape you would predict if a curriculum that proposes its own tasks and builds its own scaffolds mostly buys long-horizon agentic competence rather than knowledge. It is also exactly the shape you would predict if a training loop has learned the structure of these particular harnesses. The numbers cannot separate those two readings, and the one benchmark that would help — something the loop demonstrably never touched — is the one nobody publishes.

The gains are strikingly uneven. GPQA Diamond, a knowledge benchmark, moves 4.7 points. SWE-bench Multilingual moves 0.7. DeepSWE goes from 8 to 56 at 397B, and from a flat zero to 22 at 35B. Toolathlon-Verified goes from 43.2 to 71.2. Frontier-Bench from 2.7 to 13.5.

That pattern is exactly what you would predict if a self-generated curriculum mostly buys long-horizon agentic competence rather than knowledge — the tasks it proposes are agentic tasks, the scaffolds it builds are agent scaffolds, and knowledge was never the bottleneck.

It is also exactly what you would predict if a training loop has learned the structure of these particular harnesses. The published numbers cannot separate those readings, and I do not think that is a criticism unique to this release so much as a limitation of the whole genre.

Against the frontier

BenchmarkOrnith-1.5 (397B)Claude Opus 4.8GLM-5.2 (753B)DeepSeek-V4-Flash (284B)Kimi K3 (2.8T)
Terminal-Bench 2.1 (Terminus-2)86.185.081.082.788.3
Terminal-Bench 2.1 (Claude Code)85.278.982.781.8—
SWE-bench Verified86.085.883.081.686.2
SWE-bench Pro65.168.062.164.4—
SWE-bench Multilingual79.675.778.477.9—
DeepSWE56.059.046.254.467.5
HLE (no tools)44.649.840.535.043.5
GPQA Diamond92.893.691.291.493.5
Toolathlon-Verified71.276.248.270.373.2
BrowseComp86.684.385.684.891.2

The claim in the announcement is performance "comparable to Claude Opus 4.8", and on this evidence that is fair rather than inflated — it leads on three of the ten rows above and trails on five, mostly by a couple of points. Against open weights it is more decisive: it beats GLM-5.2, a model nearly twice its size, on nine of ten.

The gap to Kimi K3 on DeepSWE (56.0 against 67.5) and BrowseComp (86.6 against 91.2) is the honest counterweight, and Kimi K3 is a 2.8T model.

The 35B is the one to look at

The flagship result is the headline; the middle rung is the interesting engineering.

one training loop, three scales · every number averaged over five independent runsMIT · FP8, GGUF, MLX, NVFP4
against its own size class, and one model eleven times larger
Terminal-Bench 2.1
Ornith-1.5-35B-A3B
68.5
Qwen3.6-35B-A3B · same shape
49.2
Muse-Glimmer-30B · dense
51.7
Gemma-4-31B · dense
43.4
Qwen3.5-397B · 11× larger
48.6
SWE-bench Verified
Ornith-1.5-35B-A3B
79.0
Qwen3.6-35B-A3B · same shape
73.4
Muse-Glimmer-30B · dense
76.0
Gemma-4-31B · dense
52.0
Qwen3.5-397B · 11× larger
76.4
SWE Atlas – QnA
Ornith-1.5-35B-A3B
39.8
Qwen3.6-35B-A3B · same shape
15.5
Qwen3.5-397B · 11× larger
20.4
The rung that most rewards attention. Same parameter count and same active count as Qwen3.6-35B-A3B, and it leads every coding and agentic benchmark reported — several by more than fifteen points. It also beats Qwen3.5-397B on Terminal-Bench, SWE-bench Pro, NL2Repo and SWE Atlas QnA.

Select the 35B and look at the SWE Atlas QnA row. 39.8 against 15.5 for a model with the identical parameter count and identical active count — Qwen3.6-35B-A3B is not a weaker architecture, it is the same architecture without this training loop. And 39.8 against 20.4 for Qwen3.5-397B, a model eleven times larger.

That comparison is the one that carries the release’s actual argument, more than the flagship matching Claude Opus 4.8 does. A flagship result at 397B could be a scale result. A same-shape model doubling its peer on an agentic benchmark cannot be.

Against its direct architectural peer — Qwen3.6-35B-A3B, same parameter count, same active count — the 35B leads on every coding and agentic benchmark reported, several by more than fifteen points. That is not a scale result and it cannot be explained by a better base model, because it is the same shape.

At the bottom of the ladder, Ornith-1.5-9B reaches 47.0 on Terminal-Bench 2.1 through the Claude Code harness and 70.6 on SWE-bench Verified, with a quantized Mobile build that runs on a phone.

What I would want before believing it

The anti-hacking safeguards are on the evaluation, not the training. The eval protocol is genuinely careful — git history stripped from repo images so the model cannot read prior commits, network access disabled, GitHub and pip blocked for NL2Repo, five runs averaged. That is more rigour than most releases show. But every one of those safeguards protects the benchmark from the model. The loop that generates tasks and harnesses runs upstream of all of it, and H(h), the hack-resistance term, is defined in a sentence and never measured.

A self-generated curriculum has no held-out set by construction. The buffer B that novelty is scored against contains "previously generated or trained-on tasks". If the loop drifts toward the distribution of the benchmarks it is evaluated on — and every incentive in the training signal points that way — the novelty term will not notice, because novelty is measured against the model's own history, not against the eval suite. The DeepSWE jump from 8 to 56 is either a remarkable result or the clearest possible symptom, and nothing published distinguishes them.

σ is unpublished and it sets the whole curriculum. The width of the frontier band decides how much of the difficulty range earns reward at all. A tight σ means a narrow, aggressive curriculum; a loose one means the term barely binds. p* = 0.2 is given and justified; the parameter that determines what 0.2 actually excludes is not.

Ornith-1.0's DeepSWE score was 8, and 0 at 35B. A benchmark where the previous generation scored essentially nothing is the one where the new generation gains most. That could mean the loop unlocked a capability. It could also mean the 1.0 harness could not drive the benchmark at all — a formatting or protocol failure rather than a capability one — and 1.5 fixed the plumbing. Forty-eight points is a lot to attribute to a training loop without saying which.

Why it is still worth attention

Strip the leaderboard and the structural claim is that the three things an RL pipeline treats as fixed inputs are all learnable, and they can be learned against a single reward. Task curation is expensive and human. Harness design is expensive and human. Both are also, obviously, functions the model could approximate — and once they are inside the loop, the curriculum becomes a function of the policy instead of a constant beside it.

Whether that is sustainable or merely self-confirming is the question the field has to answer, and it will not be answered by benchmarks the loop can see. But shipping three scales of it under MIT, with quantized builds down to a phone and eval protocols written out in detail, is the version of the claim that other people can actually check.

Cite this article

For attribution, please use the following reference or BibTeX:

Satyajit Ghana, "Ornith-1.5: the model writes the exam, builds the marking scheme, then sits it", ai.thesatyajit.com, August 2026.

bibtex
@misc{ghana2026ornith15,
  author = {Satyajit Ghana},
  title  = {Ornith-1.5: the model writes the exam, builds the marking scheme, then sits it},
  url    = {https://ai.thesatyajit.com/articles/ornith-1-5},
  year   = {2026}
}
share