Ornith-1.5: the model writes the exam, builds the marking scheme, then sits it

2026-08-23 · 16 min · reinforcement-learning · agents · self-improvement

Why read this

Explains Ornith's loop that rewards its own tasks, scaffolds and rollouts, reads the uneven generation gains, and names what would tell skill from harness fit.

Runs on a phone
A new technique
Explained from first principles

Training & RLMITPractitioner model

How this was scored

Is it new?: 2 of 3: A real new idea, method or capability
Can I trust it?: 1 of 3: Spot-checks a few numbers
Can I run it?: 2 of 3: Open code or weights with real limits
Will I understand it?: 2 of 3: Mechanism from first principles with figures
Can I act on it?: 1 of 3: General advice
Will it last?: 1 of 3: Relevant for months
Does it affect many?: 1 of 3: A specialist community
Only here?: 1 of 3: Some original analysis

Score 49 of 100, ranked 469 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Most RL post-training pipelines have three fixed inputs: a set of human-curated tasks, a hand-designed harness that scores them, and a policy that learns against both. Ornith-1.5 makes all three learned, in the same loop, with one reward signal propagated across them.

The model proposes new tasks. For each task it generates or refines a scaffold — the instructions, tools, decomposition strategy and orchestration used to attack the problem. Conditioned on both, the policy produces solution rollouts. Reward from the rollout flows back through all three stages, so the system learns to produce better solutions and to generate more useful training tasks and to construct more effective scaffolds. All three are optimized with GRPO.


Release	Ornith-1.5 · HF collection · Aug 2026
Models	397B MoE, 35B-A3B MoE, 9B dense — plus a quantized 9B-Mobile for phones
License	MIT, with FP8, GGUF, MLX and NVFP4 builds
Lineage	extends Ornith-1.0, itself built on Qwen3.5 and Gemma 4 with CPT, mid-training and post-training
Flagship	Terminal-Bench 2.1 86.1 · SWE-bench Verified 86.0 · HLE 44.6 · DeepSWE 56.0
Reference	Claude Opus 4.8 scores 85.0 and 85.8 on the first two
Protocol	every number averaged over five independent runs, with anti-hacking safeguards

The loop

Each training cycle is three stages. Given an environment or codebase, high-level instructions about the task type, and the model's own history of what it has already solved, the system proposes progressively harder tasks that go beyond that history — exposing capability gaps and pushing the training frontier outward.

Then for each task it builds a scaffold. Then the policy attempts it. Reward propagates back through all three.

task generation, scaffold construction and rollouts — all three optimized with GRPOstage 1: propose a task

R_task = V · D · N

V(q,s)is it valid and verifiable? — a hard gate: V = 0 zeroes everything

D(q,s,{τ})is it at the right difficulty? — a Gaussian on the model's own success rate, peaked at p★ = 0.2

N(q)is it sufficiently novel? — 1 minus the max similarity against the task buffer

Given an environment or codebase, high-level instructions about the task type, and the model's own history of what it has already solved, the system proposes progressively harder tasks that go beyond that history. Difficulty is estimated from the current policy's rollouts, so the curriculum moves as the policy improves.

Read the two arrows on the left as the actual claim. Reward from a rollout does not just update the policy — it updates the thing that proposed the task and the thing that graded it, so a task that produced useful learning signal makes the proposer more likely to produce tasks like it, and a harness that scored honestly makes the scaffold generator more likely to build harnesses like that.

Both rewards are products of three terms rather than sums, and in both cases the third term is the adversarial one. On the task side, novelty stops the proposer farming variations of a single good problem. On the harness side, hack resistance is the only thing standing between “the model designs its own grader” and the obvious failure mode — and it is the term defined in a sentence and never measured.

A flow diagram: the Ornith model, given an environment or codebase, emits a Task; the task branches into scaffold 1, 2 and 3; each scaffold branches into three solution rollouts; all nine rollouts feed a Reward node, and a GRPO 'RL update' arrow runs from the reward back to the model. — Ornith's own picture of the loop: one proposed task, several generated scaffolds, several rollouts under each, and a single reward that flows back through all three stages as one GRPO update (Ornith-1.5 release page, ornith.ai).

The interesting engineering is in how those rewards are shaped, because a proposer that is rewarded for difficulty will happily generate nonsense.

R_task = V(q,s) · D(q,s,{τ}) · N(q)reward 0.850

success rate p20%

σ (unpublished)0.15

novelty N0.85

V 1.00×D 1.000×N 0.85=0.850near the frontier

Drag p from 0.2 rightwards and watch the reward collapse. That is the self-curricular mechanism in one motion: as the policy gets better at a task, the reward for proposing that task falls, so the generator is pushed toward harder problems without anyone writing a schedule. The curriculum evolves because difficulty is measured against the current model’s own rollouts rather than against a fixed rubric.

The target is 0.2, not 0.5, and the reason is stated plainly: challenging but still yielding enough successful trajectories for reinforcement learning to have signal. A task the model solves half the time is not at its frontier; a task it solves never produces no gradient at all.

And the multiplication matters more than any of the three terms. Under a sum, a proposer could farm novelty by generating exotic nonsense, or farm difficulty by generating unsolvable tasks. The product, with validity as a hard zero, means a proposed task has to be valid and frontier-difficulty and new, simultaneously, or it is worth nothing.

The task reward is a product, not a sum:

R_{\text{task}} = \underbrace{V(q,s)}_{\text{valid and verifiable?}} \times \underbrace{D(q,s,\{\tau_i\})}_{\text{right difficulty?}} \times \underbrace{N(q)}_{\text{novel enough?}}

V checks that the generated task and scaffold form a well-defined learning environment: does the scaffold run, do high-confidence solutions pass, do clearly incorrect ones fail, does the evaluation match the specification. It is a hard gate — V = 0 zeroes the whole reward — which is what stops a malformed task from collecting reward simply by appearing difficult.

D estimates difficulty from the model's own rollouts: sample N attempts, compute the empirical success rate p, and reward tasks near a target frontier p* = 0.2 with a Gaussian. Twenty per cent, not fifty, and the reasoning is stated: challenging but still yielding enough successful trajectories for RL to have signal.

N subtracts the maximum similarity against a buffer of previously generated or trained-on tasks.

The self-curricular property falls out of D alone. As the model gets better and starts solving a task more reliably, p climbs past p* and the reward for proposing that task falls — so the generator is pushed toward harder problems without anyone maintaining a schedule. The curriculum evolves because difficulty is measured against the current model rather than against a fixed rubric.

The harness gets its own three-factor product:

R_{\text{harness}} = \underbrace{C(q,h)}_{\text{task alignment}} \times \underbrace{F(h,\{\tau_i\})}_{\text{reward fidelity}} \times \underbrace{H(h)}_{\text{hack resistance}}

H — resistance to evaluator failures, shortcuts and reward-hacking behaviours — is the term I would want to see ablated most, because it is the only thing standing between "the model designs its own grader" and the obvious failure mode.

What it moved

Both generations are reported side by side on the same suite at 397B and 35B, which makes the delta an unusually clean read on what extending the loop to task generation actually bought.

Ornith-1.0 → Ornith-1.5 · same benchmark suite · five runs averagedbiggest gain +48.0 points

the flagship · compared against GLM-5.2, DeepSeek-V4-Flash, Claude Opus 4.8 and Kimi K3

DeepSWE

8.056.0+48.0

Toolathlon-Verified

43.271.2+28.0

HLE (no tools)

30.244.6+14.4

SWE Atlas – QnA

41.255.6+14.4

NL2Repo

48.259.5+11.3

Frontier-Bench v0.1

2.713.5+10.8

HLE (with tools)

47.556.1+8.6

Terminal Bench 2.1

77.586.1+8.6

BrowseComp

79.786.6+6.9

WideSearch

75.280.8+5.6

GPQA Diamond

88.192.8+4.7

ClawEval

77.181.4+4.3

MCP-Atlas

76.480.0+3.6

SWE-bench Verified

82.486.0+3.6

SWE-bench Pro

62.265.1+2.9

SWE-bench Multilingual

78.979.6+0.7

codingreasoningagenticfaded bar = Ornith-1.0 · solid segment = what 1.5 added

The gains are wildly uneven, and the top and bottom of this list say different things. GPQA Diamond, a knowledge benchmark, moves 4.7 points. SWE-bench Multilingual moves 0.7. DeepSWE moves from 8 to 56, and at 35B from a flat zero to 22.

That is exactly the shape you would predict if a curriculum that proposes its own tasks and builds its own scaffolds mostly buys long-horizon agentic competence rather than knowledge. It is also exactly the shape you would predict if a training loop has learned the structure of these particular harnesses. The numbers cannot separate those two readings, and the one benchmark that would help — something the loop demonstrably never touched — is the one nobody publishes.

The gains are strikingly uneven. GPQA Diamond, a knowledge benchmark, moves 4.7 points. SWE-bench Multilingual moves 0.7. DeepSWE goes from 8 to 56 at 397B, and from a flat zero to 22 at 35B. Toolathlon-Verified goes from 43.2 to 71.2. Frontier-Bench from 2.7 to 13.5.

A twelve-panel grouped bar chart titled 'LLM Performance Evaluation'. Each panel compares Ornith-1.5 (397B), Ornith-1.0 (397B), DeepSeek-V4-Flash-0731 (284B), GLM-5.2 (753B) and Claude Opus 4.8 on one benchmark: TB2.1, SWE-bench Verified, DeepSWE, Frontier-Bench v0.1, NL2Repo, SWE Atlas QnA, HLE, GPQA Diamond, MCP-Atlas, Toolathlon-Verified, WideSearch and BrowseComp. — The 397B flagship against its own previous generation and the frontier. The two orange bars in each panel are Ornith-1.5 and Ornith-1.0, which is where the uneven deltas are legible: DeepSWE 8 to 56 and Toolathlon 43.2 to 71.2 next to GPQA Diamond 88.1 to 92.8 (Ornith-1.5 release page, ornith.ai).

That pattern is exactly what you would predict if a self-generated curriculum mostly buys long-horizon agentic competence rather than knowledge — the tasks it proposes are agentic tasks, the scaffolds it builds are agent scaffolds, and knowledge was never the bottleneck.

It is also exactly what you would predict if a training loop has learned the structure of these particular harnesses. The published numbers cannot separate those readings, and I do not think that is a criticism unique to this release so much as a limitation of the whole genre.

Against the frontier

Benchmark	Ornith-1.5 (397B)	Claude Opus 4.8	GLM-5.2 (753B)	DeepSeek-V4-Flash (284B)	Kimi K3 (2.8T)
Terminal-Bench 2.1 (Terminus-2)	86.1	85.0	81.0	82.7	88.3
Terminal-Bench 2.1 (Claude Code)	85.2	78.9	82.7	81.8	—
SWE-bench Verified	86.0	85.8	83.0	81.6	86.2
SWE-bench Pro	65.1	68.0	62.1	64.4	—
SWE-bench Multilingual	79.6	75.7	78.4	77.9	—
DeepSWE	56.0	59.0	46.2	54.4	67.5
HLE (no tools)	44.6	49.8	40.5	35.0	43.5
GPQA Diamond	92.8	93.6	91.2	91.4	93.5
Toolathlon-Verified	71.2	76.2	48.2	70.3	73.2
BrowseComp	86.6	84.3	85.6	84.8	91.2

The claim in the announcement is performance "comparable to Claude Opus 4.8", and on this evidence that is fair rather than inflated — it leads on three of the ten rows above and trails on five, mostly by a couple of points. Against open weights it is more decisive: it beats GLM-5.2, a model nearly twice its size, on nine of ten.

The gap to Kimi K3 on DeepSWE (56.0 against 67.5) and BrowseComp (86.6 against 91.2) is the honest counterweight, and Kimi K3 is a 2.8T model.

The 35B is the one to look at

The flagship result is the headline; the middle rung is the interesting engineering.

one training loop, three scales · every number averaged over five independent runsMIT · FP8, GGUF, MLX, NVFP4

against its own size class, and one model eleven times larger

Terminal-Bench 2.1

Ornith-1.5-35B-A3B

68.5

Qwen3.6-35B-A3B · same shape

49.2

Muse-Glimmer-30B · dense

51.7

Gemma-4-31B · dense

43.4

Qwen3.5-397B · 11× larger

48.6

SWE-bench Verified

Ornith-1.5-35B-A3B

79.0

Qwen3.6-35B-A3B · same shape

73.4

Muse-Glimmer-30B · dense

76.0

Gemma-4-31B · dense

52.0

Qwen3.5-397B · 11× larger

76.4

SWE Atlas – QnA

Ornith-1.5-35B-A3B

39.8

Qwen3.6-35B-A3B · same shape

15.5

Qwen3.5-397B · 11× larger

20.4

The rung that most rewards attention. Same parameter count and same active count as Qwen3.6-35B-A3B, and it leads every coding and agentic benchmark reported — several by more than fifteen points. It also beats Qwen3.5-397B on Terminal-Bench, SWE-bench Pro, NL2Repo and SWE Atlas QnA.

Select the 35B and look at the SWE Atlas QnA row. 39.8 against 15.5 for a model with the identical parameter count and identical active count — Qwen3.6-35B-A3B is not a weaker architecture, it is the same architecture without this training loop. And 39.8 against 20.4 for Qwen3.5-397B, a model eleven times larger.

That comparison is the one that carries the release’s actual argument, more than the flagship matching Claude Opus 4.8 does. A flagship result at 397B could be a scale result. A same-shape model doubling its peer on an agentic benchmark cannot be.

A twelve-panel grouped bar chart comparing Ornith-1.5-35B-A3B, Ornith-1.0-35B-A3B, Qwen3.6-35B-A3B, Muse-Glimmer-30B (dense), Gemma-4-31B (dense) and, past a dashed divider, Qwen3.5-397B, across the same twelve benchmarks. — The middle rung, with its architectural peer in the same frame: Ornith-1.5-35B-A3B leads Qwen3.6-35B-A3B on every panel reported — 68.5 against 52.5 on TB2.1, 46.2 against 29.4 on NL2Repo, 22 against 0 on DeepSWE — and the dashed column on the right is Qwen3.5-397B, a model an order of magnitude larger (Ornith-1.5 release page, ornith.ai).

Against its direct architectural peer — Qwen3.6-35B-A3B, same parameter count, same active count — the 35B leads on every coding and agentic benchmark reported, several by more than fifteen points. That is not a scale result and it cannot be explained by a better base model, because it is the same shape.

At the bottom of the ladder, Ornith-1.5-9B reaches 47.0 on Terminal-Bench 2.1 through the Claude Code harness and 70.6 on SWE-bench Verified, with a quantized Mobile build that runs on a phone.

What I would want before believing it

The anti-hacking safeguards are on the evaluation, not the training. The eval protocol is genuinely careful — git history stripped from repo images so the model cannot read prior commits, network access disabled, GitHub and pip blocked for NL2Repo, five runs averaged. That is more rigour than most releases show. But every one of those safeguards protects the benchmark from the model. The loop that generates tasks and harnesses runs upstream of all of it, and H(h), the hack-resistance term, is defined in a sentence and never measured.

A self-generated curriculum has no held-out set by construction. The buffer B that novelty is scored against contains "previously generated or trained-on tasks". If the loop drifts toward the distribution of the benchmarks it is evaluated on — and every incentive in the training signal points that way — the novelty term will not notice, because novelty is measured against the model's own history, not against the eval suite. The DeepSWE jump from 8 to 56 is either a remarkable result or the clearest possible symptom, and nothing published distinguishes them.

σ is unpublished and it sets the whole curriculum. The width of the frontier band decides how much of the difficulty range earns reward at all. A tight σ means a narrow, aggressive curriculum; a loose one means the term barely binds. p* = 0.2 is given and justified; the parameter that determines what 0.2 actually excludes is not.

Ornith-1.0's DeepSWE score was 8, and 0 at 35B. A benchmark where the previous generation scored essentially nothing is the one where the new generation gains most. That could mean the loop unlocked a capability. It could also mean the 1.0 harness could not drive the benchmark at all — a formatting or protocol failure rather than a capability one — and 1.5 fixed the plumbing. Forty-eight points is a lot to attribute to a training loop without saying which.

Why it is still worth attention

Strip the leaderboard and the structural claim is that the three things an RL pipeline treats as fixed inputs are all learnable, and they can be learned against a single reward. Task curation is expensive and human. Harness design is expensive and human. Both are also, obviously, functions the model could approximate — and once they are inside the loop, the curriculum becomes a function of the policy instead of a constant beside it.

Whether that is sustainable or merely self-confirming is the question the field has to answer, and it will not be answered by benchmarks the loop can see. But shipping three scales of it under MIT, with quantized builds down to a phone and eval protocols written out in detail, is the version of the claim that other people can actually check.

Update — 2 October 2026

Ornith shipped draft models for speculative decoding: the ornith-ai/ornith-15-dflash collection, with a DFlash draft for each of the three scales — 9B, 35B-A3B and 397B (all created 20 September — verified). The mechanism is standard speculative decoding with the twist in the draft: DFlash is a lightweight block-diffusion model that proposes a block of tokens in parallel, and Ornith-1.5 then verifies the whole block in a single forward pass instead of decoding one token at a time, keeping only the prefix that matches what it would itself have produced. The output distribution is preserved by construction, so the win is wall-clock, not quality. The card wires it in at num_speculative_tokens: 8 (verified).

The reported figure is "up to 2.54× without any loss in generation quality." Both halves are the publisher's own, not re-run here: the "no loss" is the formal guarantee of rejection-sampling verification, which holds whatever the draft does; the 2.54× is an acceptance-rate measurement and, as with any speculative setup, rides on how predictable the text is and how cheaply the draft runs beside the target.

20 October: the 122B

On 20 October Ornith announced a fourth size, Ornith-1.5-122B, with FP8, NVFP4, GGUF and MLX builds the same day. The release blog was not updated and still lists three scales, so the model card and the post are the only primary sources.

ornith-ai/Ornith-1.5-122B@7681ef6 · snapshot 2026-10-23

announced

122B MoE

measured

125,086,497,008

parameters

125.09B

repo size

250.19 GB

architecture: Qwen3_5MoeForConditionalGeneration
task: text-generation
library: transformers
license: mit
safetensors: 62 shards
largest file: 5.00 GB
files: 77
downloads: 4.8K
likes: 142

parameters by dtype

BF16125.09B

Qwen3.5 MoE shape: 48 layers (12 full attention, 36 linear), 256 experts with 8 routed plus 1 shared, 27-layer vision tower. Licence MIT per metadata; no LICENSE file in the repo.

repo last modified 2026-10-21

What it is. Read together, the 62 safetensors headers hold 125,086,497,008 parameters, all BF16 (measured). Without the 0.45B vision tower and the 2.52B multi-token-prediction head, the language model is 122.11B, so the name holds (measured). Each token runs 8 of 256 routed experts plus a shared one in each of 48 layers: 3.62B of routed weight, 4.13B of attention, 0.45B of shared expert, 0.04B of routers and a 0.76B output head. That makes 9.01B active, or 9.77B with the embedding row (reasoned from the measured shapes). It is an A10B model in all but name.

Against its siblings. The three MoE sizes share one Qwen3_5MoeForConditionalGeneration config: the same vocabulary, vision tower, 262,144-token window, and the same hybrid stack in which every fourth layer is full attention with 2 KV heads and the rest are linear attention (see mixture of experts). The diff is width and experts (measured from the configs and headers; active counts reasoned):

	35B-A3B	122B	397B
Layers (full / linear)	40 (10 / 30)	48 (12 / 36)	60 (15 / 45)
Hidden size	2048	3072	4096
Routed experts, active	256, 8	256, 8	512, 10
Expert width	512	1024	1024
Checkpoint total	35.95B	125.09B	403.40B
Active per token	3.45B	9.77B	17.35B

The 122B's config is also, key for key, that of Qwen3.5-122B-A10B, and both checkpoints hold exactly 125,086,497,008 parameters (measured). Asked in a reply whether it is built on Qwen3.5, Ornith answered "yes!" (reported). Like the other two MoE sizes, it is Ornith's loop applied to an existing Qwen3.5 shape, not a new architecture.

Benchmarks (all reported, from the three cards):

	35B-A3B	122B	397B
SWE-bench Verified	79.0	84.5	86.0
SWE-bench Pro	59.6	76.9	65.1
SWE-bench Multilingual	71.4	76.4	79.6
NL2Repo	46.2	51.2	59.5
HLE (with tools)	33.4	48.0	56.1
HLE (no tools)	25.6	37.68	44.6
MCP-Atlas	70.2	74.6	80.0
Toolathlon-Verified	48.7	70.8	71.2
WideSearch	67.8	75.2	80.8
BrowseComp	67.6	78.5	86.6
ClawEval	72.5	79.4	81.4

A nine-panel bar chart titled 'Ornith 1.5 122B Performance Evaluation' comparing Ornith-1.5 (122B), Qwen3.8-Flash-Next (125B), Qwen3.8 (27B dense), Nemotron 3.5 Super VL (120B), DeepSeek-V4-Flash-0731 (284B) and Beam (501B) on TerminalBench 2.1, CyberGym, NL2Repo, SWE-bench Pro (no-hack), HLE, WideSearch, BrowseComp, MCP-Atlas and Toolathlon-Verified. — The 122B's own comparison. Note the comparator set: no Ornith-1.0-122B and no Claude Opus 4.8, so neither the generation delta nor the frontier comparison this article reads at the other sizes can be made here. The SWE-bench Pro (no-hack) panel, 76.9, is the number that does not fit its siblings (Ornith-1.5-122B model card).

On ten of those eleven rows the 122B lands between its siblings, as a middle size should, and on Toolathlon-Verified it is within 0.4 of the flagship. SWE-bench Pro is the exception: 76.9 is 11.8 points above the 397B and 8.9 above the 68.0 the 397B card gives Claude Opus 4.8 (reasoned). The 122B card labels the row "no-hack" where the others say only "SWE-bench Pro", yet the anti-hacking footnote is word for word the same on all three. A smaller sibling beating its flagship by that much on the hardest SWE suite is either a different protocol or a result that needs explaining, and the card offers neither. Terminal-Bench 2.1 is 81.2 with no harness named (67.8 and 86.1 under Terminus-2 for the 35B and 397B). New rows: CyberGym 80.0 and SciCode 53.8. Gone: DeepSWE, Frontier-Bench and GPQA Diamond. Against DeepSeek-V4-Flash-0731, the one comparator this card shares with the 397B's, the 122B leads on CyberGym and Toolathlon-Verified, ties MCP-Atlas at 74.6, and trails on Terminal-Bench, NL2Repo, HLE with tools, WideSearch and BrowseComp.

The post and the card disagree. Six of the post's eight numbers match the card. Terminal-Bench is 0.813 in the post and 81.2 on the card. The post's HLE of 0.446 is not a 122B number at all: it is the 397B's HLE without tools, where the 122B card says 37.68.

The deployment claims. The post says it runs "on a single NVIDIA DGX card" and the card says "a single 80GB GPU". The BF16 weights alone are about 250 GB (reasoned). The FP8 repository's shards total 132.95 GB and the NVFP4 one's 77.63 GB (measured from the Hub's file listing), so only NVFP4 fits 80 GB, and barely. The hybrid stack helps with the rest: only 12 layers keep a cache, 24,576 bytes a token in BF16, so a full 262,144-token context costs 6.44 GB (reasoned). The card's own vLLM recipe points at the BF16 repository, which does not fit one 80 GB GPU. It also promises "a quantized Ornith-1.5-122B-Mobile variant" for phones; no such repository exists under ornith-ai, and the sentence reads as carried over from the 9B card. And it calls the 122B "the most capable" member, when the 397B beats it on every shared row but one.

Licence. MIT in the repository metadata, as for the other sizes. The metadata's licence link points at a LICENSE file that is not in the repository, which is also true of the 9B, 35B and 397B (measured).

What is new, then, is a size, not a method: the card repeats the blog's description of the loop and discloses nothing about how the 122B was trained. It keeps the vision tower, which Ornith confirmed in a reply (reported) and the headers show (measured).

How I checked. I read the post, its two follow-ups and the first page of replies through the fxtwitter mirror. I range-read the headers of all 62 shards of the 122B, and of every shard of the 35B and 397B, and summed tensor shapes by role: routed experts, shared expert, attention, router, embeddings, vision, multi-token prediction. Active counts take the routed experts at top-k over expert count. I diffed the three config.json files against each other and the 122B's against Qwen3.5-122B-A10B's, parsed the benchmark tables out of the four cards' HTML, summed the safetensors sizes in the FP8 and NVFP4 repositories, and requested each repository's LICENSE and the Mobile repository. I ran nothing: every score above is Ornith's.

The loop

What it moved

Against the frontier

The 35B is the one to look at

What I would want before believing it

Why it is still worth attention

Update — 2 October 2026

20 October: the 122B

Cite this article