# Ornith-1.5: the model writes the exam, builds the marking scheme, then sits it
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/ornith-1-5
> date: 2026-08-23
> tags: open-weights, rl, agents, self-improvement, explainer
Most RL post-training pipelines have three fixed inputs: a set of human-curated tasks, a hand-designed harness that scores them, and a policy that learns against both. [Ornith-1.5](https://ornith.ai/ornith_1_5.html) makes all three learned, in the same loop, with one reward signal propagated across them.
The model proposes new tasks. For each task it generates or refines a scaffold — the instructions, tools, decomposition strategy and orchestration used to attack the problem. Conditioned on both, the policy produces solution rollouts. Reward from the rollout flows back through all three stages, so the system learns to produce better solutions *and* to generate more useful training tasks *and* to construct more effective scaffolds. All three are optimized with GRPO.
| | |
|---|---|
| Release | [Ornith-1.5](https://ornith.ai/ornith_1_5.html) · [HF collection](https://huggingface.co/collections/ornith-ai/ornith-15) · Aug 2026 |
| Models | **397B MoE**, **35B-A3B MoE**, **9B dense** — plus a quantized 9B-Mobile for phones |
| License | **MIT**, with FP8, GGUF, MLX and NVFP4 builds |
| Lineage | extends Ornith-1.0, itself built on Qwen3.5 and Gemma 4 with CPT, mid-training and post-training |
| Flagship | Terminal-Bench 2.1 **86.1** · SWE-bench Verified **86.0** · HLE **44.6** · DeepSWE **56.0** |
| Reference | Claude Opus 4.8 scores 85.0 and 85.8 on the first two |
| Protocol | every number averaged over **five independent runs**, with anti-hacking safeguards |
## The loop
Each training cycle is three stages. Given an environment or codebase, high-level instructions about the task type, and the model's own history of what it has already solved, the system proposes progressively harder tasks that go beyond that history — exposing capability gaps and pushing the training frontier outward.
Then for each task it builds a scaffold. Then the policy attempts it. Reward propagates back through all three.
The interesting engineering is in how those rewards are shaped, because a proposer that is rewarded for difficulty will happily generate nonsense.
The task reward is a **product**, not a sum:
$$
R_{\text{task}} = \underbrace{V(q,s)}_{\text{valid and verifiable?}} \times \underbrace{D(q,s,\{\tau_i\})}_{\text{right difficulty?}} \times \underbrace{N(q)}_{\text{novel enough?}}
$$
`V` checks that the generated task and scaffold form a well-defined learning environment: does the scaffold run, do high-confidence solutions pass, do clearly incorrect ones fail, does the evaluation match the specification. It is a hard gate — `V = 0` zeroes the whole reward — which is what stops a malformed task from collecting reward simply by appearing difficult.
`D` estimates difficulty from the model's *own* rollouts: sample `N` attempts, compute the empirical success rate `p`, and reward tasks near a target frontier `p* = 0.2` with a Gaussian. Twenty per cent, not fifty, and the reasoning is stated: challenging but still yielding enough successful trajectories for RL to have signal.
`N` subtracts the maximum similarity against a buffer of previously generated or trained-on tasks.
The self-curricular property falls out of `D` alone. As the model gets better and starts solving a task more reliably, `p` climbs past `p*` and the reward for proposing that task **falls** — so the generator is pushed toward harder problems without anyone maintaining a schedule. The curriculum evolves because difficulty is measured against the current model rather than against a fixed rubric.
The harness gets its own three-factor product:
$$
R_{\text{harness}} = \underbrace{C(q,h)}_{\text{task alignment}} \times \underbrace{F(h,\{\tau_i\})}_{\text{reward fidelity}} \times \underbrace{H(h)}_{\text{hack resistance}}
$$
`H` — resistance to evaluator failures, shortcuts and reward-hacking behaviours — is the term I would want to see ablated most, because it is the only thing standing between "the model designs its own grader" and the obvious failure mode.
## What it moved
Both generations are reported side by side on the same suite at 397B and 35B, which makes the delta an unusually clean read on what extending the loop to task generation actually bought.
The gains are strikingly uneven. GPQA Diamond, a knowledge benchmark, moves 4.7 points. SWE-bench Multilingual moves 0.7. **DeepSWE goes from 8 to 56** at 397B, and from a flat zero to 22 at 35B. Toolathlon-Verified goes from 43.2 to 71.2. Frontier-Bench from 2.7 to 13.5.
That pattern is exactly what you would predict if a self-generated curriculum mostly buys long-horizon agentic competence rather than knowledge — the tasks it proposes are agentic tasks, the scaffolds it builds are agent scaffolds, and knowledge was never the bottleneck.
It is also exactly what you would predict if a training loop has learned the structure of these particular harnesses. The published numbers cannot separate those readings, and I do not think that is a criticism unique to this release so much as a limitation of the whole genre.
## Against the frontier
| Benchmark | Ornith-1.5 (397B) | Claude Opus 4.8 | GLM-5.2 (753B) | DeepSeek-V4-Flash (284B) | Kimi K3 (2.8T) |
|---|---|---|---|---|---|
| Terminal-Bench 2.1 (Terminus-2) | **86.1** | 85.0 | 81.0 | 82.7 | 88.3 |
| Terminal-Bench 2.1 (Claude Code) | **85.2** | 78.9 | 82.7 | 81.8 | — |
| SWE-bench Verified | **86.0** | 85.8 | 83.0 | 81.6 | 86.2 |
| SWE-bench Pro | 65.1 | **68.0** | 62.1 | 64.4 | — |
| SWE-bench Multilingual | **79.6** | 75.7 | 78.4 | 77.9 | — |
| DeepSWE | 56.0 | 59.0 | 46.2 | 54.4 | **67.5** |
| HLE (no tools) | 44.6 | **49.8** | 40.5 | 35.0 | 43.5 |
| GPQA Diamond | 92.8 | **93.6** | 91.2 | 91.4 | 93.5 |
| Toolathlon-Verified | 71.2 | **76.2** | 48.2 | 70.3 | 73.2 |
| BrowseComp | 86.6 | 84.3 | 85.6 | 84.8 | **91.2** |
The claim in the announcement is performance "comparable to Claude Opus 4.8", and on this evidence that is fair rather than inflated — it leads on three of the ten rows above and trails on five, mostly by a couple of points. Against open weights it is more decisive: it beats GLM-5.2, a model nearly twice its size, on nine of ten.
The gap to Kimi K3 on DeepSWE (56.0 against 67.5) and BrowseComp (86.6 against 91.2) is the honest counterweight, and Kimi K3 is a 2.8T model.
## The 35B is the one to look at
The flagship result is the headline; the middle rung is the interesting engineering.
Against its direct architectural peer — Qwen3.6-35B-A3B, same parameter count, same active count — the 35B leads on every coding and agentic benchmark reported, several by more than fifteen points. That is not a scale result and it cannot be explained by a better base model, because it *is* the same shape.
At the bottom of the ladder, Ornith-1.5-9B reaches 47.0 on Terminal-Bench 2.1 through the Claude Code harness and 70.6 on SWE-bench Verified, with a quantized Mobile build that runs on a phone.
## What I would want before believing it
**The anti-hacking safeguards are on the evaluation, not the training.** The eval protocol is genuinely careful — git history stripped from repo images so the model cannot read prior commits, network access disabled, GitHub and pip blocked for NL2Repo, five runs averaged. That is more rigour than most releases show. But every one of those safeguards protects the *benchmark* from the model. The loop that generates tasks and harnesses runs upstream of all of it, and `H(h)`, the hack-resistance term, is defined in a sentence and never measured.
**A self-generated curriculum has no held-out set by construction.** The buffer `B` that novelty is scored against contains "previously generated or trained-on tasks". If the loop drifts toward the distribution of the benchmarks it is evaluated on — and every incentive in the training signal points that way — the novelty term will not notice, because novelty is measured against the model's own history, not against the eval suite. The DeepSWE jump from 8 to 56 is either a remarkable result or the clearest possible symptom, and nothing published distinguishes them.
**σ is unpublished and it sets the whole curriculum.** The width of the frontier band decides how much of the difficulty range earns reward at all. A tight σ means a narrow, aggressive curriculum; a loose one means the term barely binds. `p* = 0.2` is given and justified; the parameter that determines what 0.2 actually excludes is not.
**Ornith-1.0's DeepSWE score was 8, and 0 at 35B.** A benchmark where the previous generation scored essentially nothing is the one where the new generation gains most. That could mean the loop unlocked a capability. It could also mean the 1.0 harness could not drive the benchmark at all — a formatting or protocol failure rather than a capability one — and 1.5 fixed the plumbing. Forty-eight points is a lot to attribute to a training loop without saying which.
## Why it is still worth attention
Strip the leaderboard and the structural claim is that **the three things an RL pipeline treats as fixed inputs are all learnable, and they can be learned against a single reward**. Task curation is expensive and human. Harness design is expensive and human. Both are also, obviously, functions the model could approximate — and once they are inside the loop, the curriculum becomes a function of the policy instead of a constant beside it.
Whether that is sustainable or merely self-confirming is the question the field has to answer, and it will not be answered by benchmarks the loop can see. But shipping three scales of it under MIT, with quantized builds down to a phone and eval protocols written out in detail, is the version of the claim that other people can actually check.