# Ultramarine 1 Flash: RL on a board memo, graded by 31 sentences > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/ultramarine-1-flash > date: 2026-10-14 > tags: reinforcement-learning, post-training, agents, rl-environments, glm, benchmarks, synthetic-data, lora Paper Instruments released [Ultramarine 1 Flash](https://www.paperinstruments.com/blog/ultramarine) today. It is [GLM-5.3-Flash](/articles/glm-5-3-flash) after a long run of reinforcement learning on knowledge work: diligence memos, board pre-reads, credit-agreement reviews, warehouse tickets. The launch post on X makes four claims. It says the model performs similarly to Claude Fable 5. It says it is 60x cheaper. It says it was "post-trained by a team of 5 for ~\$300k all-in". And it calls the model open-source, with weights "releasing in 2 weeks". The interesting part is the training signal. Coding RL has unit tests. A board memo has none. Ultramarine's reward is an LLM judge reading a weighted list of propositions, and most of the blog's engineering goes into keeping that signal honest. So I start there, with the blog's worked example and the open-source grader it uses. The claims come after. Labels: **measured** means I computed it from a file, the released code or the blog's own data. **Reported** is Paper Instruments' number, not re-run. **Reasoned** is my arithmetic on the other two. | | | |---|---| | Base | `zai-org/GLM-5.3-Flash`, 320B total and 18B active, MIT ([site teardown](/articles/glm-5-3-flash)); the same base became [Tev-320B](/articles/tev-gym) this week, with verifiable per-step rewards instead of a judge | | Method | Agentic RL, rubric rewards from an LLM judge, CISPO loss, LoRA rank 64 (reported) | | Curriculum | 4 domains, 19 practice areas, 88 environment types, all synthetic (reported; 88 measured from Table 1) | | Harness | Feather: `read`, `edit`, `write`, `grep`, `glob`, `bash`, `todo_write` (reported, and in the released code) | | Context, turns | 131,072 tokens, up to 250 agent turns (reported) | | Out today | API, the Feather harness and a reference training recipe (MIT), the `rubric` grader (MIT) | | Not out | Weights, the task data, the snapshot builder, training length and compute | ## Why a memo is hard to reward Policy-gradient RL needs one number per rollout. The model makes several attempts at a task, each gets a reward, and the update pushes up the attempts that beat their siblings. That number has to rank attempts the way an expert would. For code the number is cheap and almost incorruptible: run the tests. For a board pre-read the only available grader is another model. That opens three failure modes. - **The judge has tastes the task does not.** Length, confident tone, headings and tables tend to read as quality. A policy optimised against a judge finds those tastes faster than it finds the substance. - **The criterion is too tight.** Write the reward from one golden answer and every valid alternative scores as wrong. The policy learns the phrasing of the reference rather than the job. - **The criterion is too loose.** "Provides a useful forecast update for the board" rewards anything that sounds like one, including the plausible-sounding wrong answer. The blog's first figure draws the last two as boundaries over six candidate answers.
Their rule is that a criterion tests "one coherent proposition", and names the entities, dates and thresholds needed to check it. Independent accomplishments get separate criteria so that partial work earns partial credit. A judge then marks each one met or unmet. That is the [Rubrics as Rewards](https://arxiv.org/abs/2507.17746) idea, pushed into long agentic tasks. ## How 31 sentences become one reward The blog publishes a complete training task. An agent gets a deal-room filesystem of 31 files (measured, from the listed tree), email and calendar MCP servers, and an instruction to update the forecast section of a board pre-read `.docx` in place. The grader is 31 weighted criteria in six groups. I summed them: positive weights total **+247**, negative weights total **−210** (measured). Eight of the 31 are negative: things a careless memo does, such as quoting the stale 11.7x multiple or telling management it may use the new numbers on the roadshow before counsel clears them. That one costs 50. The grader is [`PerCriterionOneShotGrader`](https://github.com/paper-instruments/rubric) from Paper Instruments' open `rubric` package, with a custom prompt they do not publish. I read the package. It asks the judge for a MET or UNMET verdict on every criterion in one call, then aggregates: $$ \text{raw} = \sum_{i} \mathbb{1}[\text{MET}_i]\, w_i, \qquad r = \operatorname{clip}\!\left(\frac{\text{raw}}{\sum_i \max(0, w_i)},\, 0,\, 1\right) $$ A negative criterion is phrased as the error, so MET means "the error is present" and its negative weight subtracts. The default prompt spends most of its length on that inversion, with examples such as "Avoid doing X because…" counting as not committing X. Load the presets and three properties fall out. 1. **Correct numbers are not enough.** The "right numbers, one track" memo gets every figure right and still scores 164 of 247, a reward of 0.664 (measured from the weights). It merges the board's approval with counsel's clearance, which costs 25, and so misses the 18-point criterion that rewards separating them. The reward is shaped around the judgement, not the arithmetic. 2. **The floor erases information.** The stale-basis memo has a raw score of −71, which clips to zero (measured). So does an empty file. Below the floor, the optimiser cannot tell a bad answer from a worse one. That is a defensible choice, because clipping stops one catastrophic criterion from dominating a group's variance. It is still information thrown away. 3. **Criteria overlap.** Criterion 14 ("v7 is the sole basis for Board, fairness, and transaction-valuation work", +10) and criterion 24 ("Uses Management Forecast v7 … as the forecast basis", +18) reward close to the same proposition. Criteria 11 and 16 both penalise presenting the stale v5 case. Getting the basis right is worth 28 points and getting it wrong costs 50 (reasoned, from the published rubric). The blog's own principle says independent accomplishments get separate criteria. These two pairs are not independent, so the rubric weights the forecast basis more heavily than the group totals suggest. One quirk in the package matters for training. If the judge's JSON leaves a criterion out, the aggregator records it as UNMET (measured, `per_criterion_one_shot_grader.py`). For a positive criterion, that is a lost point. For a negative one, it is a penalty that silently never applies. A judge that drops items under a long context lets errors through. I cannot tell whether Paper Instruments' custom prompt or their retry logic guards against this. The released code does not show it. The judge sees the deliverable "parsed and rendered as text". The post names neither the judge used in training nor its agreement with human experts; it points to a separate post, "Notes on Choosing a Rubric Judge", for the method. ## Where 88 kinds of tasks come from The blog rejects production traces, because a reward written from one user's request bakes in that user's unstated preferences ("top 10 medtech companies" by revenue, or by market cap?), and expert-written tasks, as expensive and narrow. It chose fully synthetic generation. The curriculum spans finance (34 environment types), legal (15), consulting (28) and operations (11). That is 88, and the table lists exactly 88 named environments (measured from Table 1). Each practice area is a generation pipeline that samples a taxonomy: deliverable format, sources of complexity, stakeholders, conflicting sources, the stage of the deal. The blog puts each area's configuration space at 10⁷ to 10¹⁰ combinations (reported). An environment is a filesystem of real SEC filings and synthetic contracts and emails, plus live systems the agent changes (CRM, ERP, warehouse management), and a private ground-truth record only the task-proposer agents see. The reward-hacking defence is mostly in how the tasks are made. Producing one task "involves several dozen independent agent loops and multiple rounds of review" (reported). Those rounds check grounding, realism, difficulty and whether the rubric matches the assignment. Classifiers screen for answer leakage and reward hacking, "iteratively added or refined in response to failures observed during training". A last filter drops tasks by base-model performance. Tasks the base model always solves or never solves give every rollout in a group the same reward, so their advantage is zero and they teach nothing. The blog does not say how many tasks were generated, how many survived the filters, or what any of the hacking classifiers caught. Those are the numbers that would say whether the defence worked. ## The harness is part of the reward Ultramarine trains inside its own harness, Feather. Paper Instruments surveyed twelve coding harnesses (Claude Code, Codex, OpenCode, OpenHands, Qwen Code and others) and built one on their shared conventions instead of training on several. Three choices bear directly on reward integrity. - **The agent loop runs outside the sandbox.** Commands execute in an isolated E2B sandbox, but the loop, the credentials and the grader do not. Grading material arrives only after the rollout. In the blog's words, "nothing the model does in the sandbox can alter its own harness, credentials, or grader". This is the structural answer to the failures documented in [MiMo's environments](/articles/mimo-env-reward-hacking), where the answer was sitting in `.git`. - **Environment failures are excluded, not zeroed.** A malformed tool call is the policy's fault, so the error goes back to the model and costs a penalty. A lost sandbox says nothing about the policy, so the attempt is dropped and restarted. Scoring it as zero would add noise to every group it lands in. - **The training prompt says almost nothing.** The training system prompt is three sentences. I counted 42 words in `system_prompt.md` in the released code (measured). Feather's own production prompt, in the same repo, is 3,944 words (measured). The blog argues that prescriptive instructions raise test-time scores but narrow exploration, so production instructions came in "only … in final stages of training". Feather's edit tools also fingerprint each file on read and refuse an edit if it changed since, and Office files are edited through Paper Instruments' MIT forks `paper-docx`, `paper-pptx` and `paper-xlsx`, which refuse operations they cannot do safely. A silently corrupted workbook would otherwise reach the judge as a quality failure. ## The optimiser The loss is a modified [CISPO](/articles/first-order-rl), from MiniMax-M1. For rollouts $i$ in a group $g$ on one task: $$ \mathcal{L}(\theta) = -\frac{1}{N}\sum_{i,t} m_{i,t}\,\operatorname{sg}\!\left[\min\!\left(\rho_{i,t}, 5\right)\right] A_i \log\pi_\theta(a_{i,t}\mid h_{i,t}), \qquad A_i = \tilde r_i - \frac{1}{|g|}\sum_{j\in g}\tilde r_j $$ Here $\rho_{i,t}$ is the probability ratio between the current policy and the policy that sampled the rollout. $m_{i,t}$ masks to model-generated tokens and $N$ counts them. $\operatorname{sg}$ is stop-gradient. Unlike PPO's clip, CISPO caps the importance weight and stops its gradient, so a token whose ratio runs high still receives a gradient. It just gets less weight. The released recipe defaults to the same cap: `clip_high_threshold: 5.0` (measured). The other pieces, all reported: - **Mean-centred advantage, no standard-deviation division.** Following Dr. GRPO, so a group whose rewards barely differ does not have its noise blown up to unit scale. - **Zero-advantage filtering**, from DAPO: groups where every rollout scored the same are dropped. - **Rollout Routing Replay (R3).** GLM-5.3-Flash is a mixture-of-experts model with 288 routed experts. If the inference engine and the trainer route a token to different experts, the importance ratio measures routing disagreement, not policy change. R3 replays the inference-time routes in training. - **Asynchronous, PipelineRL-style updates.** Finished groups stream into a buffer, and a full batch triggers exactly one optimiser step. There are no mini-steps and no separate "old policy" forward pass. Inference weights refresh mid-rollout. Their earlier synchronous setup with three policies is gone. - **LoRA rank 64, AdamW, learning rate 5×10⁻⁵, batch 64×8**, temperature 1, no top-p or top-k. A batch of 64 tasks with 8 rollouts each is 512 trajectories per step (reasoned). - **Shaping.** A capped penalty per invalid tool call, a turn factor that fades positive reward towards zero as turns approach the cap, and token penalties beyond an allowance. The token penalties apply only when the group's mean reward is positive. What the table does not give is the length of training: no step count, no GPU type, no GPU-hours. "Through prolonged training" in the thread is all there is. So the "~\$300k all-in" from the tweet, which is not in the blog, cannot be checked against anything (reasoned). ## Checking "similar performance to Fable 5" The blog's Table 8 has six knowledge-work rows and two coding rows "for reference". Every model ran at its highest reasoning setting.
Against Claude Fable 5, Ultramarine is ahead on four rows: JobBench (+5.5), PI Knowledge Work (+1.5), FrontierFinance (+4.1) and AutomationBench (+4.0). It is behind on four: DiligenceBench (−2.3), APEX-Agents 1.1 (−6.6), Terminal-Bench 2.1 (−1.1) and DeepSWE (−8.4). All measured from Table 8. Across the six knowledge-work rows the mean gap is +1.0 (reasoned). "Similar" is a fair word for that. "Fable-class", as the product page puts it, depends on which rows you read. Three caveats sit in the table's footnotes. - **Two of the six knowledge-work rows are Paper Instruments' own benchmarks.** PI Knowledge Work is internal and judged by GPT-5.6 Terra. DiligenceBench is "our own benchmark", and Paper Instruments ran every model on it. The tweet's headline comparison, 59.3 against 57.8, is on the internal suite. Ultramarine was trained on the same four domains, by the same people, with the same kind of rubric grading. On the other four knowledge-work rows it leads Fable on three, by a mean of +1.75 (reasoned). - **The comparators come from mixed sources.** On JobBench, FrontierFinance and APEX-Agents, Fable's score comes from a public leaderboard. On AutomationBench it is "the public-set scores reported by their developers". Ultramarine and some comparators were run by Paper Instruments in the official harnesses. Same benchmark, different operators. - **Fable's largest lead is bigger than any of Ultramarine's.** APEX-Agents 1.1, run in Mercor's leaderboard harness and scored Pass@1 over three runs, goes to Fable by 6.6 points. Ultramarine's largest lead is 5.5, on JobBench. Against its own base, the story is cleaner. Ultramarine improves on GLM-5.3-Flash on all six knowledge-work rows, by **4.1 points on average** (reported; my recomputation from Table 8 gives 4.08). PI Knowledge Work gains the most (+7.3), AutomationBench the least (+1.4). Both coding rows fall slightly: Terminal-Bench 2.1 from 84.3 to 83.2, DeepSWE from 63.4 to 61.5 (measured from Table 8). The base-model figures for Terminal-Bench, DeepSWE and AutomationBench (84.3, 63.4, 48.8) match the numbers in the GLM-5.3-Flash card as [this site's teardown](/articles/glm-5-3-flash) recorded them (measured). Targeted RL moved the domains it targeted and cost a little elsewhere. That is the honest result, and the blog's table shows it rather than hiding it. The blog also reports 6.5% fewer output tokens than the base across the six benchmarks: 30.5M against 32.7M. Those two rounded totals give 6.7% (reasoned), which is consistent within rounding. ## Checking "60x cheaper" The cost claim comes from one chart. It plots PI Knowledge Work score against cost per task at each developer's official API price as of October 2026.
\$8.19 for Fable 5 at max reasoning divided by \$0.13 for Ultramarine is **63x** (reasoned). The "60x" is that, rounded down. Four things qualify it. - **It is a price ratio, set partly by the seller.** The Ultramarine API charges \$0.15 per million input tokens, \$0.03 cached and \$0.50 output, "the same as GLM-5.3 Flash" (reported, product page). The ratio measures an 18B-active open MoE priced like one against a closed frontier model at its most expensive setting. GLM-5.3-Flash itself costs \$0.10 a task on the same chart, which is 82x below Fable 5 at max (reasoned). - **Ultramarine costs more per task than its base.** \$0.13 against \$0.10 is 30% more (reasoned), at the same token prices. So on this suite Ultramarine used more tokens per task than GLM-5.3-Flash, even though it used 6.5% fewer across all six benchmarks. That is likely the per-task extra work it learned to do. The blog does not break it down. - **Max is not Fable 5's best value point.** On this suite Fable 5 scores 50.8 at low reasoning (\$1.82) and 48.6 at medium (\$2.85). So 57.8 at \$8.19 is the only Fable 5 setting that outscores the base model. And Claude Fable 5.1 at max scores 65.5 for \$8.64, 6.2 points above Ultramarine (reported, chart labels). - **The nearest rival is cheap and closed.** GPT-6 Luna at max reasoning scores 58.3 for \$0.11 (reported), within a point of Ultramarine and a little cheaper. The Pareto claim does hold on the published points. Of the 39 other runs, 27 cost more and score lower (reported, and measured from the chart labels). No run is both cheaper and better (measured). Ultramarine is on the frontier of this chart. The chart is Paper Instruments' benchmark, judged by a model they chose. The blog's one external anchor is a correlation of r = 0.81 between PI Knowledge Work and the Artificial Analysis Intelligence Index, over the 24 runs both measured (reported). That supports the suite as a general-capability ranking. It says nothing about whether a model trained on the suite's own domains is advantaged on it. ## What "open-source" means today I checked Hugging Face (`api/models?search=ultramarine`, and the obvious author names) and found no weights (measured, 2026-10-14). The thread says weights will be linked "within 2 weeks". What is public: - **Feather** (`paper-instruments/feather`, MIT, 2,822 tracked files at commit `d864944`, 2026-10-13; measured). It holds the harness, a Harbor evaluation package and `feather-train`. That last one covers rollouts and grading on E2B snapshots, plus "a reference Tinker training integration through Verifiers". This is the "training code" the thread promises, and it is already out. It is a reference recipe rather than the run itself. The example config trains `Qwen/Qwen3.6-27B` at learning rate 1×10⁻⁵ for two steps, where the blog used 5×10⁻⁵ on GLM-5.3-Flash. Its documented length penalty is a simpler form than the blog's equation, with no turn-fade or extreme-length term (measured from `grading.md` and `examples/tinker.json`). It also "does not provide a snapshot-baking command", and the task graders are baked into those snapshots. - **`rubric`** (MIT). The licence file's copyright line reads "The LLM Data Company", and the README links that organisation's repository (measured). It is a small package: 31 tracked files, three graders. - **`paper-docx`, `paper-pptx`, `paper-xlsx`.** MIT forks of python-docx, python-pptx and openpyxl (measured from their licence files). Not public: the weights, the 88 environment pipelines, the tasks and rubrics beyond the one worked example, the custom judge prompt, and the training logs. Without the tasks, the recipe cannot be reproduced, only re-implemented. "Team of 5" and "~\$300k" appear only in the tweet. They are reported, unverifiable, and absent from the technical post. ## What I take from it The model is a careful but unremarkable result: a strong open MoE, RL-tuned toward one domain, gains about four points there and gives up one or two on code. The recipe is the contribution. It treats non-verifiable RL as data engineering: the reward is only as good as the boundary each criterion draws, so the effort goes into adversarial criteria, task filtering and walling the grader off from the policy, not into a stronger judge. What the post does not yet show is evidence that the defences held: reward-versus-human-agreement curves over training, the hacking classifiers' catch rate, any held-out grader. The tweet's claims to check are mostly about the bill and the comparison. The ones I would want the tech report to answer are about the reward. Other GLM-5.3-Flash derivatives this site has measured: the [MLX port](/articles/glm-5-3-flash-mlx) and the [abliterated FP8 variant](/articles/glm-5-3-flash-uncensored). For the verifiable side of the problem, see [Scaling agentic RL](/articles/scaling-agentic-rl) and [MiMo's graded environments](/articles/mimo-rl-environments); for the road not taken here, [multi-harness RL](/articles/multi-harness-rl).