2026-07-24 · 32 min · reinforcement-learning · agents · environments · infrastructure · prime-intellect · huggingface · reward-design · explainer
The reinforcement-learning recipe for agents is, by now, boring in the good way: put the agent in an environment, let it act, check whether it succeeded, and reward it for succeeding. The hard part was never the algorithm. It is the phrase "check whether it succeeded." At scale you need hundreds of thousands of tasks that each come with a sandbox the agent can act in and a grader that produces a clean, reproducible reward — and the open-source ecosystem, for all its great datasets, does not ship that. Every SWE benchmark, every terminal corpus, every search eval invents its own harness, its own image conventions, its own grading scripts, and its own failure modes. They do not compose.
Prime Intellect's post (Daniel Auras and team, July 2026) is an engineering answer to exactly that: they took 23 agentic tasksets across three domains and put them behind one taskset API — roughly 365,000 tasks (~198,000 software-engineering across 20+ languages, ~28,600 terminal, ~137,600 search), each with a prebuilt sandbox image, each grader withheld until scoring, many re-uploaded only after gold-validation. This piece walks the thesis, the design, the catalog, and the caveats.
The bottleneck is the environment, not the algorithm
Here is the shape of the problem the post is solving. Each upstream taskset made reasonable choices for its own harness, and those choices don't compose: SWE-bench applies test patches inside a generated eval script; R2E-Gym bakes tests into the image and compares against expected outputs; every search benchmark invents its own judge. If you want to train one agent across all of them, you have to normalize those lifecycles without breaking each taskset's own scoring semantics — because the scoring semantics are the whole point. A reward you can't trust is worse than no reward.
Prime Intellect frames it in one line: "A taskset row is only useful if it can produce a clean reward signal." And a "surprising fraction" of open agentic data fails that precondition — broken images, network-dependent tests, expected outputs that drifted, and tasks that score as solved without touching the code at all. Scaling RL, in this telling, is far less about the loss function (see Ring-Zero and frontier RL economics for the loss/systems side) and far more about manufacturing verified, reproducible environments in bulk.
One environment, three layers
The enabling idea is verifiers v1, which decomposes an environment into three independent layers:
- Taskset — the data and its scoring logic (what problem, what counts as solved). This post is the taskset layer.
- Harness — how the agent is driven (Codex, their own harness, or yours).
- Runtime — where it executes (a Prime sandbox, local Docker, …).
Because those layers are independent, one command can run any taskset in any harness on any runtime:
# ScaleSWE, in the Codex harness, on Prime Sandboxes:
uv run eval scaleswe-v1 --harness.id codex --harness.runtime.type prime -n 3The taskset-layer packaging format is called Harbor: SWE-bench Verified "runs the Harbor Hub packaging against the official instance images," and Terminal-Bench 2 is wrapped "through the same Harbor taskset, so the eval suite and the training corpora share one task format and one scoring contract." That last clause is the whole design in miniature — the thing you evaluate on and the thing you train on speak the same format and the same scoring contract.
A rollout, end to end
Concretely, every taskset exposes the same handful of hooks, and a rollout walks them in the same order. The one beat worth internalizing is the integrity move: during a rollout the agent lives inside the same sandbox as the grading machinery, so "anything readable in the container is fair game for a reward hack." So the grading material — the test patch, the expected outputs, the grader — is withheld until scoring, then restored only to compute the reward. Step through it:
The harness runs the agent inside the sandbox. The test patch and grader stay withheld — anything readable in the container is fair game for a reward hack.
The contract that makes this uniform is small — a typed data schema plus four hooks:
import verifiers.v1 as vf
class MyTaskData(vf.TaskData):
base_commit: str # state the sandbox resets to
test_patch: str # grading material — withheld until scoring
gold_patch: str # reference fix — used only by `validate`
class MyTask(vf.Task[MyTaskData]):
async def setup(self, runtime): ... # prepare the repo in the task's image
async def finalize(self, trace, runtime): ... # capture the agent's diff into the trace
@vf.reward
async def solved(self, runtime) -> float: # restore tests, apply test_patch,
... # run the taskset's own upstream grader
async def validate(self, runtime) -> bool: # gold patch must score 1.0;
... # the no-op (setup-only) run must notSome upstream authors deliberately ship the tests readable — R2E-Gym keeps its grading tests at /r2e_tests, Multi-SWE leaves grading scripts and test.patch under /home. That is fine for an attach-at-eval harness, but not for a live RL sandbox under optimization pressure, so Prime Intellect's integrations hide those artifacts and restore them only for scoring.
Four decisions that make tasksets compose
The integrations keep every taskset's original grading path — upstream log parsers, upstream report generation, upstream test commands — and normalize everything around it. Four decisions do the work:
- One API. Every taskset loads from a typed config (dataset, split, filters), provisions a sandbox from the task's image, and scores with the taskset's own logic. Swapping splits or adding a
filter_fnworks the same everywhere. - One image registry. Task images live in Prime's own registry, co-located with the sandboxes — ~135,000 prebuilt open-source task images, which they claim is the largest such catalog hosted by any sandbox provider. The point is operational: no Docker Hub rate limits "when running a thousand concurrent rollouts," and reproducibility from a pinned per-task image rather than a build step that can drift.
- One integrity standard. Grading material withheld until scoring, as above — the reward-hack defense.
- One validation bar. Before a dataset earns a default slot, it runs gold-patch and no-op validation, and a cleaned version is re-uploaded with exclusions preserved. That is the next section.
The catalog: 23 tasksets, three domains
Pick a domain to see its tasksets and their (shipped) task counts. The three domains are lopsided — software engineering and search dominate the raw count; terminal is smaller but denser in verified benchmarks.
Every taskset loads through the same contract — a typed config, a sandbox from the task's own prebuilt image, and the taskset's own upstream grader — so one agent can train across all three domains without a bespoke harness per dataset.
The domain split of the ~365,000 total:
Software engineering
Real repositories, real diffs; the reward is whether hidden tests pass after the agent's patch. Counts are the displayed shipped totals; parenthetical notes flag the gold-validated re-upload sizes where they differ.
| Taskset | Tasks | What it is |
|---|---|---|
| SWE-bench Verified | 500 | Human-filtered GitHub issues in major Python repos; the canonical benchmark |
| SWE-bench Multilingual | 300 | The canonical set across C, C++, Go, Java, JS/TS, PHP, Ruby, Rust |
| SWE-bench Pro | 731 | Harder successor; large-scale diffs from license-friendly repos |
| SWE-smith | 83,519 | Bugs injected into healthy repos, keeping the tests that catch them (8 languages) |
| R2E-Gym | 4,578 | Executable envs from real commits with synthesized issues (4,522 gold-validated) |
| Multi-SWE | 6,835 | Containerized RL + eval instances across 7 languages (2,232 in the validated RL set) |
| SWE-rebench-V2 | 32,079 | Continuously mined fresh PRs, 20 languages, decontaminated by recency (6,275 verified) |
| Scale-SWE | 17,202 | Python tasks with test patches applied just before eval (from 20,181 raw) |
| SWE-Lego | 15,903 | SWE-bench-style training data at scale; tests applied only at scoring |
| OpenSWE | 36,884 | Tasks paired with per-task eval scripts kept out of the sandbox until scoring |
| Senior SWE-Bench | 50 | Investigation/design tasks from 12 production repos; pytest/vitest + optional LLM rubric |
Terminal
Give the agent a shell and a goal; a hidden pytest grader checks the end state. Smaller in raw count, but this is where the community-standard evals live.
| Taskset | Tasks | What it is |
|---|---|---|
| TMax | 14,600 | Terminal tasks, each pinned to a prebuilt image; all 14,600 boot-and-setup verified |
| Terminal-Lego | ~13,800 | Docker-verified Terminal-Bench-style tasks built from real StackOverflow issues |
| OpenThoughts-TBLite | 100 | High-signal 100-task terminal-agent benchmark; hidden grader |
| Terminal-Bench 2 | 89 | Community-standard eval; 89 rigorously verified tasks |
Search
The search tasksets share one design decision: they are harness-agnostic and tool-free. The taskset ships questions and scoring only — the harness brings its own search tool (the Codex harness's built-in web search, Prime's search skill, or yours). The same tasks then train and evaluate any search-capable agent without the environment prescribing a retrieval pipeline.
| Taskset | Tasks | What it is |
|---|---|---|
| PaperSearchQA | 59,907 | Biomedical deep-research QA (54,907 train + 5,000 test); judge-graded |
| WideSeek | 44,632 | WideSearch-style table compilation; scored by item-level cell F1 |
| S1-DeepResearch | ~15,000 | Multi-hop resolution questions with gold answers; judge-graded |
| OpenSeeker | 11,677 | Web-research QA with the original judge prompt |
| DeepDive | 3,250 | Hard multi-hop research (2,234 RL + 1,016 SFT); strict boxed-answer judge |
| BrowseComp | 1,266 | OpenAI's browsing benchmark, in its Explanation/Exact-Answer/Confidence format |
| REDSearcher | 1,000 | Long-horizon web-research questions |
| BrowseComp-Plus | 830 | BrowseComp re-grounded in a fixed 100,195-doc corpus, with a controlled BM25 search tool |
BrowseComp-Plus is the one exception to bring-your-own-search: because it serves the benchmark's own BM25 retriever over a fixed corpus, the retriever becomes a controlled variable and runs are reproducible — evidence recall is tracked alongside accuracy.
Gold-validated, then re-uploaded
This is the part that separates the catalog from a link farm. For each dataset, Prime Intellect ran the gold patch through the full scoring path in fresh sandboxes, retried failures up to 10× to separate flaky from deterministically broken, ran independent second passes to catch noisy rows, and ran multiple no-edit passes to drop tasks that score 1.0 with no fix at all. The two-sided precondition is simple: gold patch applied → tests pass; no patch → tests fail. Every dropped row is persisted in the re-upload so you can audit the exclusion.
The shrinkage is not cosmetic — for the noisiest sources, most of the raw rows do not survive:
| Verified re-upload | Raw | Verified | What dropped |
|---|---|---|---|
| R2E-Gym-Subset-Verified | 4,578 | 4,522 | 56 network/timing-sensitive aiohttp/tornado tests |
| SWE-Lego-Real-Data-Verified | 4,432 | 4,323 | flaky rows, via two independent passes |
| Multi-SWE-RL-Verified | 4,703 | 2,232 | a no-edit filter caught tasks gradeable as solved with zero edits |
| SWE-rebench-V2-Filtered-Verified | 32,079 | 6,275 | wholesale-broken images; inline GitHub issue/PR references scrubbed |
| SWE-Bench-Verified-Quick | 500 | 468 | the slowest examples, for quick online-evals |
Two of these deserve a callout. SWE-rebench-V2 goes from 32,079 to 6,275 — an 80% cut — and its design goal is worth stealing: it "continuously mines fresh GitHub PRs into tasks… naturally decontaminated by recency." If your tasks are always newer than any model's training cutoff, benchmark contamination stops being a worry by construction. And Multi-SWE's no-edit filter is the quiet hero: a task that grades as solved before the agent does anything is pure reward-hack fuel, and it takes a dedicated pass to find them. The same tooling ships publicly — uv run validate <taskset-id> is the model-free sibling of eval, running the gold check and the setup-only no-op check in independent runtimes.
Why this matters for RL at scale
Strip the product framing and the reusable lesson is a data-engineering one. RL at scale does not fail on the gradient; it fails on thousands of tiny reward bugs — a flaky test, a drifted output, a container that won't boot, a task solvable without work — each of which quietly poisons the learning signal. The contribution here is treating environments as a manufactured, versioned, validated artifact: one task format, one scoring contract, prebuilt per-task images for reproducibility, grading hidden until scoring for integrity, and a gold/no-op validation gate before anything is trusted. That is the same discipline data teams already apply to training corpora, finally applied to the reward side — which, for agentic RL, is where the actual difficulty lives.
Where the reward signal still lies
The post is refreshingly candid that this is mitigation, not a solved problem. A reward signal can lie in two directions, and Prime Intellect names both.
The take
The headline number — 365,000 environments — is the least interesting thing here. The interesting thing is the contract: 23 datasets that each shipped their own harness, image conventions, and grader now load through one typed API, run on prebuilt per-task images, hide their grading material until scoring, and pass a gold/no-op validation gate before they are trusted — with the failed rows kept for audit. That is the unglamorous, correct answer to "how do you get reproducible reward at scale," and it is exactly the layer that has been missing while everyone argued about losses. Take the counts as shipped figures and the training value as unbenchmarked; take the design pattern as the real deliverable. If agentic RL is bottlenecked on verified environments — and the evidence says it is — then a validated, versioned, one-contract catalog is a more load-bearing contribution than another clever objective.
When there's no gold patch: RL over taste
Everything above assumes you can tell, mechanically, whether a rollout succeeded: a hidden test passes or it doesn't, a gold patch scores 1.0 and a no-op scores 0. That assumption is the reason 23 tasksets can share one contract at all — the taskset supplies a fact, and the fact either shows up in a run's trace or it doesn't. Hugging Face's Sergio Paniego posted a walkthrough in September 2026 of an RL environment built on the opposite premise, and reading it next to Prime Intellect's post sharpens both: same GRPO loop, same "a policy under optimization pressure finds whatever seam is left" worry, and a reward with no ground truth at all — a model that says "I like this one more," not a grader that says "this one is correct."
The project is an open, from-scratch reproduction of Surya Narreddi's viral watercolour project — also covered on this site, in Five judges were worth one opinion: a language model fine-tuned to paint by writing JavaScript against p5.brush, a library that simulates pigment bleed, paper texture, and brush pressure rather than drawing shapes. Narreddi's own post described the recipe but shipped no code. Paniego's does the opposite: the environment, the reference pool, three trained checkpoints, and the per-rollout data behind every number in the post are all public, running end to end on Hugging Face infrastructure — TRL for GRPO, OpenEnv for the environment contract, training on Jobs, the environment and the scorer as Spaces, the judge through Inference Providers.
OpenEnv, next to the taskset/harness/runtime split
OpenEnv is Hugging Face's own environment framework, and it is a much thinner thing than the Harbor contract above — worth naming precisely, because the difference is the point. Every environment subclasses one abstract class:
class Environment(ABC, Generic[ActT, ObsT, StateT]):
"""Base class for all environment servers following Gym/Gymnasium API."""
SUPPORTS_CONCURRENT_SESSIONS: bool = False
rubric: Optional["Rubric"]
@abstractmethod
def reset(self, seed=None, episode_id=None, **kwargs) -> ObsT: ...
@abstractmethod
def step(self, action: ActT, timeout_s=None, **kwargs) -> ObsT: ...
@property
@abstractmethod
def state(self) -> StateT: ...That's Gym/Gymnasium's API — reset/step/state — served over HTTP or a persistent WebSocket rather than called in-process, with Action, Observation, and State as typed Pydantic models an environment author defines per task. An optional Rubric composes the reward inside step(): a Gate that zeroes everything below it, feeding a WeightedSum of scalar terms — exactly how the watercolour environment's own reward tree is built, below.
Put next to verifiers v1's taskset/harness/runtime split, the two frameworks solve adjacent problems at very different altitudes:
| verifiers v1 / Harbor (Prime Intellect) | OpenEnv (Hugging Face) | |
|---|---|---|
| Unit of composition | a taskset — data plus scoring logic, loaded from a typed config | one Environment subclass per task, hand-written |
| Reward contract | TaskData fields (base_commit, test_patch, gold_patch) plus a @vf.reward method that restores withheld material and calls the taskset's own upstream grader | Action/Observation/State Pydantic models, an optional Rubric tree (Gate → WeightedSum) evaluated inside step() |
| Execution axis | harness (how the agent is driven) × runtime (where it executes), independent and swappable via one CLI flag | the Environment runs behind an HTTP/WebSocket server (here, a Docker Space); a client (EnvClient/GenericEnvClient) drives it — harness and runtime aren't separated concepts |
| Validation before shipping | uv run validate <taskset-id> — gold patch must score 1.0, no-op must not, in fresh sandboxes, retried up to 10× | none built into the framework — the watercolour environment's honesty gate (below) is several hundred hand-written, hand-tested lines, split across the admission gate and the static source checks it calls |
| Catalog | one API loads 23 datasets, ~365,000 tasks, across three domains | one environment; there is no cross-taskset registry to load into |
Neither framework does the other's job worse — they aren't aimed at the same job. OpenEnv is closer to a protocol: the same reset/step/state shape any RL framework already assumes, wired for a network boundary so an environment can run in a container a trainer never has to trust with GPU access. It ships no equivalent of Harbor's validated, versioned catalog, and it isn't trying to. The cost of that thinness lands on whoever writes the environment: everything Prime Intellect's validate command automates — proving a reward is trustworthy before anyone trains against it — has to be built by hand, one environment at a time. The watercolour environment's core/gate.py, core/scoring.py, and core/quality.py — several hundred lines, densely commented with the exact failure each check exists to catch — are that validation work, just uncollected into a framework and unamortized across a catalog.
The loop: a coding model that paints
p5.brush, by Alejandro Campos Uribe, exposes 47 methods that simulate a physical medium: pigment bleeds past the edge of a fill, paper has texture, flow fields drag brushwork around. The environment restricts the model to ten of them, and none of the ten takes a string argument — no brush.set, no brush.field, no brush.hatchStyle. That restriction is itself a finding, carried over from Narreddi's write-up and independently reconfirmed while building this environment: a 400-line API reference produced code that invented plausible-sounding methods that don't exist, while a short allowlist of string-free calls left nothing to hallucinate a name for. Classifying twenty-one JavaScript errors from two training runs found that ten of them were exactly this — an invented brush or field name where the model had guessed at a string the reference document had merely made plausible. Restricting what can be called is not the same move as shortening the documentation, and only the first one worked.

A submission that clears the gate looks like this — a real fixture from the environment's own test suite, using only the ten allowed calls:
async function setup() {
createCanvas(600, 600, WEBGL);
brush.scaleBrushes(3);
angleMode(DEGREES);
noLoop();
}
function draw() {
translate(-width / 2, -height / 2);
background("#f9f5f0");
brush.noStroke();
brush.fillBleed(0.25);
brush.fillTexture(0.5, 0.4);
brush.fill("#6b8f5a", 200);
brush.beginShape(0);
brush.vertex(294, 300);
brush.vertex(306, 300);
brush.vertex(306, 470);
brush.vertex(294, 470);
brush.endShape(true);
// ...petals, then the centre, follow the same beginShape/vertex/endShape/fill pattern
}A headless Chromium renders the WEBGL canvas to a PNG, and the gate runs first, for free, before anything reaches a judge: the source has to compile, use brush.* rather than bare p5 primitives, paint at least some minimum fraction of the canvas, and pass two honesty checks that exist for the same reason Prime Intellect withholds grading material — a policy under pressure will find whichever seam is left. Two fixtures from the environment's test suite show what the seam looks like here:
// cheat_external_image.js — load someone else's painting instead of drawing one
let img;
function setup(){ createCanvas(600,600,WEBGL); background("#fff");
img = loadImage("https://upload.wikimedia.org/watercolour.png"); }
function draw(){ image(img,-300,-300,600,600); noLoop(); }// cheat_text_label.js — paint almost nothing, then write the answer as text
function setup(){ createCanvas(600,600,WEBGL); background("#fcf8f2"); brush.scaleBrushes(2); }
function draw(){
brush.set("marker","#e08a72",1); brush.fill("#e08a72",150); brush.circle(0,-40,80,true);
textSize(42); fill(0); text("a beautiful watercolour hibiscus", -280, 200);
noLoop();
}There's no hidden test either of these could pass or fail — nothing here is verified in the SWE sense. What catches them is a mechanical check: external_access flags a loadImage call, writes_text flags a text() call. It's the honesty-gate stop on the spectrum below — a ground truth about whether the model painted at all, sitting in front of a reward with no ground truth about whether the painting is any good.
LLM judge, pairwise
HuggingEnvs env · Qwen3-VL-30B-A3B-Instructwhich is better, A or B?
grades The render against 4 references drawn from a 178-painting hand-rated pool, scored in both presentation orders; the reward is the fraction of comparisons won.
exploit path Position bias on a real tie — both presentation orders picked whichever image came first when neither painting was actually better. Caught by scoring both orders and paying a tie half credit, not prevented. The score still only ever means “closer to these 178 paintings.”
The reward function, four terms
Past the gate, the reward is a weighted sum — the same rubric Narreddi's write-up converged on:
| term | weight | what it measures |
|---|---|---|
gate | 0.05 | compiled, painted something, didn't cheat |
length | 0.05 | a ramp from 150 tokens (zero) to 3,000 tokens (full credit), zero again past 6,000 |
| pairwise judge | 0.60 | win fraction against 4 references sampled from a 178-painting hand-rated pool |
| HPSv3 / stand-in | 0.30 | an absolute mark on the render alone, no reference |
def build_rubric() -> Rubric:
return Sequential(
Gate(GatePassed(), threshold=1.0),
WeightedSum(
[GatePassed(), LengthRamp(), JudgeScore(), QualityScore()],
weights=[GATE_WEIGHT, LENGTH_WEIGHT, JUDGE_WEIGHT, QUALITY_WEIGHT],
),
)Two details only show up once you read the code rather than the prose. The length term used to be a flat band — one point for anything between 150 and 1,200 tokens — and a term that scores 1.0 for nearly every rollout contributes nothing to a GRPO group; it was rewritten as a ramp specifically because it was dead weight, the same diagnosis the sibling article on this site made of Narreddi's first, nine-signal rubric. And the slot HPSv3 fills isn't always HPSv3: the real 7B preference model pins an old transformers release and needs a GPU the environment's own container doesn't have, so by default the slot is filled by asking the vision judge model itself for a mark out of ten, with a real HPSv3Scorer swapped in over HTTP against a separate Space (watercolour-hpsv3, a100-large) only when one is configured. Validated side by side on the same pool before being wired in, the two disagree on how confidently they discriminate: the stand-in gives love/okay/meh tiers 9.0 / 8.4 / 7.4 — overlapping — while real HPSv3 gives them +3.5 / +3.6 / −7.5, no overlap at all.
Three checkpoints, one variable
Three adapters are published, and the Hub API confirms what the names imply: all three are LoRA adapters over the same base, Qwen/Qwen3.5-35B-A3B — a multimodal mixture-of-experts — with adapter_model.safetensors at 121,864,672 bytes in every one of the three repos. Same base model, same adapter size, same rank; the only thing that moved is the split between the two model-judge weights:
| run | pairwise judge | HPSv3 slot | steps | role |
|---|---|---|---|---|
judge-led | 0.60 | 0.30 | 110 | the original mix |
hps-led | 0.30 | 0.60 | 110 | the middle point |
hps-only | 0.00 | 0.90 | 60 | validation: does the pipeline learn at all |
That is a controlled ablation in a sense the sibling article on this site could only ask for and not run: that piece flagged that HPSv3 and a pairwise judge might be two names for one opinion, correlated highly enough to be worth less than their combined weight suggests. Here the two weights are the only thing that changes across three otherwise-identical runs, and the answer is no — moving the split visibly changes what the policy converges to, not just how fast:

Mean group reward, first third of training against the final third: hps-only moves 0.58 → 0.71 (Δ+0.13) over 60 steps; judge-led moves 0.45 → 0.72 (Δ+0.27) over 110; hps-led moves 0.57 → 0.82 (Δ+0.24) over 110. judge-led — carrying the most of the author's own taste, at 0.60 — starts lowest and spends its first thirty steps nearly flat before it moves, which is the expected shape: the more weight a reward puts on one person's pairwise calls instead of a model averaged over 1.17M human comparisons, the narrower the target, and the harder it is to find early.

By the author's own, explicitly-labelled-subjective verdict, hps-only converges hardest and stays closest to one palette, hps-led paints convincing watercolours that share an almost house "wet-on-wet" look, and judge-led ends up the most diverse and artistically interesting of the three. Whether that ranking is right is a matter of taste — which is the point this whole section has been building to.
What it actually learned
The part most write-ups skip, and the part with the most in it. In every run, the first thing the policy learns is not to paint better — it's to stop painting badly. Rollouts scoring under 0.3 (near-blank canvases, shapeless washes) fall from 99 to 16 across judge-led's three thirds and from 37 to 4 across hps-led's; in hps-only, three-quarters of the entire rise in group-mean reward comes from bad paintings simply becoming rare. That fact reframes the reward curves above: most of the climb in a GRPO group mean is the distribution's floor rising, not its ceiling.
You can see this by comparing the median painting per step against the best painting per step. In hps-only, the best-of-step barely moves — +0.034 across the whole run — while the median moves +0.155. HPSv3's slot, once it sees petals arranged around a centre and a stem, stops asking for more: reliability climbs, quality among the already-good paintings does not. The pairwise judge is the term that moves the ceiling instead of the floor: with a reference left to actually beat, a good painting can keep getting better, adding +0.12 to the best-of-step in judge-led and +0.16 in hps-led — and paint coverage doubles under both (0.11 → 0.23, 0.13 → 0.30) where hps-only barely moves it at all.
One more finding worth keeping, because it's a clean, small example of a policy doing exactly what the reward pays for and nothing else. The system prompt asks for fifteen to thirty filled shapes; the real mean across every run sits at seven to nine, and shape count barely correlates with reward in any of the three (+0.000, −0.14, +0.07). Nothing in the reward function reads shape count, so nothing about it gets obeyed — a miniature version of Prime Intellect's own point that a reward signal, not a prompt, is what actually steers a policy under RL.
And within each run, the paintings converge toward each other as training advances — the median frames across a run's steps read like takes of the same flower, because GRPO is doing exactly what it's built to do against a pool built from one subject. Jason Liu's line about taste generalizes past this one project: AI shifted the bottleneck from making to noticing. The pool decides what counts as variety the same way it decides what counts as quality — both Alex Yango's animal paintings and a hand-rated canvas-animation reproduction of the same recipe exist because someone built a different pool, not a different algorithm.
Infra is hard, again
Prime Intellect's integrity principle — anything readable in the container is fair game for a reward hack — has a quieter cousin here: anything that fails silently gets read as a bad painting. A render that timed out or a scorer that never answered was entering the reward as a flat 0.0, indistinguishable from a genuinely bad painting, in about 1.5% of rollouts across every run and up to 5.2% in the worst one. The fix has the same shape as Prime Intellect's own false-negative problem — a reward that penalizes something other than the thing it's supposed to measure — and the same shape as its fix: those paths now return None, and the rollout is dropped from the group instead of scored zero.
The other infra failure is a real bug, found and fixed upstream in OpenEnv itself. EnvClient holds one persistent WebSocket per session; when the far end closed it — a keepalive timeout, a tunnel dropping the connection, the server restarting — the cached client object still held a non-None reference to it, so the client never reconnected and every later call raised ConnectionClosed for the rest of the process's life. It cost two half-finished, multi-hour runs to trace. The fix, merged as OpenEnv PR #1103:
+from websockets.protocol import State
...
if self._ws is not None:
- if self._ws_loop is asyncio.get_running_loop():
+ if self._ws.state in (State.CLOSING, State.CLOSED):
+ # Closed by the far end: a keepalive timeout, a tunnel dropping
+ # the socket, the server restarting. Only a demonstrably closed
+ # socket is dropped, so one still CONNECTING is left alone.
+ self._ws = None
+ self._ws_loop = None
+ elif self._ws_loop is asyncio.get_running_loop():
return self async def _receive(self) -> Dict[str, Any]:
"""Receive and parse a message from the WebSocket."""
+ await self._ensure_connected()
assert self._ws is not NoneA three-way ablation training against a live judge for hours at a time is exactly the workload that finds this kind of bug. Prime Intellect's sandboxes are disposable per task and die with the rollout; a WebSocket held open against a Space for a multi-hour GRPO run is infrastructure that has to survive, and the failure modes are correspondingly different.
What it costs
Numbers, rounded, for the runs that finished. A step is eight rollouts and takes 15 to 18 minutes, of which 70 to 80% is rendering — a single render takes 69 to 96 seconds against a 90-second deadline, because the Space has no GPU and Chromium renders the WEBGL canvas, bleeds and textures included, in software. (The disclosure is refreshingly plain: the author expected it to be faster and never found the full cause.)
| piece | what it needs |
|---|---|
| trainer | 1 H200 — 18 hours for 60 steps, about 34 for 110 |
| HPSv3 scorer | an a100-large Space, up for the entire run |
| the environment | a cpu-upgrade Space; renders comfortably inside the deadline |
| pairwise judge | Inference Providers quota for Qwen/Qwen3-VL-30B-A3B-Instruct |
| the pool (one-off) | iNaturalist photos, Inference Providers quota for four generator models, and rating time |

The post names the asymmetry worth sitting with directly: "a scorer can cost more than the training that uses it." A gold-patch verifier is disposable — a sandbox boots, runs the grader, and dies with the rollout. A preference model as a live judge is a service: HPSv3's Space has to be kept warm for the run's entire duration even though it does one forward pass per rollout, and forgetting to pause it after a run ends is a real, named failure mode in the post. Taste doesn't just change what the reward can tell you. It changes what the reward costs to keep asking.
The open question underneath it
Everything above traces back to one line in the original post: 178 hand-rated paintings, every one of them already a model's output, are what this trained model has learned to call beautiful. There is no gold patch for a watercolour. The pool is not a stand-in for ground truth the way a hidden test is — it is the ground truth, entirely, and it is 178 images one person rated by hand out of four models' worth of candidates. Point the same environment at a different pool and the reward changes with zero lines of code touched; point it at a differently-curated 178 the way Alex Yango did for animals, and the identical algorithm produces a different aesthetic out the other end.
Put the two posts side by side and the lesson generalizes past either domain. However the reward arrives — a hidden test, a rule-based gate, a judge's opinion, a preference model frozen after 1.17M comparisons — the question that decides whether RL actually works is the same question, asked in a harsher key as the ground truth thins out: what, exactly, can this reward not tell the difference between? Prime Intellect spent an entire engineering post answering that for code. Hugging Face's reproduction spends this one admitting there is no clean answer for taste, and publishes every artifact anyway so the question stays open to anyone who wants to push on it.
Built on Prime Intellect's Scaling Agentic RL: 365,000+ Environments for SWE, Terminal, and Search (Daniel Auras and the Prime Intellect Team, July 2026), with the taskset details drawn from the post and the research-environments and verifiers repos it links. All task counts and validation figures are Prime Intellect's own reported numbers. The two interactive diagrams for that section are my redrawings of the mechanism (the rollout pipeline and the taskset map), not reproductions of the post's charts; the hero image is the post's own cover art. There is no independent benchmark of training outcomes in that source, and I have not run one.
The "RL over taste" section is built on Hugging Face's Training a coding model to paint watercolours with TRL and OpenEnv (Sergio Paniego, September 2026), with code and design details drawn from the post and the OpenEnv (including PR #1103) and HuggingEnvs repos it links, plus the HPSv3 model card and the three checkpoints' own metadata on the Hub. Reward numbers, training curves, and checkpoint deltas are the post's own reported figures, cross-checked against the environment's published source where the post itself doesn't spell out a mechanism. Five images in that section are the post's own (flattened onto white, capped in width, otherwise unedited); the reward-spectrum widget is my own illustrative diagram, not a reproduction of anything in either source. I have not independently retrained or re-scored any of the three checkpoints.