# Five judges were worth one opinion: RL on a reward you have to author yourself
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/paint-with-code
> date: 2026-08-24
> tags: rl, reward-design, grpo, creative-tools, explainer
When you make an image with a diffusion model, the only way to participate is the prompt. You cannot open the output and change one thing. To move a petal you go back to the text box and roll again, and hope the rest survives.
[Surya Narreddi and Cameron Franz](https://surya.website/rling-qwen-to-paint-with-code) fine-tuned a reasoning model — **Qwen3-30B-A3B-Thinking** — to paint by writing **p5.brush JavaScript** instead. The artefact is source code, so the picture is editable at the line level. That premise is a good one and it is not what makes the write-up worth reading. What makes it worth reading is that it is one of the few honest accounts of trying to run RL against a reward nobody can verify — and of the specific way that reward fell over.
| | |
|---|---|
| Post | [Training AI to Paint with Code](https://surya.website/rling-qwen-to-paint-with-code) · Surya Narreddi · March 2026 |
| Team | Surya Narreddi (design, development, RL research) · **Cameron Franz** (built the training infrastructure) · Alex Wang |
| Method | GRPO · policy emits a complete p5.brush sketch · sandboxed Puppeteer renders it · a separate judge model scores the PNG |
| Policy | **Qwen3-30B-A3B-Thinking**, per the label on the project page's own step-625 sample |
| First rubric | **9 signals** · plateaued at ~0.65 · every rollout a flat, five-petal clip-art flower |
| Diagnosis | five of the nine correlated **0.85–0.95** · code length, ~⅓ of the reward, saturated by **step 30** · HPSv3, the only signal with variance, weighted **0.10** |
| Second rubric | **4 signals** · same base model, same data · old plateau reached **3× faster**, then kept climbing |
| A point above it | a labelled sample reads **step 625, r = 0.79** — one datapoint past the 0.65 plateau, not a ceiling |
| Side effect | generated code compressed **13,500 → under 2,000 tokens** |
| Reference pool | **581** paintings — 117 love-tier + 266 okay from 1,664 hand-rated generations, plus 198 supplements |
| System prompt | GEPA, **200 iterations** → allowlist of **8** brush methods, no API docs, no examples |
| Status | ongoing · one more training run planned · full technical report promised for June |
## The loop
Nothing in that loop is exotic. A prompt goes in, a headless browser renders a canvas, GRPO normalizes rewards within a group of rollouts on the same prompt. The whole design surface is the two stages in the middle: **what the judge is asked**, and **what it is asked to compare against**. Both are hand-authored, and both are where the first run went wrong.
## Nine signals, one opinion
The first rubric had nine components: a compilation gate; a check that the sketch used p5.brush rather than native p5; a code-length ramp targeting around 3,000 tokens; HPSv3, a human-preference model; prompt adherence judged by a council of GPT-5.4 and Gemini; and four more quality judges — recognisability, aesthetics, technique, depth.
It plateaued around 0.65 and stayed there. The reward kept climbing. The pictures did not.
That final frame is the thing to sit with. It is not a broken render or a diverged policy. It is a *competent* answer to the question that was actually asked, and the question was badly posed.
The diagnosis came from reading the sub-rewards separately rather than as a sum. Three facts:
- the four quality judges plus prompt adherence were **correlated with each other at 0.85 to 0.95** — five measurements of one thing
- code length, contributing **roughly a third of the total reward**, had **saturated by step thirty** and produced zero gradient afterward
- **HPSv3, the one signal showing real variance, was weighted at 0.10**
The post reports the correlation and moves on. The arithmetic it implies is worth doing, because it turns a qualitative complaint into a number. For `k` equally-weighted signals sharing a common pairwise correlation `ρ`, the effective number of independent signals is
$$
n_{\text{eff}} = \frac{k}{1 + (k-1)\rho}
$$
At `k = 5` and `ρ = 0.90` that is 5 / 4.6 = **1.09**. Across the whole reported 0.85–0.95 band it runs from 1.04 to 1.14. Five judges, five API bills, five sets of latency — and between them, **one opinion**.
Now stack it up. A third of the reward was dead weight after step 30. Something like half of it was one opinion stated five times. The single measurement that could have disagreed was worth a tenth. A policy optimizing that is being shouted at about one thing, whispered at about a second, and told nothing at all by the rest — and a flat, symmetrical, five-petal flower is a very reasonable thing to converge on when the loudest signal is "does this read as a flower".
The failure is worth naming precisely because it is invisible from the training curve. Reward went up. Nine signals were being tracked. Every one of them was behaving. You only see it by asking a question that summed rewards cannot answer — *how many of these are independent?* — and that question has a closed-form answer that costs nothing to compute.
## Fix one: ask a relative question
The original rubric asked the judge to score each rollout from zero to ten. The scores came back compressed near zero.
The replacement asks something else. Show the judge the rollout and two references sampled from the pool, and ask: *which of these is the better hibiscus watercolour?* The reward is the fraction of comparisons won.
There is a genuine puzzle here that the widget is built around. Two references means the pairwise reward can only be 0, 0.5 or 1. **Three levels, where the scale it replaced had eleven.** On paper that is a downgrade.
It is not, and the reason generalises past this project. GRPO normalizes rewards within a group, so what propagates is the *ranking* of the rollouts in that group, not the magnitude of any one score. A group of eight rollouts contains 28 pairs, and every pair the reward assigns the same number to is a pair the update learns nothing from. Eleven levels spent on two adjacent integers leaves most of those pairs tied. Three levels that actually track quality do not.
Dynamic range is a property of the answers you get back, not of the scale you offered. The post's own phrasing is the right one: the judge *"handles a relative question more reliably than an abstract scale."*
## Fix two: build something to compare against
The second half is 1,664 images rated one at a time into love, okay and nope.
117 landed in love-tier and seeded the comparison pool. 266 were okay. The remaining **1,281 — about 77% of everything generated — were nope**, which is a number worth keeping in mind next time a pipeline reports its best-of-n.
But the pool that shipped is not the 117. It is 581: the love-tier, the okay-tier, and 198 supplements from a separate generation run, added to widen the comparison set in colours where hand-rated examples were thin.
The post frames the additions as coverage, and they are. There is a second reason they matter that follows directly from what a comparison reward *is*. Beating an opponent tells you nothing when you always win or always lose; the information lives where the outcome is uncertain. Judge a mid-training rollout against nothing but the best 117 paintings in the collection and it loses nearly every comparison, the reward pins near zero, and the gradient goes flat — **the same compression failure that the switch away from 0–10 scoring was meant to fix, reintroduced through the opponent distribution instead of the scale**.
Padding the pool with merely-okay paintings looks like lowering the bar. It is really moving the opponents to where losing is still informative.
Every image in that pool is model output, incidentally — p5.brush is niche enough that there was no corpus of human examples to draw on. So the reward model for "good painting" is anchored to one person's ratings of a machine's attempts. That is not a criticism; it is the honest shape of the problem, and the post says so.
## The system prompt finding
Early versions of the system prompt included a 400-line p5.brush API reference. The model produced confident, well-formatted code that invented APIs that did not exist.
The fix ran through GEPA — a prompt-optimisation library that evolves a prompt against a scoring function — for 200 iterations against a taste-anchored 7-shot judge. It converged on a strict allowlist of **eight** brush methods, no API documentation, and no examples. The first version to get three visible hibiscus blobs out of three generations was the one written after throwing the 400-line reference out entirely.
More documentation about the library made the model hallucinate more of the library, which is counterintuitive until you notice that an API reference and an allowlist are different speech acts. A reference says *here is what exists*, and a model writing fluent code against a large described surface will reach for the method that ought to exist. An allowlist says *here is what you may call*, and a call outside it is a visible violation rather than a plausible guess.
I would hold the generalisation slightly more loosely than the post does. Two prompts were run; nothing on the axis between them was. And GEPA rewrote the entire prompt over 200 iterations, so "we removed the API reference" is one edit inside a search over many, not a controlled ablation. The direction is convincing and the mechanism is plausible; the dose-response curve the generalisation implies has two points on it.
## What it actually produces
Everything above is about a reward function. This is what came out of the other end.
That label is worth reading twice, because it settles two things the prose leaves open. The policy being fine-tuned is **Qwen3-30B-A3B-Thinking** — a reasoning model, not an image model, which is the whole premise made concrete. And **r = 0.79 at step 625** is a real datapoint above the 0.65 plateau the first rubric died on, rather than an assurance that the curve kept going.
It is one point, not a ceiling. But it is the difference between "kept climbing" as a claim and "kept climbing" as a number, and I had written the ceiling off as unmeasured before I looked properly at the page's own images.
what the medium can do — nineteen frames, three outputs each
This is the part a reward-function post-mortem tends to leave out, and it is the part that says whether the fix worked. The failure mode being diagnosed was **mode collapse** — every rollout the same flat five-petal flower — so the only convincing evidence of a repair is range. Poppy fields, a bee, lavender, a night sky, a black spiral of hatching: these are not variations on one composition, and none of them is the clip-art flower.
five more, held longer
Worth keeping in view, though: every one of these is a *flower*. The prompts, the 581-painting reference pool and the taste-anchored judge are all hibiscus-shaped, so the range on display is range within one subject. Whether the same rubric transfers to a domain where the author has no hand-rated pool is exactly the question the unbuilt reward model was meant to answer.
## What I would want measured next
**The reward model that did not get built.** The post names it: the next step, not taken, was training a small reward model on the 1,664 ratings themselves, so the sense of *good* could be applied without comparing against the pool every time. That is the step that decides whether this is a technique or a one-off. Pairwise-against-a-pool is bounded by the pool; a learned reward is bounded by whether taste generalises out of 1,664 labels. Those are very different ceilings and only one of them has been probed.
**The 3× is a speed claim, and the interesting number is still the ceiling.** "Reached the previous plateau three times faster and kept climbing" is the right thing to report from a run in progress, and it is not the same as knowing where the second run stops. The step-625 sample above gives one point at r = 0.79, which is genuinely above the old plateau — but a single labelled frame is not a curve, and the concern survives it: a reward that climbs past 0.65 might be finding real quality or might be finding the second rubric's own exploitable corner. The first rubric's reward also climbed, right up to the clip-art flower. What would settle it is the reward curve with samples pinned to it at intervals, which is a chart the project must already have.
**Code compressing 13,500 → under 2,000 tokens needs one more sentence.** The post reads it as the model learning that verbose code did not help, and that is the natural reading. But a binary length check replaced a length *ramp*, so the pressure that was pushing toward 3,000 tokens simply left, and both stories predict the same drop. Do the short sketches score better on the pairwise judge than the long ones did, holding the rubric fixed? One scatter plot settles it.
**Correlation was measured once, on the rubric that failed.** The fix was to delete the redundant judges. The check that the new rubric is not quietly redundant too — HPSv3 at 0.30 and the pairwise judge at 0.60 are both, in the end, opinions about whether the picture looks good — is the same cheap computation that found the problem the first time. `n_eff` for `k = 2` at `ρ = 0.7` is 1.18.
## Someone ran the ablation
A year is a long time for an open question to sit unanswered, and this one did not sit that long. On 3 September 2026, [Sergio Paniego posted a walkthrough](https://huggingface.co/blog/train-to-paint-with-code) of an open, from-scratch reproduction of the same project: a different base model, a much smaller pool, and — the part that matters for everything above — every artefact published. The recipe, the environment, the reference pool, three trained LoRA adapters, their rollout datasets, and the per-step training curves as CSV.
Narreddi's write-up describes a recipe; it ships no code. Paniego's post closes with a section literally titled "What I changed from the original," which is the tell that this is not a fresh take on the idea — it is a rebuild against the same blueprint, with every divergence named. That is why it belongs here rather than as a separate piece: it answers part of what the previous section asked for, on a different model and a different pool, and it leaves part of it exactly as open as before.
This section is the experiment, not the infrastructure. OpenEnv's `reset`/`step`/`state` contract, its `Rubric` tree, and how that compares to Prime Intellect's taskset/harness/runtime split are covered from the infrastructure side in [Scaling agentic RL: 365,000 environments behind one contract](/articles/scaling-agentic-rl#openenv-next-to-the-tasksetharnessruntime-split), which reproduces the same project for a different argument — that piece is the one to read for the environment-as-contract angle. What follows is the reward design, the ablation, what it produced, and what it cost.
| | |
|---|---|
| Post | [Training a coding model to paint watercolours with TRL and OpenEnv](https://huggingface.co/blog/train-to-paint-with-code) · Sergio Paniego · Hugging Face · September 2026 |
| Base model | **Qwen/Qwen3.5-35B-A3B**, LoRA rank 16 — a different model from Narreddi's Qwen3-30B-A3B-Thinking |
| Reference pool | **178** paintings (love + okay tiers), from 206 candidates over three refinement rounds — Narreddi's 581 was never published, so this is a fresh pool, not a reuse |
| Three runs | one variable each: `judge-led` 0.60/0.30, `hps-led` 0.30/0.60, `hps-only` 0.00/0.90 (pairwise judge / HPSv3 slot) |
| Everything published | recipe, environment, pool, three adapters, three rollout datasets, per-step curves as CSV |
### What changed, and what didn't
The four-term reward is Narreddi's, kept almost exactly: a binary `gate` at 0.05, a `length` term at 0.05, the pairwise judge at 0.60, HPSv3's slot at 0.30. One term changed shape, and the reason is the same diagnosis this article opened with. The original length term was a flat band — one point for anything between 150 and 1,200 tokens — and a term that scores 1.0 for nearly every rollout contributes nothing to a GRPO group. It was rewritten as a ramp:
```python
MIN_LENGTH_TOKENS = 150
TARGET_LENGTH_TOKENS = 3000
RUNAWAY_LENGTH_TOKENS = 6000
def length_score(source: str) -> float:
tokens = len(source) / 4
if tokens < MIN_LENGTH_TOKENS or tokens > RUNAWAY_LENGTH_TOKENS:
return 0.0
if tokens >= TARGET_LENGTH_TOKENS:
return 1.0
return (tokens - MIN_LENGTH_TOKENS) / (TARGET_LENGTH_TOKENS - MIN_LENGTH_TOKENS)
```
That is the environment's own source, not a paraphrase, and its docstring says the quiet part out loud: "a term that is one for every rollout contributes nothing to a GRPO group, so the only signal about elaboration in the whole rubric was doing no work." It is the identical complaint the section above makes about a code-length gate that saturated by step thirty — reached independently, in a different codebase, before this article existed to compare it against.
Two smaller, explicitly deliberate changes sit next to it. The pairwise judge now draws its four references half from `love` and half from `okay`, rather than the top tier alone:
```python
DEFAULT_MIX = {"love": 0.50, "okay": 0.50, "rung": 0.0}
```
The reasoning is the same shape as [the pool-widening argument two sections up](#fix-two-build-something-to-compare-against): sampling only what a weak early policy cannot beat pins the reward near zero and the gradient goes flat. And the system prompt gained one sentence — paint each petal two or three times, a big pass first and a smaller, more opaque one inside it — which the author reports made the outputs noticeably more colourful. Everything else, `all-linear` LoRA targeting for a mixture-of-experts base, infrastructure failures returning `None` instead of a silent zero, stopping the judge runs at step 110 of a 200-step launch, are decisions the original post never specified, made once here and reported as such.
The trainer needed its own four fixes before any of the above mattered, and they are worth naming because they are not reward-design at all — they are the difference between a reward that cannot possibly work and one that might:
| setting | from | to | measured reason |
|---|---|---|---|
| learning rate | 2e-5 | **5e-5** | at 1e-6, entropy, completion length and the paintings themselves sat unchanged over thirteen steps while the reward oscillated in the noise of the reference draw |
| scheduler | `linear` | **`constant_with_warmup`** | linear decay had spent 79% of one run's total parameter-space displacement by step 33 of 60 |
| `scale_rewards` | `group` | **`none`** | dividing advantages by the group's own standard deviation let one gate rejection — present in 55% of groups — shrink the other seven rollouts' advantages by a factor of 0.76 to 0.84 |
| `target_modules` | hand-written list | **`all-linear`** | the hand list is written for a dense transformer and reaches 0.9% of this mixture-of-experts model's weights |
None of that is visible in a reward curve. It is the kind of failure this whole article is about, one level down: not "the reward is measuring the wrong thing" but "the reward cannot reach the weights it is supposed to move," which looks identical from outside the training loop until someone reads the optimiser settings.
The last change is a quieter callback to [the system-prompt finding](#the-system-prompt-finding) above. Building this environment independently reconfirmed it with a number: classifying twenty-one JavaScript errors from two training runs found ten were invented brush or field names — a call the model plausibly guessed at because the surrounding documentation described a larger surface than the training-time allowlist actually grants. The fix that stuck was the same *shape* of fix Narreddi's GEPA search converged on: a short, string-free allowlist (ten methods here, eight in Narreddi's) rather than a fuller reference — none of the ten take a string argument, so there is no name left to invent. Two independent teams, two different models, the same failure mode and the same class of fix — which is closer to a real replication than the original section could claim for itself, where "two prompts were run, nothing on the axis between them was."
HPSv3's slot at 0.30 is not always HPSv3. The environment container has no GPU and the real model pins an incompatible `transformers` version, so by default the slot is filled by asking the judge model itself for an absolute mark out of ten — a stand-in validated before being wired in, and measured against the real thing:
```
stand-in love 9.0 okay 8.4 meh 7.4 (overlapping — one "meh" outscored one "love")
HPSv3 love +3.5 okay +3.6 meh −7.5 (no overlap at all)
```
The three published runs, though, do use the real model: the environment's own hardware table lists a dedicated `a100-large` Space for HPSv3 as a cost every run pays for its whole duration, and `WATERCOLOUR_HPSV3_URL` has to point at it or the term silently scores zero. So the ablation below is a real preference model against a pairwise judge, not two flavours of the same VLM — but the stand-in exists, is documented, and is what a reader without spare GPU budget gets by default. Worth knowing which one you are looking at.
### Three adapters, one number moved
Three checkpoints are published: `watercolour-grpo-judge-led`, `watercolour-grpo-hps-led`, and `watercolour-grpo-hps-only`, all under `HuggingEnvs` on the Hub. The Hub API and each repo's `adapter_config.json` — checked directly here, not taken from the post — say all three are LoRA over the identical base, `Qwen/Qwen3.5-35B-A3B`, rank 16, alpha 32, dropout 0, the identical seventeen-module `all-linear` target set (each repo serialises the set in a different order — a Python set carries no guaranteed iteration order — but sorted, the three lists are identical), and an `adapter_model.safetensors` of the identical size in all three: 121,864,672 bytes. The sha256 of that file differs in every repo. Same architecture, same size, to the byte — and three genuinely different sets of trained weights. Which is exactly what a real ablation should look like: if the files were bit-for-bit identical, no training happened.
The only thing that was ever meant to move is the split between the pairwise judge and HPSv3's slot, and this is the direct test of the question the first half of this article could only ask: are those two terms measuring one thing or two?
The raw correlation between the two terms' per-step group means — computed here from the published CSVs, a number neither post reports — sits at 0.68 to 0.76 across the two runs that carry both. Run the same `n = 2 / (1 + ρ)` arithmetic this article opened with and that is barely more than one opinion between them, which would seem to confirm the original worry outright. It doesn't, quite: most of that correlation is the two curves climbing together as the policy improves over 110 steps, not the two judges agreeing about any single picture. Strip the shared trend — correlate step-to-step deltas instead of the raw series — and it drops to 0.14–0.46:
$$
n_{\text{eff}} = \frac{2}{1 + \rho}
$$
at which point the two terms are worth 1.4 to 1.8 opinions out of a possible two, not 1.1. Neither number is the one the original correlation math actually needs — that calculation wants agreement between two judges scoring the *same* rollout at a *fixed* checkpoint, and both of these are aggregate statistics across a moving policy. The cleaner version of this test is sitting in the published rollout datasets, one row per submission with both scores attached, and nobody has run it yet.
What settles the question better than any correlation coefficient is behavioural: moving the split from 0.60/0.30 to 0.30/0.60 to 0.00/0.90 changed the shape of three otherwise-identical training runs, and it changed what came out the other end. `hps-only` converges hardest and settles on a handful of colours. `hps-led` paints convincing watercolours with a shared wet-on-wet look that reads as almost a house style. `judge-led` — the split Narreddi's own write-up converged on — ends up the most diverse and, by the author's own explicitly-subjective verdict, the most artistically interesting. Two terms that were truly one opinion stated twice would have produced one outcome regardless of how the weight moved between them. They did not.
### What it produced, and what it cost
Range, not just quality, is the thing to look for here, for the same reason it mattered in [the earlier gallery](#what-it-actually-produces): a rubric that has genuinely stopped rewarding one flat composition should produce more than one flat composition. It does. Every one of these is still a flower — the pool, the criteria sentence, and the judge are all hibiscus-shaped, so range within the subject is what the reward can show, not range across subjects — but within that subject the tiles vary in palette, density, and how much of the canvas carries paint in a way the original's plateau never did.
The cost side has no dollar figures in the source, so none appear here. What it names instead: one H200 for eighteen hours to reach step 60, about thirty-four for step 110. HPSv3 kept on a dedicated `a100-large` Space for the entire run, whether or not a given step happens to call it. A step is eight rollouts and takes fifteen to eighteen minutes, of which seventy to eighty per cent is rendering — a single headless-Chromium render takes 69 to 96 seconds against a 90-second deadline, software-rendering a WEBGL canvas with no GPU in the container, and the author reports expecting it to be faster without finding the full cause. The line worth keeping from the source, verbatim: a scorer can cost more than the training that uses it. [The infrastructure diagram and the full cost breakdown](/articles/scaling-agentic-rl#what-it-costs) live in the sibling piece; the number this article adds to it is the one above — no run in this ablation ever lost its gradient (`frac_reward_zero_std` stayed at 0.000 throughout all three), which is the property the *first* rubric this article opened with did not have past step thirty.
### What this answers, and what it still doesn't
The previous section closed with four things worth measuring next. This is not a continuation of Narreddi's own run — different model, different pool, a different team — so none of it settles a question about *that* checkpoint specifically. It does say something about the shape of the problem in general, one bullet at a time.
**The reward model that did not get built still hasn't been.** Paniego's own closing section names the identical gap: "178 paintings made by models define what this trained model considers beautiful. The pool is the bottleneck." What changed is the workaround — borrowing a general preference model trained on 1.17 million unrelated human comparisons (or, absent that, a zero-shot mark from the judge itself) rather than training a small model on the 178 or 1,664 in-domain ratings. That is a different answer to a nearby question, not the original one.
**The ceiling question has real curves under it now, and they still don't resolve it.** All three runs kept a positive slope to the point they were stopped, and no group in any of them ever lost its gradient — a materially better property than the original rubric had by step thirty. But `judge-led` and `hps-led` were launched for 200 steps and stopped at 110 to save compute, still climbing. The ceiling is still unmeasured; what's new is evidence that a two-term, judge-plus-preference-model reward can climb past where a nine-signal one stalled without visibly running out of room in 110 steps.
**The length-compression question was not tested — it was pre-empted.** Whether short sketches score better than long ones, holding the rubric fixed, needed the original flat band to be run and then compared against a ramp. Paniego's environment ships with the ramp already in place, for the same reasoning this article gives independently. The controlled comparison the original section asked for still doesn't exist; what exists now is a second team reaching the identical fix without reading the first team's diagnosis.
**The redundancy question is the one this actually moves.** The bullet above this one, written before any of the runs above existed, guessed `n_eff` for `k = 2` at `ρ = 0.7` and got 1.18 — almost exactly where the raw, trend-inflated correlation measured here lands (1.14 to 1.19). Detrended, it moves to 1.4–1.8. Neither number is the clean per-rollout measurement the guess was really asking for, and the right version of that test is sitting unrun in the published rollout data. But three checkpoints that share everything except one weight, producing three visibly different training curves and three visibly different painting styles, is stronger evidence than any correlation coefficient that a pairwise judge and a preference model are not simply one opinion counted twice — which is the most concrete answer any section of this article has gotten.
## Why this one is worth your time
The write-up is not a paper. It is a blog post about a project that is still running, with a technical report promised later, and it reports one plateau, one diagnosis, one fix, and no baselines. Take the numbers as a field report.
Take the *structure* more seriously than that, because it is the clearest small example I have seen of a failure mode that is going to keep happening. Everyone doing RL on subjective work will build a rubric. Rubrics accrete signals, because adding one is easy and each addition feels like a defensible improvement — of course recognisability matters, of course technique matters. Nothing in the training curve ever tells you that you added the same signal four times. The reward goes up. The work does not get better. And the model, which is doing exactly what you asked, hands you back a flat flower with five rounded petals.
The correction is not more judges. It is `k / (1 + (k−1)ρ)`, computed before you trust the sum.
