2026-09-22 · 15 min · explainer · agents · reinforcement-learning · evaluation · llm
"Collective" in a paper about agents usually means one of three things: a shared replay buffer, a shared skill library the agents read at run time, or a joint policy update across a population. Reinforcing Agents with Collective Skills (Xu et al., NVIDIA, September 2026) means none of them.
It means the training distribution comes from what thousands of people
bothered to write down. A public Agent Skill is a folder rooted at a SKILL.md
— frontmatter with a name, a description and trigger conditions, then free-form
instructions, optionally with scripts/, references/ and assets/. Since the
format appeared in late 2025, practitioners have published thousands of them.
The paper's claim is that read as data, that corpus is three things at once: a
sample of the tasks people actually want delegated, a set of pointers to real
artifacts, and a body of quality criteria that outcome tests cannot express.
So Skill2Env is a compiler. SKILL.md in, containerised reinforcement-learning
environments out. 3.4k Skills became 7,971 Harbor tasks across thirteen
domains, and RL over them moved Qwen3.8-27B +4.7 points on Terminal-Bench 2.1
and +4.3 on their own held-out set.
That is the headline. The result I have spent longer on is the one that lost.
The compiler, and the two gates that matter
Static checks first: layout, symlinks that must resolve inside their own subtree, Dockerfiles whose build context cannot reach privileged files, base images pinned to a content digest from an approved registry. Then two Harbor trials in fresh containers.
The Oracle must earn full reward on every metric. A no-op agent must earn zero on every metric. Fail either and the candidate is dropped.
The two gates in stage four are doing more work than the three model stages before them. An Oracle that must score full marks catches a verifier that is impossible to satisfy; a no-op agent that must score zero catches a verifier that passes for free — the single most common way a synthesised RL task is silently worthless. Neither gate involves a model, which is why they can be trusted about the models that wrote the task.
Two things in that pipeline are worth taking rather than admiring.
The ordering rule in stage three. The creator agent builds the world, writes the instruction, implements the tests and writes the rubric, and only then writes the reference solution. The paper says why in one sentence:
Tests and rubric are frozen before the solution exists so that the solution has to satisfy the grading contract rather than the other way round; a failing reference solution is treated as a solution bug, not a reason to weaken the verifier.
Anyone who has watched a model write both a test and the code it tests knows which way that pressure runs by default.
The two acceptance gates. Every candidate is built and run twice in fresh containers: the reference solution must earn full reward on every metric, and a no-op agent must earn zero on every metric. Fail either and the task is dropped. The first catches a verifier nothing can satisfy; the second catches a verifier that passes for free, which is the most common way a synthesised RL task is silently worthless. Neither gate involves a model, which is precisely why they can be trusted about tasks that models wrote.
This is the same pair Prime Intellect's taskset work landed on independently — gold-patch and no-op validation on every re-upload, across ~365,000 tasks. Two groups, two corpora, the same two assertions. At this point the Oracle/NOP pair should be table stakes for anyone publishing a synthesised environment set, and its absence should read as a missing section.
There is also a deliberate omission worth naming, because it cuts the other way from most pipelines. Skill2Env does not run a validation pass where a teacher model attempts each task and discards the ones it fails — the standard move. Their reasoning: that caps task difficulty at the validator's ability and roughly doubles generation cost, and group-based RL with online dynamic filtering already discards prompts whose rollouts carry no advantage. Unsolvable and trivial tasks get filtered on-policy instead of a priori. That is a defensible trade and it means the corpus contains tasks nobody has solved, by design.
What "collective" buys, measured two ways
The first measurement is coverage, and it is the cleanest evidence in the paper.
Software engineering is 22.5% of Skill2Env, against pie charts to its right that are one or two slices wide. Business, marketing, research, documents, product, design, productivity and health together outweigh it. If you believe that agentic RL generalises from the distribution it trains on, that chart is the paper's thesis rendered as area.
The second measurement asks whether the compiled task still carries anything of the Skill, and it is the kind of check I wish more data papers ran.
The row worth staring at is the second. If the rubric were generic agent advice, it would retrieve nothing in particular from 3.4k documents. At 68.5% top-1 it identifies its own Skill almost as well as the whole task does, which is the paper’s evidence that the compiler carried the practitioner’s methodology forward rather than paraphrasing it into boilerplate. TF-IDF is a lexical measure, so some of that is shared vocabulary rather than shared method — but shared vocabulary at 2,300x chance is still a signal that the rubric is about this job and not about jobs in general.
A TF-IDF retrieval probe over all 8k tasks: query with a task's instruction and
rubric, search the 3.4k original SKILL.md documents, and the true source is
top-1 for 73.2% and top-10 for 94.6%, against 0.03% chance. The number
that earns its place is the ablation: the rubric alone retrieves its Skill at
68.5% top-1. Generic agent advice would retrieve nothing in particular. A
rubric that fingerprints its own Skill at 2,300x chance is carrying that Skill's
specific methodology.
So "collective" is defensible as a distribution claim and as a fidelity claim. The question is what happens when you actually train on the methodology half.
The result that lost
Two runs. Identical data, identical 300 steps, identical DPPO recipe — group advantages over G = 8 rollouts, the binary-KL token mask at δ = 0.05, no reference model, no KL penalty, no entropy bonus. One term of difference in the reward:
where is the programmatic Harbor reward averaged over up to six
named test metrics, and is an LLM judge's score for the trace,
given the task's rubric.md and a "Constitution" penalising aimless loops,
reward hacking and unresponsive answers. The judge can move the reward by at most
±0.2.
Click through all four. The outcome-only run wins every benchmark; the rubric run wins the only measurement that asks whether the agent worked the way the Skill says to. The paper does not hide this and does not spin it either: the judge term stays flat from the first update to the last, the programmatic reward sits below the outcome-only run for most of training, and the authors write that “the methodology the Skills write down may simply not be the distribution that maximizes benchmark pass rates.” That sentence is the whole result.
Outcome-only RL wins every benchmark. The rubric run gives back 4.0 points on
Terminal-Bench 2.1, 3.0 on S2EBench pass and 11.5 on S2EBench mean score. And on
the one measurement that asks whether the agent worked the way the Skill says
to — 200 paired trajectories, a judge holding the source SKILL.md, A/B order
randomised — the rubric checkpoint is preferred 73.0% of the time against the
base model, where outcome-only manages 54.5%.
The paper's diagnosis is careful and I think correct:
The rubric run's programmatic reward sits between 0.5 and 0.6 for most of training, below the outcome-only run, while its judge term stays flat from the first update to the last. The policy gives up some test passes for behavior the rubric rewards, and the judge signal shows no trend it could be climbing.
A flat judge term across 300 updates is the detail to sit with. It is not that the judge is broken — it assigns −2 or lower to 21% of traces and grades evidenced reward hacking at −5, so it discriminates. It is that the policy never found a direction that moved it. Meanwhile the additive form lets a task that failed every test still earn positive reward, and the judge sees the agent's messages but not the filesystem. The authors list all three as untuned choices rather than defending them.
But they also name the possibility that is not a tuning problem, and it is the sentence I would put on the cover:
The methodology the Skills write down may simply not be the distribution that maximizes benchmark pass rates.
That is the honest statement of what "collective" costs. A corpus of human methodology encodes how people think a job should be done — evidence traced, citations reconciled, defaults respected. A benchmark encodes whether the tests pass. Optimising the first at λ = 0.2 measurably degrades the second, and the only instrument that sees the gain is a preference test whose judge is another model reading the Skill.
Scale, in the right units
I am glad the paper drew this one. Isolated, "+4.7 points from 300 RL steps" sounds like a step change. In the frame, it is a 27B open model closing roughly an eighth of the distance to the best cloud agent, from a corpus that cost over $90,000 in API usage to generate with GPT-5.6 Sol at xhigh reasoning effort. Both facts are true and the second one is not in the abstract.
The transfer argument does hold up, and it is guarded: Terminal-Bench 2.1 shares no task above 13-gram Jaccard similarity of 0.8 with the training corpus, and the training set was never tuned toward it beyond picking a technical subset. A gain that survives that check reads as a general improvement in planning, tool use and finishing rather than a memorised task family.
One more negative result deserves repeating because it replicates somebody else's. They rolled out every task twice with GLM-5.3 — 15,968 trajectories, mean reward 0.74, median 19 model calls and 23 tool calls — and used them for SFT. On Qwen3.8-27B it slightly raised S2EBench and lowered Terminal-Bench. On Qwen3.5-4B it collapsed the model: 18.7 → 3.4 on Terminal-Bench, 5.8 → 0.0 on S2EBench pass, with repeated thinking and tool-call loops. Their reading is that the teacher's interleaved-thinking style overwrites patterns the student's own post-training relies on without transferring the competence. TMax reported the same shape with a weaker teacher. Two papers, two teachers, one lesson: SFT on a better model's traces is not a free cold start for an already post-trained agent.
What I would check before using it
The released dataset is not the dataset the results came from. Footnote 1, and it is a real caveat rather than a formality:
For legal reasons, the initial dataset released is generated with Kimi-K3-max under the same pipeline, dropping a small subset from source Skills with concerning influence.
The paper's 7,971 tasks and every number above come from a GPT-5.6 Sol corpus. What you can download was regenerated with a different model and is missing some Skills. Same pipeline, same gates, and the Oracle/NOP checks mean the released tasks are at least valid — but nobody has published a number comparing the two corpora, and "same pipeline" is not the same as "same data."
S2EBench is private. Seventy-nine hand-verified tasks, reviewed for unambiguous instructions, faithfulness to the source Skill and well-rounded tests. The review criteria are stated, the tasks are not released. The largest single number in the paper — S2EBench mean score, 56.6 → 75.1 — is on a set nobody outside NVIDIA can run. Terminal-Bench 2.1 is the checkable half, and it moved 4.7.
The rubric channel is unfinished and the paper says so. A single λ, an additive form that lets a failed task earn positive reward, and a judge blind to the filesystem. Anyone building on this should read Section 5.2 as a list of three experiments rather than a result.
The take
The mechanism is a compiler with two model-free gates, and it is good. The corpus is the broadest terminal-agent training distribution anyone has published, and the domain chart is the argument. The retrieval probe is the right check and it passes.
But the word in the title is doing something specific, and it is worth being precise about. "Collective" is not a training architecture here. It is a claim about where supervision comes from: not a taxonomy someone designed, not a repository crawl, but the accumulated residue of people writing down how their own jobs should be done. That is a genuinely different source of signal and this is the first paper I have seen treat it as a dataset rather than as a prompt.
And then the corpus turns around and reports that optimising the part of itself that encodes how costs you the part that measures whether. The authors could have buried that in an appendix. They ran it as a headline ablation, showed the judge term going flat for 300 steps, and wrote down the interpretation that hurts their own framing. That is the most valuable thing in the paper, and it is a result the field will have to deal with long after this particular corpus is superseded: collective preference and benchmark pass rate are two objectives, and right now we can only optimise one of them at a time.
What would change my mind
5 claims above, and what would falsify each
"Collective" here means the task distribution, not shared replay, a shared skill library at train time, or joint policy updates.
Read off the pipeline: Skills are consumed once, offline, to produce containerised tasks, and RL runs on a single policy against those tasks. It falls if some part of the system I missed lets a training agent read the Skill corpus at rollout time, or shares experience across concurrent policies. Nothing in Sections 3 or 4 describes either, but I read the paper and the released code index, not the code.
Adding the rubric channel costs benchmark points and buys Skill-alignment preference.
One seed, one λ, one judge model, 300 steps. A λ sweep that finds a setting improving all three benchmark metrics over outcome-only would retire the tension outright, and the paper itself predicts this is possible. Equally: the preference test's judge is the same model family used as the training judge, which is a conflict the paper does not flag. A human preference study, or a different judge, could move the 73.0% substantially.
The Oracle-must-pass / no-op-must-fail pair is what makes a synthesised environment set trustworthy.
Both gates test the verifier, not the task. A task can pass both and still be trivially solvable by a shortcut the reference solution happens not to take, or unfaithful to the Skill it came from. If someone hand-audits a random 100 of the released tasks and finds a meaningful rate of shortcut-solvable environments, the gates are necessary and clearly not sufficient, and my "table stakes" framing is too generous.
The rubric carries Skill-specific methodology, at 68.5% top-1 retrieval against 0.03% chance.
TF-IDF is lexical. A rubric that merely repeats the Skill's distinctive nouns — tool names, file extensions, domain jargon — would retrieve it without carrying any methodology at all. An embedding-based probe, or a probe with the Skill's proper nouns stripped from both sides, would separate shared vocabulary from shared method. The paper runs the lexical version only.
The +4.7 on Terminal-Bench 2.1 is a general gain, not a memorised task family.
The decontamination check is 13-gram Jaccard at 0.8, which is a surface-form test. Two tasks can share a solution strategy without sharing 13-grams. A held-out family the corpus demonstrably never touches — a domain absent from all 3.4k Skills — would be a stronger test, and S2EBench cannot be it, since it is drawn from SkillHub too.
Source: Binfeng Xu, Jian Hu, Hao Zhang, Shaokun Zhang, Yifan Zhang, Michael Demoret, Jan Kautz and Yi Dong, Reinforcing Agents with Collective Skills (NVIDIA, 21 September 2026). Code at NVlabs/Skill2Env. Figures 1, 2 and 3 are reproduced from the paper for commentary; every quotation is from its text. The compiler, reward-tension and retrieval-probe interactives are my own illustrations, built from the paper's Tables 1 and 2 and Section 3.3 — no numbers in them are invented, and none are measured by me.