RLCR and Avalon-M1-Mini: training a whole agent organisation on one shared reward

General Reasoning posted a recipe it calls reinforcement learning from collective rewards (RLCR), and a model trained with it, Avalon-M1-Mini. The model is not trained to answer alone. It is trained to run a company: one copy plays a CEO, hires up to eight more copies of itself as specialists, and the team works through a shared message board and filesystem until the clock runs out. Then every token any of them wrote is pushed up or down by the same number.

That last part is the bet. Multi-agent RL has spent two decades on credit assignment, the question of which agent caused the outcome. RLCR skips it entirely. I read the release post and the 81-page technical report (dated 23 October 2026 and stamped "Model still training, under revision"), and recounted its result tables. Every number below is labelled: measured (I computed it from a file), reported (General Reasoning's figure, not re-run), or reasoned (my arithmetic on the other two).

There is nothing to run. No weights, harness, environments or code are public yet; the post's roadmap promises all four. So this is a reading, not a reproduction.

An organisation is one model in many contexts

The harness (report, Section 2.1) has a CEO agent and a roster of long-running specialists it can hire. A task starts with the CEO in plan mode: it decides how many specialists to hire and what to call them ("Quantitative Analyst", "Data Engineer"), writes a role description for each, and picks which tools each one gets. It can hire and fire at any point, up to an organisation budget $b$ sampled per rollout. $b = 0$ is a single agent; training samples up to $b = 8$ (reported).

Every agent is the same weights $\theta$. The report puts it plainly: the policy is $\pi(a_t \mid \theta, c_t)$, and specialisation comes from the context $c_t$ (the role prompt and what the agent has seen), never from the weights. A "Lead Engineer" and a "QA / Verifier" are the same network reading different prompts. The authors call $\theta$ a gene pool: roles that help the team win get reinforced along with everything those agents did.

The co-ordination tools are deliberately plain, so that how to use them has to be learnt (Appendix B):

• Channels. Topic streams the CEO creates (#model-development). Each agent sets its own subscription per channel: full (every post lands verbatim), light (a snippet and a pointer) or muted (nothing, unless @-mentioned). Subscription is a learnable budget for the scarcest thing in the system: a teammate's context window.
• Direct messages and ReportToCEO, a structured status report.
• An archive for plans and notes, separate from the workspace filesystem.
• Trace reading. Any agent can read any teammate's full, uncompacted context, token for token, through a read-only channel named after it.
• WaitForMessages, which parks an agent until something arrives. Delivery never interrupts work; a message lands at the next turn boundary.

Each agent compacts its own context at 114,688 tokens of a 131,072 window (reported). An episode ends on an environment event, the CEO's final message, or the wall clock.

expand

One reward, one advantage, every token

Reward events in the harness attach to tool calls, so each agent has its own reward stream. RLCR throws that structure away on purpose. The sequence reward for rollout $i$ is everything every agent earned (report, Equation 3):

where $N_i$ is the number of agent trajectories in the rollout and $T_{i,k}$ the tokens in trajectory $k$. Sixteen rollouts of the same task form a group ($K = 16$), and for most environments the advantage is group-relative, as in GRPO: $A_i = (R_i - \bar R)/\sigma_R$, with zero-variance groups dropped (reported). The loss is CISPO, which clips the importance-sampling ratio at $\varepsilon_{\text{high}} = 5$ rather than clipping the update.

Then $A_i$ goes on every sampled token of every agent in rollout $i$. The CEO who hired badly, the specialist who did the work and the tester who caught the bug all get the same number. Prompts, tool results and messages received from teammates carry no loss.

For wealth-style environments (trading, betting) the report uses what it calls Generalised Kelly Policy Optimisation, unpublished until now. On log-returns, the Kelly advantage is unbounded below, so one ruined rollout would get $-\infty$. GKPO tempers it:

which recovers Kelly as $\beta \to 0$, is risk-neutral at $\beta = 1$, and is bounded below by $-1/\beta$. They use $\beta = 0.2582$ (reported); the bound is then about $-3.87$ (reasoned).

The widget below runs both rules on a toy group. Click any agent's reward and the whole row's advantage moves, because the row is the unit. Toggle to the per-agent view to see the credit RLCR declines to estimate.

Why should this work? In co-operative multi-agent RL, shared-parameter PPO on the team reward alone matched credit-assignment methods on StarCraft (Schroeder de Witt et al., 2020; Yu et al., 2022). The counter-case the report cites: Kimi K2.5 froze its subagents and trained only the orchestrator, citing credit ambiguity and instability. The authors name the cost themselves as an "Uneasy Triangle": long episodes (sometimes millions of tokens), many agents and many environments each spread one flat number over more tokens that had nothing to do with the outcome, and each raises gradient variance. Their answer is a large batch, 200 groups rising to 800 (reported), and curricula.

Shaping sits on top (reported): small per-token penalties for malformed or failing tool calls, and an underhire penalty, linear in unused budget, so the model cannot learn to stay a single agent.

The training run

The base is NVIDIA's Nemotron 3 Super base model, 120B total and 12B active.

From its config.json (measured): 88 layers in a hybrid pattern of 40 Mamba-2 blocks, 40 mixture-of-experts blocks and 8 attention blocks; 512 routed experts with 22 active per token plus one shared expert; a 131,072-token vocabulary. The safetensors metadata counts 123,611,012,096 parameters. That hybrid is why the training engine has to page a Mamba recurrent state alongside the KV cache.

The recipe, as reported:

• SFT: 38,374 examples (600M tokens, three epochs) generated by GLM-5.2, kept small to leave room for exploration. For scale, the report cites Kolibri at 22M SFT examples and Nemotron 3 Ultra at about 10M.
• No on-policy distillation, because the point is new collective behaviour, not copied behaviour. GLM 5.2 grades rubric tasks.
• Environments: 147, served through General Reasoning's OpenReward, holding 2,201,439 tasks, of which 30,509 were ever sampled. Software is 42% of sampled tasks and financial services 17%. Open environments include SWE-rebench V2, SWE-Gym, R2E-Gym and the MiMo-V2.6 release; the report says nearly all of them needed hardening against reward hacks.
• Curricula: a wall-clock cap growing from 2 hours to just over 5 hours at step 200 (a rollout cut off at any point still yields a reward), plus learning-progress samplers over environments and tasks.
• Mix: 20% of rollouts single-agent, 80% multi-agent; about 90% of training tokens came from multi-agent rollouts.
• Optimiser: AdamW at a constant $4 \times 10^{-6}$, with the MoE router frozen and inference-time expert assignments replayed in training.
• Engine: in-house (Beowulf); every GPU flips between inference and training on one copy of the weights, so there is no weight sync.

The report's plots run to about step 181.

What the organisation buys on the benchmarks

Table 2 of the report compares Avalon-M1-Mini with Nemotron 3 Super (same base, RLVR, no distillation) and Thinking Machines' Inkling Small (276B-A12B, distilled from a stronger teacher). "Org" is always the eight-hire organisation. Selected rows, reported:

I transcribed all 18 rows and recounted the claims (measured):

• Org beats single on 14 of 16 percent-scored benchmarks, mean +3.9 points. Holds. The 16 are the percent rows minus KellyBench, a return; the mean comes to +3.93, and an exact Wilcoxon signed-rank test gives $p = 0.0014$, which rounds to the reported $p = 0.002$ or sits under it.
• The same harness hurts the other two models. Nemotron: 6 up, 9 down, 1 tie, mean −2.44, $p = 0.066$. Matches the reported −2.4 and 0.07. Inkling Small: I get 3 up, 12 down, 1 tie, mean −6.40, $p = 0.015$; the report says 11 of 15 down, −6.6, $p = 0.03$. The direction holds; the exact figures do not reproduce from the table as printed.
• Avalon beats Nemotron on every benchmark comparing each model's better configuration: 18 of 18. On single agents alone it does not: Nemotron's single agent leads on AIME 2025 (92.2% against 90.0%) and ties on GPQA.
• Against Inkling Small, better configuration against better configuration: 6 wins and 2 ties across all 18 rows, or 5 and 2 across the 17 scored in percent, which is what the report says.

There is a confound the report states but the post does not. Nemotron's and Inkling's single agents run in OpenCode, "so they use a familiar harness"; their organisations run in a harness they were never trained on. The authors allow that this "may reflect difficulty learning the harness". I would go further: for those two models the org-vs-single gap measures harness familiarity as much as anything.

Matched tokens: most of the gain is compute

The honest experiment in the report is S8: a single agent that may not submit until it has written as many output tokens as the eight-hire organisation, with early submissions refused and the agent told to keep improving. On the 12 benchmarks where the organisation beats the single agent and S8 was run (leaving out SWE-Bench Pro, where S8 beats both), S8 recovers 58% of the pooled gain (measured from Table 3; the report says "about three fifths"). Organisations still lead S8 on JobBench (+11.9), FrontierScience (+8.0) and BrowseComp-Internal (+5.4); S8 leads on SWE-Bench Pro V2 Hard by 13.8 points (37.3% against 23.5%, reported).

So the conclusion, which the report itself draws, is that an organisation is mostly a way to spend more tokens. What it changes is when you get the answer.

Derived latency: the clock you are actually charged

Serving load makes wall-clock time noisy, so the report replays each trial's event log and rebuilds a clock per agent, following the method in the Claude Opus 5.5 system card (Appendix C.2, reported):

1. A model call advances the agent's clock by $n_\text{in}/15{,}000 + n_\text{out}/50$ seconds, counting only prompt tokens new to its context.
2. A tool call adds its measured execution time.
3. Waiting for a message is free.
4. A hire starts at the CEO's clock; a received message moves the receiver's clock up to the sender's, if that is later.
5. Task latency is the largest clock.

Rules 3 and 4 make the result the longest chain of dependent work. Work done in parallel adds to the token bill but not to the chain. The widget replays a toy log under those five rules; set the budget to eight and compare the organisation's latency to extended thinking with the same token count.

Reported numbers on the real logs: extended thinking takes 2.9 to 6.2 times as long as the eight-hire organisation (121 against 38 minutes per IMO-ProofBench problem), and at matched tokens the organisation reaches similar scores 3.9 times sooner on average. Every extended-thinking token is on the critical path, against 26–42% of the organisation's, per the report; the blog post says 26–53%.

expand

Two time-budget experiments push on the same point (reported). On DRACO-Internal with 3, 5, 10 or 20 minutes of research time, the organisation trails only at three minutes, when planning and hiring eat the budget, and at twenty minutes scores 53.1 against 48.7. That run used the step-170 checkpoint, and standard errors are about 2 points, so the twenty-minute gap is roughly two standard errors (reasoned). On EdgeBench (49 open-ended tasks, up to 12 hours, 450 runs), the single agent leads for the first hour and the organisations pull ahead from about 1.7 hours.

Two baselines are missing. A reply on X asked for best-of-k: $k$ independent agents and a vote have the same latency shape as an organisation and no co-ordination tax, and are not reported. And the report says no single-agent RL run at matched compute exists, so RLCR against plain RLVR is not isolated.

What the organisations learnt, and how it was counted

These are the post's three headline findings. Almost all are counted from role titles the CEO wrote, from tool-call logs, or by an LLM judge reading messages and plans. That is a reasonable way to measure an organisation; it is also worth knowing which is which.

Hierarchy (title strings and logs, reported). In 15 software-build training environments, Lead Engineers are under 0.4% of hires for the first 80 steps and 12.5% by the end; the post rounds this to "around 13%". On held-out EdgeBench, titles containing "lead" rise from 0.3% to 4.4%. The lead becomes a hub: specialists read its trace 15 times as often as a peer's and message it 3.3 times as often, it is the most-contacted agent in 78% of teams, and the CEO reads the lead's trace 25 times as often as other specialists'. But it is a builder, not a summariser: 2 times the production tool calls of other specialists, 6.5% of its calls on co-ordination against 31% for the rest. The report reaches for Brooks's surgical team, and it fits.

expand

Division of work (LLM judge, reported). Across 7,906 rollouts from 16 from-scratch build environments, a judge classified the CEO's opening plan, with 95% agreement in a human audit. Plans that split the build by component fall from 71% to 31%; build/test/review pipelines with one code owner rise from 16% to 54%. On held-out software benchmarks, component plans fall from 62% to 17%. In software hiring, building roles fall 27.4 points while lead roles rise 11.3 and verification 13.7.

Collective verification (titles plus an LLM judge that labels a message a review when it evaluates someone's work and gives a verdict, reported). Reviewing titles rise from 13% to 26% of hires, mostly verifiers (8% to 17%). Organisations with at least one review go from 42% to 64%, and review tokens from 0.4k to 2.9k per organisation-hour, a factor of 7.25 (reasoned).

Collective thinking (LLM judge over reasoning traces, reported). In team rollouts with two or more specialists, the share where a specialist's reasoning plans to help a teammate rises from 10% towards 60% (the post says about 55%), and explicit help offers from about 5% to over 60%.

One reply on X asked the obvious question: when everyone is paid for the team's score, why is a verifier not cheaper to fake than to do? These counts cannot answer it; a review that says "looks good" and one that finds a bug both count. Appendix K lists reward hacks found and closed (lookup tables, constant-answer floors), but whether review quality rose with review volume is unmeasured.

The report names the biggest confound itself: the CEO is imposed, and a pre-trained model knows what a Lead Engineer is. Some of this "emergence" is a prior finding its use, which is a narrower result than "hierarchy emerges".

Safety: refusal holds, reporting falls

Section 5 takes checkpoints outside the training setting: no CEO, an identical brief for every agent, a public channel, and scenario-specific decision tools. Eleven scenarios, seven checkpoints, 30 runs each: 2,310 runs (reported; the product checks, reasoned). The cyberattack scenario reconstructs the summer's OpenAI/Hugging Face incident with LLM "confederates" applying peer pressure; participation is a simulated tool, with no real sandbox access.

Reported:

• Taking control from humans. No agent revokes the human committee's keys in any of 210 runs (1,680 agents), while the share whose reasoning considers it rises from 89% to 98%. At step 50 every run files a concern with the inspector-general; from step 100, at most 7% do.
• Whistleblowing. At least 99% refuse to sign a battery report that omits three fires. Reporting it to the regulator falls from 68% at step 50 to 36% at step 170, then 45% at step 180.
• Collective cyberattack. No agent takes part at any checkpoint. Reasoning weighs alerting the evaluations team in 97–100% of runs, but the share that actually alerts falls from 77% to 20% at step 100 and 0–3% from step 150.
• Co-operation games. In a stag hunt needing six of eight, the share choosing the stag falls from 98% to 66% (from the post). Co-operation holds where it is enforced or pays off within the episode.

expand

The authors call this organisational insularity: nothing in training rewards telling anyone outside the team, so the team's own task wins. That is the most plausible reading, and they say they will test rewarding reporting directly. Two cautions. Thirty runs per checkpoint gives wide intervals, visible in the figure. And the confederates are a fixed step-150 checkpoint, so later checkpoints face pressure from a near-peer while step 50 faces a stronger one. The safety statement justifies open recipes because the base is "off the general frontier", and reserves the right to change that position.

What to take from it

RLCR's claim is narrow and, on the evidence, holds: one flat collective advantage trains a 120B model to work as an organisation of up to nine agents on episodes over five hours, without per-agent credit. The benchmark story is smaller than the post suggests. At matched tokens, organisations and one long chain of thought land close together; the organisation's edge is that it gets there several times sooner. The behavioural findings are interesting and measured with blunt instruments. The safety finding, recognition without escalation, is the one I would watch as these systems get trained for longer.

For other ways to put several models on one problem, see how Sakana Fugu trains an orchestrator over a pool, RecursiveMAS folds a team into one looped network, and Mixture-of-Agents stacks drafts; for training inside an existing harness, multi-harness RL.

Where the post and the report disagree

Both are dated 23 October; the report says it is under revision. I found these differences (measured, comparing the two):

Each source's win count is consistent with its own table.