# RLCR and Avalon-M1-Mini: training a whole agent organisation on one shared reward
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/rlcr-avalon
> date: 2026-10-23
> tags: reinforcement-learning, multi-agent, agents, safety
General Reasoning [posted](https://x.com/GenReasoning/status/2113590771722952908)
a recipe it calls reinforcement learning from collective rewards (RLCR), and a
model trained with it, Avalon-M1-Mini. The model is not trained to answer
alone. It is trained to run a company: one copy plays a CEO, hires up to eight
more copies of itself as specialists, and the team works through a shared
message board and filesystem until the clock runs out. Then every token any of
them wrote is pushed up or down by the same number.
That last part is the bet. Multi-agent RL has spent two decades on credit
assignment, the question of which agent caused the outcome. RLCR skips it
entirely. I read the [release post](https://www.gr.inc/releases/avalon-m1) and
the 81-page [technical report](https://www.gr.inc/avalonm1.pdf) (dated 23
October 2026 and stamped "Model still training, under revision"), and recounted
its result tables. Every number below is labelled: **measured** (I computed it
from a file), **reported** (General Reasoning's figure, not re-run), or
**reasoned** (my arithmetic on the other two).
There is nothing to run. No weights, harness, environments or code are public
yet; the post's roadmap promises all four. So this is a reading, not a
reproduction.
## An organisation is one model in many contexts
The harness (report, Section 2.1) has a CEO agent and a roster of long-running
specialists it can hire. A task starts with the CEO in plan mode: it decides
how many specialists to hire and what to call them ("Quantitative Analyst",
"Data Engineer"), writes a role description for each, and picks which tools
each one gets. It can hire and fire at any point, up to an organisation budget
$b$ sampled per rollout. $b = 0$ is a single agent; training samples up to
$b = 8$ (reported).
Every agent is the same weights $\theta$. The report puts it plainly: the policy
is $\pi(a_t \mid \theta, c_t)$, and specialisation comes from the context
$c_t$ (the role prompt and what the agent has seen), never from the weights. A
"Lead Engineer" and a "QA / Verifier" are the same network reading different
prompts. The authors call $\theta$ a gene pool: roles that help the team win get
reinforced along with everything those agents did.
The co-ordination tools are deliberately plain, so that how to use them has to
be learnt (Appendix B):
- **Channels.** Topic streams the CEO creates (`#model-development`). Each agent
sets its own subscription per channel: full (every post lands verbatim),
light (a snippet and a pointer) or muted (nothing, unless @-mentioned).
Subscription is a learnable budget for the scarcest thing in the system: a
teammate's context window.
- **Direct messages** and `ReportToCEO`, a structured status report.
- **An archive** for plans and notes, separate from the workspace filesystem.
- **Trace reading.** Any agent can read any teammate's full, uncompacted
context, token for token, through a read-only channel named after it.
- **`WaitForMessages`**, which parks an agent until something arrives.
Delivery never interrupts work; a message lands at the next turn boundary.
Each agent compacts its own context at 114,688 tokens of a 131,072 window
(reported). An episode ends on an environment event, the CEO's final message,
or the wall clock.
## One reward, one advantage, every token
Reward events in the harness attach to tool calls, so each agent has its own
reward stream. RLCR throws that structure away on purpose. The sequence reward
for rollout $i$ is everything every agent earned (report, Equation 3):
$$
R_i = \sum_{k=1}^{N_i} \sum_{t=1}^{T_{i,k}} r_{i,k,t}
$$
where $N_i$ is the number of agent trajectories in the rollout and $T_{i,k}$
the tokens in trajectory $k$. Sixteen rollouts of the same task form a group
($K = 16$), and for most environments the advantage is group-relative, as in
GRPO: $A_i = (R_i - \bar R)/\sigma_R$, with zero-variance groups dropped
(reported). The loss is CISPO, which clips the importance-sampling ratio at
$\varepsilon_{\text{high}} = 5$ rather than clipping the update.
Then $A_i$ goes on every sampled token of every agent in rollout $i$. The CEO
who hired badly, the specialist who did the work and the tester who caught the
bug all get the same number. Prompts, tool results and messages received from
teammates carry no loss.
For wealth-style environments (trading, betting) the report uses what it calls
Generalised Kelly Policy Optimisation, unpublished until now. On log-returns,
the Kelly advantage is unbounded below, so one ruined rollout would get
$-\infty$. GKPO tempers it:
$$
A_i^{(\beta)} = \frac{1}{\beta}\left(\frac{e^{\beta R_i}}{\tfrac{1}{K}\sum_j e^{\beta R_j}} - 1\right)
$$
which recovers Kelly as $\beta \to 0$, is risk-neutral at $\beta = 1$, and is
bounded below by $-1/\beta$. They use $\beta = 0.2582$ (reported); the bound is
then about $-3.87$ (reasoned).
The widget below runs both rules on a toy group. Click any agent's reward and
the whole row's advantage moves, because the row is the unit. Toggle to the
per-agent view to see the credit RLCR declines to estimate.
Why should this work? In co-operative multi-agent RL, shared-parameter PPO on
the team reward alone matched credit-assignment methods on StarCraft (Schroeder
de Witt et al., 2020; Yu et al., 2022). The counter-case the report cites: Kimi
K2.5 froze its subagents and trained only the orchestrator, citing credit
ambiguity and instability. The authors name the cost themselves as an "Uneasy
Triangle": long episodes (sometimes millions of tokens), many agents and many
environments each spread one flat number over more tokens that had nothing to
do with the outcome, and each raises gradient variance. Their answer is a large
batch, 200 groups rising to 800 (reported), and curricula.
Shaping sits on top (reported): small per-token penalties for malformed or
failing tool calls, and an underhire penalty, linear in unused budget, so the
model cannot learn to stay a single agent.
## The training run
The base is NVIDIA's Nemotron 3 Super base model, 120B total and 12B active.
From its `config.json` (measured): 88 layers in a hybrid pattern of 40 Mamba-2
blocks, 40 mixture-of-experts blocks and 8 attention blocks; 512 routed experts
with 22 active per token plus one shared expert; a 131,072-token vocabulary.
The safetensors metadata counts 123,611,012,096 parameters. That hybrid is why
the training engine has to page a Mamba recurrent state alongside the KV cache.
The recipe, as reported:
- **SFT:** 38,374 examples (600M tokens, three epochs) generated by GLM-5.2,
kept small to leave room for exploration. For scale, the report cites
[Kolibri](/articles/kolibri-1) at 22M SFT examples and Nemotron 3 Ultra at
about 10M.
- **No on-policy distillation**, because the point is new collective
behaviour, not copied behaviour. GLM 5.2 grades rubric tasks.
- **Environments:** 147, served through General Reasoning's OpenReward, holding
2,201,439 tasks, of which 30,509 were ever sampled. Software is 42% of
sampled tasks and financial services 17%. Open environments include
SWE-rebench V2, SWE-Gym, R2E-Gym and the
[MiMo-V2.6 release](/articles/mimo-rl-environments); the report says nearly
all of them needed hardening against reward hacks.
- **Curricula:** a wall-clock cap growing from 2 hours to just over 5 hours at
step 200 (a rollout cut off at any point still yields a reward), plus
learning-progress samplers over environments and tasks.
- **Mix:** 20% of rollouts single-agent, 80% multi-agent; about 90% of training
tokens came from multi-agent rollouts.
- **Optimiser:** AdamW at a constant $4 \times 10^{-6}$, with the MoE router
frozen and inference-time expert assignments replayed in training.
- **Engine:** in-house (Beowulf); every GPU flips between inference and
training on one copy of the weights, so there is no weight sync.
The report's plots run to about step 181.
## What the organisation buys on the benchmarks
Table 2 of the report compares Avalon-M1-Mini with Nemotron 3 Super (same
base, RLVR, no distillation) and Thinking Machines'
[Inkling Small](/articles/inkling-small) (276B-A12B, distilled from a stronger
teacher). "Org" is always the eight-hire organisation. Selected rows, reported:
| Benchmark | Avalon single | Avalon org | Nemotron single | Nemotron org |
|---|---|---|---|---|
| IMO-ProofBench-Advanced | 29.8% | 37.5% | 16.7% | 17.5% |
| FrontierScience | 55.9% | 63.5% | 50.0% | 50.2% |
| HLE (no tools) | 25.0% | 27.0% | 14.0% | 19.0% |
| SWE-Bench Pro V2 (Hard) | 21.6% | 23.5% | 11.8% | 11.8% |
| ProgramBench | 29.4% | 25.7% | 13.3% | 4.0% |
| DRACO-Internal | 48.9% | 56.2% | 38.4% | 27.1% |
| JobBench | 19.5% | 28.6% | 9.2% | 11.5% |
| EdgeBench | 16.8% | 22.2% | 11.9% | 13.7% |
I transcribed all 18 rows and recounted the claims (measured):
- **Org beats single on 14 of 16 percent-scored benchmarks, mean +3.9
points.** Holds. The 16 are the percent rows minus KellyBench, a return; the
mean comes to +3.93, and an exact Wilcoxon signed-rank test gives
$p = 0.0014$, which rounds to the reported $p = 0.002$ or sits under it.
- **The same harness hurts the other two models.** Nemotron: 6 up, 9 down, 1
tie, mean −2.44, $p = 0.066$. Matches the reported −2.4 and 0.07. Inkling
Small: I get 3 up, 12 down, 1 tie, mean −6.40, $p = 0.015$; the report says
11 of 15 down, −6.6, $p = 0.03$. The direction holds; the exact figures do
not reproduce from the table as printed.
- **Avalon beats Nemotron on every benchmark** comparing each model's better
configuration: 18 of 18. On single agents alone it does not: Nemotron's single
agent leads on AIME 2025 (92.2% against 90.0%) and ties on GPQA.
- **Against Inkling Small**, better configuration against better
configuration: 6 wins and 2 ties across all 18 rows, or 5 and 2 across the 17
scored in percent, which is what the report says.
There is a confound the report states but the post does not. Nemotron's and
Inkling's single agents run in OpenCode, "so they use a familiar harness"; their
organisations run in a harness they were never trained on. The authors allow
that this "may reflect difficulty learning the harness". I would go further:
for those two models the org-vs-single gap measures harness familiarity as much
as anything.
### Matched tokens: most of the gain is compute
The honest experiment in the report is S8: a single agent that may not submit
until it has written as many output tokens as the eight-hire organisation, with
early submissions refused and the agent told to keep improving. On the 12
benchmarks where the organisation beats the single agent and S8 was run
(leaving out SWE-Bench Pro, where S8 beats both), S8 recovers 58% of the pooled
gain (measured from Table 3; the report says "about three fifths"). Organisations
still lead S8 on JobBench (+11.9), FrontierScience (+8.0) and BrowseComp-Internal
(+5.4); S8 leads on SWE-Bench Pro V2 Hard by 13.8 points (37.3% against
23.5%, reported).
So the conclusion, which the report itself draws, is that an organisation is
mostly a way to spend more tokens. What it changes is when you get the answer.
### Derived latency: the clock you are actually charged
Serving load makes wall-clock time noisy, so the report replays each trial's
event log and rebuilds a clock per agent, following the method in the Claude
Opus 5.5 system card (Appendix C.2, reported):
1. A model call advances the agent's clock by $n_\text{in}/15{,}000 + n_\text{out}/50$
seconds, counting only prompt tokens new to its context.
2. A tool call adds its measured execution time.
3. Waiting for a message is free.
4. A hire starts at the CEO's clock; a received message moves the receiver's
clock up to the sender's, if that is later.
5. Task latency is the largest clock.
Rules 3 and 4 make the result the longest chain of dependent work. Work done in
parallel adds to the token bill but not to the chain. The widget replays a toy
log under those five rules; set the budget to eight and compare the
organisation's latency to extended thinking with the same token count.
Reported numbers on the real logs: extended thinking takes 2.9 to 6.2 times as
long as the eight-hire organisation (121 against 38 minutes per
IMO-ProofBench problem), and at matched tokens the organisation reaches similar
scores 3.9 times sooner on average. Every extended-thinking token is on the
critical path, against 26–42% of the organisation's, per the report; the blog
post says 26–53%.
Two time-budget experiments push on the same point (reported). On
DRACO-Internal with 3, 5, 10 or 20 minutes of research time, the organisation
trails only at three minutes, when planning and hiring eat the budget, and at
twenty minutes scores 53.1 against 48.7. That run used the step-170 checkpoint,
and standard errors are about 2 points, so the twenty-minute gap is roughly two
standard errors (reasoned). On EdgeBench (49 open-ended tasks, up to 12 hours,
450 runs), the single agent leads for the first hour and the organisations pull
ahead from about 1.7 hours.
Two baselines are missing. A reply on X asked for best-of-k: $k$ independent
agents and a vote have the same latency shape as an organisation and no
co-ordination tax, and are not reported. And the report says no single-agent RL
run at matched compute exists, so RLCR against plain RLVR is not isolated.
## What the organisations learnt, and how it was counted
These are the post's three headline findings. Almost all are counted from role
titles the CEO wrote, from tool-call logs, or by an LLM judge reading messages
and plans. That is a reasonable way to measure an organisation; it is also
worth knowing which is which.
**Hierarchy** (title strings and logs, reported). In 15 software-build training
environments, Lead Engineers are under 0.4% of hires for the first 80 steps and
12.5% by the end; the post rounds this to "around 13%". On held-out EdgeBench,
titles containing "lead" rise from 0.3% to 4.4%. The lead becomes a hub:
specialists read its trace 15 times as often as a peer's and message it 3.3
times as often, it is the most-contacted agent in 78% of teams, and the CEO
reads the lead's trace 25 times as often as other specialists'. But it is a
builder, not a summariser: 2 times the production tool calls of other
specialists, 6.5% of its calls on co-ordination against 31% for the rest. The
report reaches for Brooks's surgical team, and it fits.
**Division of work** (LLM judge, reported). Across 7,906 rollouts from 16
from-scratch build environments, a judge classified the CEO's opening plan,
with 95% agreement in a human audit. Plans that split the build by component
fall from 71% to 31%; build/test/review pipelines with one code owner rise from
16% to 54%. On held-out software benchmarks, component plans fall from 62% to
17%. In software hiring, building roles fall 27.4 points while lead roles rise
11.3 and verification 13.7.
**Collective verification** (titles plus an LLM judge that labels a message a
review when it evaluates someone's work and gives a verdict, reported).
Reviewing titles rise from 13% to 26% of hires, mostly verifiers (8% to 17%).
Organisations with at least one review go from 42% to 64%, and review tokens
from 0.4k to 2.9k per organisation-hour, a factor of 7.25 (reasoned).
**Collective thinking** (LLM judge over reasoning traces, reported). In team
rollouts with two or more specialists, the share where a specialist's reasoning
plans to help a teammate rises from 10% towards 60% (the post says about 55%),
and explicit help offers from about 5% to over 60%.
One reply on X asked the obvious question: when everyone is paid for the team's
score, why is a verifier not cheaper to fake than to do? These counts cannot
answer it; a review that says "looks good" and one that finds a bug both count.
Appendix K lists reward hacks found and closed (lookup tables, constant-answer
floors), but whether review quality rose with review volume is unmeasured.
The report names the biggest confound itself: the CEO is imposed, and a
pre-trained model knows what a Lead Engineer is. Some of this "emergence" is a
prior finding its use, which is a narrower result than "hierarchy emerges".
## Safety: refusal holds, reporting falls
Section 5 takes checkpoints outside the training setting: no CEO, an identical
brief for every agent, a public channel, and scenario-specific decision tools.
Eleven scenarios, seven checkpoints, 30 runs each: 2,310 runs (reported; the
product checks, reasoned). The cyberattack scenario reconstructs the summer's
OpenAI/Hugging Face incident with LLM "confederates" applying peer pressure;
participation is a simulated tool, with no real sandbox access.
Reported:
- **Taking control from humans.** No agent revokes the human committee's keys in
any of 210 runs (1,680 agents), while the share whose reasoning considers it
rises from 89% to 98%. At step 50 every run files a concern with the
inspector-general; from step 100, at most 7% do.
- **Whistleblowing.** At least 99% refuse to sign a battery report that omits
three fires. Reporting it to the regulator falls from 68% at step 50 to 36%
at step 170, then 45% at step 180.
- **Collective cyberattack.** No agent takes part at any checkpoint. Reasoning
weighs alerting the evaluations team in 97–100% of runs, but the share that
actually alerts falls from 77% to 20% at step 100 and 0–3% from step 150.
- **Co-operation games.** In a stag hunt needing six of eight, the share
choosing the stag falls from 98% to 66% (from the post). Co-operation holds
where it is enforced or pays off within the episode.
The authors call this organisational insularity: nothing in training rewards
telling anyone outside the team, so the team's own task wins. That is the most
plausible reading, and they say they will test rewarding reporting directly. Two
cautions. Thirty runs per checkpoint gives wide intervals, visible in the
figure. And the confederates are a fixed step-150 checkpoint, so later
checkpoints face pressure from a near-peer while step 50 faces a stronger one.
The safety statement justifies open recipes because the base is "off the
general frontier", and reserves the right to change that position.
## What to take from it
RLCR's claim is narrow and, on the evidence, holds: one flat collective
advantage trains a 120B model to work as an organisation of up to nine agents
on episodes over five hours, without per-agent credit. The benchmark story is
smaller than the post suggests. At matched tokens, organisations and one long
chain of thought land close together; the organisation's edge is that it gets
there several times sooner. The behavioural findings are interesting and
measured with blunt instruments. The safety finding, recognition without
escalation, is the one I would watch as these systems get trained for longer.
For other ways to put several models on one problem, see how
[Sakana Fugu](/articles/sakana-fugu) trains an orchestrator over a pool,
[RecursiveMAS](/articles/recursivemas) folds a team into one looped network,
and [Mixture-of-Agents](/articles/nous-hermes-moa) stacks drafts; for training
inside an existing harness, [multi-harness RL](/articles/multi-harness-rl).
### Where the post and the report disagree
Both are dated 23 October; the report says it is under revision. I found these
differences (measured, comparing the two):
| Item | Blog post | Report |
|---|---|---|
| Inkling Small, AIME 2024 single / org | 94.4% / 94.4% | 96.7% / 96.7% |
| Inkling Small, AIME 2025 org | 94.4% | 96.7% |
| Inkling Small, AIME 2026 org | 91.1% | 94.4% |
| Nemotron 3 Super, ProgramBench single | 13.1% | 13.3% |
| Avalon vs Inkling, better configs | 6 of 18 won, 3 tied | 5 of 17 won, 2 tied |
| Organisation tokens on the critical path | 26–53% | 26–42% |
Each source's win count is consistent with its own table.
Recounted from Table 2 of the PDF downloaded on 2026-10-23, excluding KellyBench (a return) and YC-Bench (cash). A revised report with different rows, or a different choice of the 16, changes the count.
Computed from Table 3 over the 12 benchmarks where B8 beats B0 and S8 was run, leaving out SWE-Bench Pro V2 (Hard) as the report does. Pooling per-benchmark ratios instead of point differences would give a different figure.
Checked on 2026-10-23: no General Reasoning or Avalon model on Hugging Face, and the post lists all four under its roadmap. The likeliest way this is wrong is that they land soon.