# Limite 1B - Violetto: the throughput claim is in config.json, not the blog

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/limite-1b-violetto
> date: 2026-09-22
> tags: kv-cache, long-context, math, benchmarks, small-models, distillation, reinforcement-learning

Paradigma released [Limite 1B - Violetto](https://paradigma.inc/blog/limite-1b-violetto/)
yesterday, and it comes with two different taglines. The
[GitHub README](https://github.com/paradigma-inc/limite-violetto) says *"A model for
high-frequency mathematical intelligence."* The release blog says
*"high-throughput solutions of difficult mathematical problems"* and, in its meta description,
*"high-throughput mathematical reasoning."*

Neither page contains a tokens-per-second number, a latency figure, a batch size, a hardware
target, or the word "milliseconds". I grepped both. The serving instructions are a vLLM plugin
that requires `tensor_parallel_size=1`, and that is the whole of the deployment story.

So the interesting question is not whether the claim is true. It is whether the claim is
*attached to anything*. It is — just not in the prose. `config.json` is unusually explicit for a
model that had no technical report when I first wrote this, and one field in it does all the
throughput work.

<Callout type="note">
**Updated 7 October 2026.** Paradigma published the
[technical report](https://paradigma.inc/research/limite-1b-violetto/) on 6 October. It confirms
the config reading below and corrects one thing I got wrong about the headline chart. Everything
new is in [the report, two weeks later](#the-report-two-weeks-later): the architecture pieces the
config could not show, the full post-training recipe, what the released eval suite does and does
not cover, and a second look at the compute chart. Earlier paragraphs the report overturned are
marked where they stand.
</Callout>

<ModelCard repo="paradigma-inc/limite-1b-violetto" />

## The stack, from the config

<LayerStack />

1,035,253,888 parameters, bf16, 2.07 GB on disk. 48 blocks, hidden 1280, intermediate 3328,
10 query heads over 2 KV heads at head\_dim 128, Qwen's tokenizer (`bos 151643`, vocab padded
to 151,680), `tie_word_embeddings: true`. Nothing surprising so far.

Then the rest of the file, which reads like a NanoGPT speedrun changelog rather than a model
card:

```json
{
  "global_every": 4,
  "global_layers": [3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47],
  "global_nope": true,
  "sliding_window": 1024,
  "sliding_window_convention": "k >= q - sliding_window, inclusive of the query token",
  "ve_layers": [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46],
  "ve_dim": 128,
  "mudd": true, "mudd_layers": [24, 47], "mudd_taps": 3,
  "mudd_tap_idx": { "24": [0, 12, 24], "47": [0, 23, 47] },
  "attn_gate_applied": "per_head_before_o_proj",
  "qk_norm": "rms_pre_rope",
  "rms_norm_has_weight": false,
  "attention_softmax_scale": 0.1,
  "rope_base_global": 1024.0,
  "rope_frac": 0.5,
  "softcap_logits": { "kind": "sigmoid", "a": 23.0, "b": 5.0, "c": 7.5 }
}
```

Four of these are worth stopping on, because the blog's one sentence about architecture —
*"strongly inspired by recent advancements made by the community on pre-training speedrun
competitions"* — is the only explanation on offer, and the fields say more than the sentence.

**Value embeddings.** `model.value_embeds.weight` is a *second* embedding table, `151,680 × 256`,
38.8M parameters, injected at 16 of the 48 blocks. `ve_stored_heads: 2` and
`ve_head_slice: "first_num_key_value_heads"` mean it feeds the V path of the two KV heads
directly from the token id, before QK-norm. This is the speedrun's U-net value-embedding trick,
and it is 3.8% of the model spent on a lookup table that skips the residual stream entirely.
(The report adds the part the config hid: a per-token, per-head gate decides how much of the
lookup to mix in, read off just the first 12 channels of the residual stream.)

**MuDD.** Multi-way dynamic dense connections, at exactly two sites: block 24 taps the outputs
of blocks 0, 12 and 24, and block 47 taps 0, 23 and 47. The parameters are trivial —
`mudd.dense1` is `[32, 1280]`, `dense2` is `[48, 3, 32]` — so this is a routing structure, not a
capacity addition. Two deliberate long-range shortcuts at the half-way point and the exit.

**Norms with no weight.** `rms_norm_has_weight: false` throughout. Every RMSNorm is a pure
normalisation with no learnable gain. That is a speedrun-derived simplification and it removes a
hundred-odd thousand parameters, which is not the point; the point is that it removes a family of scale
degrees of freedom that usually have to be tuned. The report shows they did not vanish entirely:
each block scales its residual and its branch output with learned per-layer scalars, so the gain
moved from inside the norm to the residual add.

**A pinned softmax scale.** `attention_softmax_scale: 0.1`, not `1/sqrt(128) = 0.0884`. Combined
with `rope_base_global: 1024.0` — three to four orders of magnitude below the 10⁶–10⁷ that a
131K-context model normally uses — and `rope_frac: 0.5`, so only half the head dimension is
rotated at all, with the 12 global blocks using no positional encoding whatsoever. The
positional scheme here is: local blocks get short-range RoPE on half their channels, global
blocks get nothing and rely on the causal mask. It is the NoPE-plus-local-RoPE arrangement,
committed to harder than most.

None of these four costs anything at inference. The fifth field does.

## The only derivable throughput claim

<KvBudget />

36 of 48 blocks attend over 1,025 keys regardless of how long the sequence is. Only 12 grow.

The arithmetic is four numbers from `config.json` and nothing else. KV bytes per token per
block = 2 (K and V) × `num_key_value_heads` (2) × `head_dim` (128) × 2 bytes = **1024 B**. A
global block at 131,072 tokens holds 134 MB; a windowed block stops at 1.05 MB. So the full
131K cache is `12 × 134 MB + 36 × 1.05 MB` = **1.65 GB**, against **6.44 GB** if all 48 blocks
were global. A factor of **3.91**.

That is the mechanism behind "high-throughput", and it is worth saying plainly what kind of
claim it is. It is not a speed claim — a windowed block and a global block cost roughly the same
per token at short context. It is a *concurrency* claim: on one 80 GB accelerator with the 2.07
GB checkpoint resident and 4 GB set aside for activations and runtime, the 73.9
GB left over holds 44 simultaneous 131K-token sequences instead of 11. For a workload that is thousands of independent competition problems
run in parallel, which is exactly what a maths model is for, that is the whole game.

It is also the ceiling, not a measurement. Nobody has published a real one.

<RepoCard repo="paradigma-inc/limite-violetto" />

<Callout type="note">
Paradigma ships a custom vLLM plugin (`VLLM_PLUGINS=limite`) pinned to vLLM 0.26.0, Python
3.12, PyTorch 2.11 on CUDA 13.0, and it currently requires TP and PP of one. A plugin exists
because `model_type: "limite"` is not a Transformers architecture — value embeddings, MuDD and
a weightless-norm stack are not expressible in a stock config. That is a real engineering cost
of the speedrun parts list, paid at deployment.
</Callout>

## The maths, and the one column that behaves differently

The headline is 94.01% on AIME 2026 at an estimated 1.71 × 10²¹ training FLOPs.

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig2.png"
  alt="A dark scatter chart titled 'AIME 2026 performance vs. estimated training compute'. The x-axis is estimated training compute in 6ND FLOPs on a log scale from 10 to the 21 to beyond 10 to the 24; the y-axis is AIME 2026 percentage from 40 to 100. A large purple dot marks Limite 1B - Violetto at 94.01% and 1.71 times 10 to the 21 FLOPs, far to the left. A dashed horizontal line at that score runs right across the chart to models that match it: MiniCPM5 2B near 10 to the 22, VibeThinker 3B and Nemotron Cascade 2 near 10 to the 23, Nemotron 3 Super and Qwen3.5 4B near 10 to the 24, and Qwen3.5 9B beyond it. OLMo 3 7B Think sits at about 69% and OLMo 3 7B Instruct at about 44%."
  caption="The release's headline figure: AIME 2026 against estimated training compute, log x. The footnote is load-bearing — compute is 6ND with RL excluded, and the counted stages differ by model. (Paradigma, 'Limite 1B - Violetto', 2026-09-21.)"
/>

The 6ND estimate is checkable. `6 × 1.035e9 × 275e9 = 1.71e21`, so the x position implies about
275 billion training tokens — consistent with the blog's "less than 300B highly curated tokens",
and a genuine 1–3 orders of magnitude below the models on the same dashed line. Sample
efficiency is the real result here.

But read the axis footnote: **"RL excluded; counted training stages vary by model."** The blog
credits its competition-level maths to "a mix of synthetic data generation, curated SFT and RL
post-training", and the compute axis excludes the last of those for everyone.

When I first wrote this I called it a pre-training-efficiency chart wearing a capability headline.
The report shows that was wrong: Limite's point counts pre-training, the distillation stage and
the SFT stages, and leaves out only RL, which by my estimate is about 5% of the total. The
footnote is fine as far as it goes. What it does not mention, the model Limite was distilled
from, is the bigger gap, and [the re-audit below](#the-chart-again-with-the-recipe-in-hand) goes
through it.

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig1.png"
  alt="A dense evaluation table of 17 models with columns for release date, total parameters, AIME 2026, HMMT February 2026, APEX Shortlist, BeyondAIME, AIME 2025, HMMT February 2025 and ArXivMath May 2026. The top row, highlighted, is Limite-1B-Violetto at 1B with 94.01, 83.62, 50.80, 74.25, 90.21, 91.35 and 25.08. Larger models below include MUSE-Glimmer-30B, Gemma-4-31B-IT, NVIDIA-Nemotron-3-Super at 120B and GLM 5.2 at 753B, which scores 99.20, 92.50, 68.09 and 56.67. A footnote marks daggered results as sourced from model cards or MathArena rather than rerun."
  caption="The full evaluation table as published. Seven benchmarks, 17 models, and a ragged grid — several large models have a score in only two or three columns. (Paradigma, 'Limite 1B - Violetto', 2026-09-21.)"
/>

Rank Limite in its own table, column by column:

<BenchSplit />

Six of the seven columns are competition-format short-answer maths — AIME, HMMT, BeyondAIME, an
olympiad shortlist — and Limite is first, third, third, fourth, fifth and sixth on them, against
models up to 730 times its size. That is a real result and the BeyondAIME number in particular
(74.25 against [MUSE-Glimmer-30B](/articles/muse-glimmer)'s 70.00, which the blog names) is not
a rounding artefact.

The seventh column is **ArXivMath May 2026**, and Limite is 7th of 12 at 25.08 — behind
VibeThinker-3B (29.77), Qwen3.5-9B (31.64), Gemma-4-31B-IT (35.78) and a long way behind GLM 5.2
(56.67). It is also behind Qwen3.5-4B (26.33), a model it beats by four points on AIME 2026.

That inversion is the most informative thing in the table. On every benchmark whose format is
"produce a boxed integer", the 1B is top-six. On the one benchmark whose format is not, the
ordering reverts to something much closer to parameter count. Paradigma does not hide this —
the column is right there in their own table, and the blog is unusually frank about the model's
limits elsewhere, printing four worked examples of Violetto reinterpreting "What is
photosynthesis?" as a cell-division recurrence and "What causes the seasons?" as a calendar
arithmetic problem. A model described as *"as lightly instruction-tuned as possible, to
challenge the assumption that models need to be embedded in an assistant persona"* is going to
do that, and saying so with receipts is the right call.

It does mean the honest summary is narrower than "mathematical intelligence." This is a
competition-problem solver. [MiniCPM5-2B](/articles/minicpm5-2b) appears in the same table two
rows down, and the comparison holds up: 90.21 to Limite's 94.01 on AIME 2026, at twice the
parameters — but 21.80 to Limite's 25.08 on ArXivMath, a much smaller gap.

## What is actually released

Four repositories under `paradigma-inc`: `limite-1b-violetto` (the RL'd model),
`limite-1b-base`, `limite-1b-base-soup` and `limite-1b-value-model` — the training-time value
model, which is an unusual and useful thing to release. The serving code is Apache-2.0, as are
all four model repos: at launch the two base checkpoints carried no licence tag on the Hub, and
on 7 October both show `license:apache-2.0`. The base and Violetto configs are byte-identical
apart from the weights, so the post-training changed no architecture.

At launch the technical report was "to be released soon", and the RL recipe and the eval harness
were missing with it. The report has since filled in the recipe, and the eval harness now ships
as `limite-evals/` in the GitHub repo. Two things are still missing: the training data, and any
performance measurement at all. For a model whose name is "high-frequency", the last omission is
the one to fix first, and it is an afternoon's work: `vllm bench serve` against the plugin at a few
concurrencies, on a named card, with the prompt distribution the model is for.

## The report, two weeks later

Paradigma published the [technical report](https://paradigma.inc/research/limite-1b-violetto/)
on 6 October: 37 pages, eight authors, a PDF and a
[thread](https://x.com/paradigmainc/status/2107525673283953073) summarising it. I read it
looking for three things. Does it confirm what I inferred from the config? What does the recipe
look like once it is written down? And does the headline chart hold up now that every stage is
on the table?

Mostly yes, a lot, and less well than I expected, for a reason the chart's footnote never
mentions.

### What the config got right, and what it could not see

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig3.png"
  alt="Two-part architecture diagram. Left, one Transformer block: a MUDD mixer at the input taking x of layer l, x of floor l over 2, and x of 0; RMSNorm; a fused QKV projection; QK-Norm; partial RoPE on local layers only; causal attention with a 1024 window locally and full context globally; a value-embedding branch on layers where l is 1 mod 3 that adds a gated token embedding to v, gated from the first 12 channels; an XSA box after attention; a per-head sigmoid gate read from the first 128 channels; the output projection, RMSNorm and a SwiGLU MLP. Right, the 48-layer stack drawn as 12 groups of three local layers and one global layer, with MUDD sum nodes feeding layers 24 and 47 from the embedding and from layers 12 and 23."
  caption="The architecture as Paradigma draws it. The stack on the right is the 36-plus-12 pattern the config spells out; the block on the left shows the two pieces the config only names, Exclusive Self-Attention (XSA) and the per-head output gate. (Paradigma, Limite 1B - Violetto technical report, Figure 2.)"
/>

The layer pattern, the window, the positional scheme and the MUDD taps are exactly what the config
said: three local blocks then one global, window 1,024, partial RoPE on 64 of the 128 head
dimensions at base 1024, NoPE on the global blocks, mixing sites at 24 and 47 that read the
embedding, a mid-stack state and the current input. The report states the motive I had to infer:
"This hybrid pattern reduces KV-cache requirements while retaining regular access to the full
context." So the 3.91× arithmetic above is the design intent, not a side effect.

Two things the config named but I did not explain. `attn_gate_applied: per_head_before_o_proj` is
gated attention: each head's output is multiplied by a sigmoid gate computed from the first 128
channels of the normalised residual. And before that gate sits
[Exclusive Self-Attention](https://arxiv.org/abs/2603.09078), which subtracts a learned fraction
of each head's output along the direction of its own value vector. My reading of why: a token
attending heavily to itself mostly copies its own value back, which the residual stream already
carries, so removing that component pushes the head to bring in something else. Every one of these extras reads a
slice of the residual (12 channels for the value-embedding gate, 128 for the output gate) rather
than all 1,280, and the report says why: additions are kept "sparse in depth, width, or both" so
training throughput stays close to a plain Transformer.

The MUDD mixer is the one piece Paradigma clearly adapted rather than lifted. The original
[MUDDFormer](https://proceedings.mlr.press/v267/xiao25d.html) mixes every earlier layer into every
layer. Here a small MLP with a shared 32-wide first layer produces three weights per token, for
two mixtures (the attention input and the residual), at just two sites, initialised so the mixer
is the identity, and the report gives the trade it bought: "This sparse formulation
increases step time by only 2.3% while retaining most of the gain observed with the full MUDD
formulation", and it "performed comparably" to Hyper-Connections and Attention Residuals at their
scale. No ablation table backs either sentence, so that is a claim, but it is the right kind of
claim for a speedrun-flavoured design: a measured cost against a stated benefit.

Two facts change how I read the rest. The model is deliberately deep for its size, 48 layers at
width 1,280, on the hypothesis that hard maths "benefits from a larger number of sequential
operations." And pre-training ran at a context of only 8,192 tokens; 131K arrived during
post-training. That makes the positional choices look less like taste and more like a plan. A
local block's RoPE never sees a distance past 1,024, so nothing about it changes when the
sequence gets longer, and a NoPE global block has no frequencies to stretch. A model built this
way can be extended from 8K to 131K without touching its positional encoding, which is what
they did. The pinned 0.1 softmax scale is the one config field the report never mentions.

Then there is a sentence in the RL section that supports my reading of "high-throughput" better
than anything in the blog: "the 48-layer architecture also gives a relatively high per-token
latency despite its small parameter count." The conclusion repeats it: the deep architecture "can
limit decoding speed and the model has not been fully optimized for inference." Depth costs
latency, the window buys concurrency, and "high-throughput" means many problems at once, not any
one of them quickly. There is still no throughput number anywhere in the report.

### Pre-training: a speedrun optimiser at full length

The pre-training run is small and fast. 6 nodes of 8×H100, about 449k steps at a batch of 524k
tokens, which multiplies out to 235B tokens; the report says 236B in one place and 237B in the
abstract. Using the same 6ND rule as the chart, that is about 1.47 × 10²¹ FLOPs. At an assumed 40%
of the H100's dense bf16 peak, 48 GPUs get through that in under a day. The thread says the whole
project, "from the very first experiments", took six weeks, so the final run was a small slice of
the effort.

The optimiser is straight from the speedrun lineage the site has covered in
[Muon: orthogonalizing the update for hidden layers](/articles/muon-optimizer) and
[the nanoGPT speedrun frontier](/articles/nanogpt-speedrun-frontier).
[NorMuon](https://arxiv.org/abs/2510.05491) on every hidden matrix, which is Muon plus a per-neuron
second-moment rescale after orthogonalisation; Polar Express coefficients in place of
Newton–Schulz, five iterations; Nesterov momentum warmed from 0.85 to 0.95; and cautious weight
decay, which only decays a weight where the update is already pushing it towards zero, at 1.2
times the *squared* learning rate. Everything else (embeddings, the value-embedding table, the scalars,
gates, XSA coefficients, MUDD weights) goes to AdamW, which steps only every second iteration.
Peak learning rate for the matrices is 0.018, and the schedule is warmup, a long flat phase and a
linear decay over the last 20%. The report says the run "remained stable for approximately 450k
training steps, with no major events such as large loss spikes."

The data is maths-first by construction: 46% problem–solution pairs, 19% synthetic textbook-style
documents, 35% filtered web of which about 40% is synthetic rewrites. They filtered by sampling
each source and asking an LLM to judge error rates, re-deriving reference answers for the Q&A
part. The pool is 346B tokens, of which the run consumed 236B. Decontamination was lexical then
semantic, against every benchmark in the report.

After pre-training they "souped" the base: three short continuations on a 6B-token high-quality
mixture with different seeds and learning rates, averaged uniformly. On MATH-500 with a thinking
prompt the soup scores 80.2 against 71.6 to 74.6 for the three runs it averages and 59.2 for the
starting checkpoint. It is the first of four soups in the pipeline, which brings me to the
figure people reacted to.

### Post-training: distil, prefer, RL, patch, soup, repeat

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig4.png"
  alt="Flow diagram of post-training. Top row: Limite 1B Base, then Distillation off-policy at 65K, a soup bowl marked last 5 checkpoints, DPO, RL 1 at 65K, RL 2 at 65K. Bottom row: SFT 1 with traces to 131K, a soup bowl that also receives the RL 2 checkpoint, SFT 2 with traces to 131K, a second soup bowl that also receives the first soup, RL 3 with a critic at 131K, and finally Limite 1B Violetto."
  caption="The post-training pipeline, with a pixel-art soup bowl for every checkpoint average. This is the figure a widely shared reaction called 'this graph'. (Paradigma, Limite 1B - Violetto technical report, Figure 3.)"
/>

The reaction that went round,
["I was not emotionally ready for this graph"](https://x.com/lucrbvi/status/2107529749547581713),
was attached to this figure, not to the compute chart. I understand it. Nine boxes and three
bowls is a lot of plumbing for a 1B model, and every bowl is a weight average that someone decided
was better than either checkpoint going into it.

The first stage does most of the work. Distillation is teacher-forced and logit-level: for 5.5
million solutions written by [VibeThinker-3B](https://arxiv.org/abs/2606.16140), about 26B target
tokens, the student is trained against the teacher's top 32 log-probabilities at every position.
The filtering is careful. A trace is kept only if it reaches the gold answer; traces where the
teacher gives up are dropped by matching phrases like "given the time constraints" and "I'll just
go with"; rejected traces go to a judge that sees the problem without the gold answer, which
wins back tokens worth 18% of the corpus. The last five distillation checkpoints are averaged.

Then a short DPO pass, 140 updates of 128 pairs, on the maths subset of the OLMo team's
preference data, where the chosen answer is from Qwen3-32B and the rejected one from Qwen3-0.6B.
They average the log-ratio per token rather than summing it, so the preference signal does not
reward longer or shorter answers for their length.

The RL is where Paradigma has opinions. They run DPPO with learning rates of 1 to 2 × 10⁻⁵, ten to
twenty times the usual, and squeeze 7 to 19 optimiser updates out of every batch of rollouts. Instead
of PPO's ratio clipping, DPPO masks a token only when its probability under the current policy
has moved by more than 0.15 from the policy that generated it *and* the update would push it
further. Groups where all eight rollouts pass, or all fail, are discarded because they carry no
signal. Rollouts are collected as whole cohorts rather than streamed in as they finish, because
streaming would favour short answers and easy problems. The report is candid that this regime is
fragile: both of the first two runs climbed and then collapsed, and the shipped checkpoints are
the ones picked before the fall.

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig7.png"
  alt="Two line charts for the first DPPO run. Left: mean reward rising from about 45% at cohort 1 to about 69% near cohort 20 to 23, then falling steeply to about 37% by cohort 30, with a dashed line at step 213 near cohort 21. Right: token entropy falling from about 0.74 nats to about 0.48 nats by update 200, then climbing back to about 0.78 after update 250, with the dashed line at step 213."
  caption="The first RL run past the checkpoint they kept: reward peaks, then collapses as entropy rebounds. They shipped step 213. (Paradigma, Limite 1B - Violetto technical report, Figure 9.)"
/>

The first run is kept at update 213 of roughly 300; the second at 179, after its learning rate
was doubled partway through because reward was rising too slowly. The report counts 461 actor
updates across the three runs; the three kept checkpoints, 213, 179 and 59, add up to 451, so I
assume the other ten were thrown away. Either way it is a few hundred updates where large runs
take thousands. When reward stopped improving, they did not push RL harder.
They ran two short SFT stages on synthetic problems aimed at the areas where the checkpoint was
weakest, with solutions from Qwen3.8 models, and averaged each SFT result with the checkpoint
before it so the new skills did not wash out what RL had taught. These stages also stretched the
context from 65K to 131K. The final RL run starts from that soup and adds a critic.

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig6.png"
  alt="Three stacked line charts over 501 critic updates with a dashed divider at update 225. Top: within-problem AUC rising quickly to about 0.75 in the first phase and settling near 0.71 in the second. Middle: token explained variance rising from 0 to about 0.22 in the first phase, dropping to about 0.09 at the phase change and rising slowly to about 0.12. Bottom: training and validation value loss near 0.10 to 0.12 throughout."
  caption="Warming up the value model before the last RL run. A within-problem AUC near 0.7 means that, for the same problem, the critic ranks a passing rollout above a failing one about 70% of the time. (Paradigma, Limite 1B - Violetto technical report, Figure 8.)"
/>

The critic argument is the most interesting few pages of the report. With a 0/1 reward on the
final answer and no discounting, every token in a rollout gets the same group-relative advantage,
so a 40,000-token solution that went wrong at one step is pushed down uniformly. A value model,
initialised from the actor with a scalar head, can say *where* the expected reward dropped. They
warm it for 501 updates with the actor frozen, then train it twice per actor step during RL 3. The
GAE λ is set to 0.5 to the power 1/L, with L the rollout length, so the credit that reaches the
first token is the same half whether the solution is 10,000 tokens or 100,000. It is a neat fix
for a problem that only exists at these lengths. RL 3 improves the model "substantially within
59 actor updates", and the report attributes this to the critic together with the stronger SFT
starting point, "rather than to the critic alone." There is no critic-off run to separate them,
and Paradigma says so.

The RL problems come from two generators. An LLM writes a problem, a program that solves it and a
claimed answer, and the problem is kept only if re-running the program reproduces the answer. A
CPU-side procedural generator renders templated problem families with varied wording, tuned so
brute-force enumeration does not work. Both are filtered to problems the current checkpoint
solves about half the time, the band where a group of eight rollouts actually disagrees.

<Figure
  src="https://ai.thesatyajit.com/articles/limite-1b-violetto/fig5.png"
  alt="Flow diagram of the LLM-based problem generator. A subcategory c sampled with weight 1 minus p c, and a set of examples, curated seeds then measured generated problems, feed an LLM generator. It outputs a problem, a program and a claimed answer. Run the program: the problem is kept if program output equals the claimed answer, otherwise discarded. Kept problems have their pass rate measured on the current checkpoint, which updates the subcategory weights and feeds selected problems in the useful band back as examples."
  caption="The LLM-side problem generator: an answer only counts if the generating program reproduces it, and pass rates on the current checkpoint steer which areas get generated next. (Paradigma, Limite 1B - Violetto technical report, Figure 4.)"
/>

### Which stage bought what

The report gives a score for every benchmark after every stage, and an appendix gives the same
checkpoints evaluated again at a single context length. Those two tables do not tell the same
story, which is why the widget below shows both.

<StageLadder />

Table 5, the one in the body, evaluates the first four stages with a 61,440-token answer cap and
the last two with 126,976. Read that way, RL 1 looks like the big step: +3.96 on AIME 2026,
+13.03 on BeyondAIME. Appendix G re-runs the early checkpoints with the full 131K budget, and the
picture moves. The DPO checkpoint, given room to finish, already scores 91.25 on AIME 2026, against
83.54 under the short cap. Its answers were the longest of any stage, 32,800 tokens on average,
so the short cap was cutting it off. At the full budget RL 1 *lowers* AIME 2026 to 88.54 and RL 2
lowers it again to 86.98, while mean answer length falls from 32,800 to 18,376.

So the first two RL runs bought brevity. Brevity is worth having, because under the 61,440-token cap
they trained with, shorter is more often finished, and the cheaper rollouts made the rest of the
pipeline affordable. But from the DPO checkpoint at 131K to the final model, AIME 2026 rises 2.76
points. The large late gains on the hard columns (APEX Shortlist from 30.12 to 50.80, HMMT
February 2026 from 72.16 to 83.62) come from the SFT-and-soup step and the critic run, with answer
lengths growing back to 75,007 tokens on the APEX Shortlist. One caveat from the report itself:
the 65K and 131K evaluations "used different concurrency settings," so small differences between
the two tables are not all signal.

ArXivMath barely moves at all. From the DPO checkpoint at 131K (24.69) to the final model (25.08)
it gains 0.39 points, while every competition column gains between 2.76 and 16.29. The release
table had already shown Limite seventh of twelve there. The stage table shows that nothing in
post-training after DPO touched it. Paradigma's own introduction concedes the model "may struggle
with knowledge of the literature," and this is what that looks like in numbers.

### The chart again, with the recipe in hand

Appendix F finally pins the compute axis down. For Limite, 6ND "counts pretraining, offline
distillation and SFT, and excludes RL." That reconciles with what I backed out of the x-position
earlier: 1.71 × 10²¹ over 6 × 1.035B parameters is about 275B tokens, and 236B of pre-training
plus 26B of distillation targets leaves about 13B for the soup mixture and the two short SFTs.
The report never prints the sum, but the pieces fit, and they fit the thread's "less than 300B
training tokens (including post-training)."

How much does leaving RL out move the dot? The report gives enough to estimate it. Up to the
checkpoints they kept, RL 1 generated about 21 cohorts of 704 problems × 8 rollouts, RL 2 about 20
cohorts at its 9 updates per cohort, and RL 3 about 15 cohorts of 256 × 8: roughly 260,000
rollouts. At about 25,000 tokens each for the first two runs and 45,000 for the third (Table 6's
answer lengths at those stages) it comes to around 7B generated tokens. Generating, training on
every one of them, and a generous allowance for the critic comes to under 10²⁰ FLOPs, about 5% of
the plotted figure. Double it for long-context attention and it is still a tenth, which moves the
dot 0.04 of a decade on a log axis. I raised this as a falsifier when the chart came out; the
answer is that the RL exclusion is harmless.

The omission that is not harmless is the teacher. Distillation is where the base model's 16.88 on
AIME 2026 (few-shot with a thinking prompt and an 8K answer budget, so not strictly comparable)
becomes 81.67, and
those 26B target tokens were written by VibeThinker-3B. Limite's x-position charges it for
*reading* them, at 1B-parameter cost. It does not charge anything for the 3B model that had to
exist to write them, and that model sits on the same chart at about 10²³ FLOPs, with a footnote
saying only its "Coder training stage" was counted. In Table 7, VibeThinker-3B scores 93.85 on
AIME 2026 and Limite 94.01, each with a standard error of about half a point. To its credit, the
PDF version of the table bolds both, as its caption's overlap rule requires. On the chart, the
student sits nearly two orders of magnitude to the left of its own teacher at the same height, and the
picture reads as two independent ways of reaching 94%.

Other comparisons on the chart run the opposite way. VibeThinker-3B's point counts only one stage
and MiniCPM5-2B's only its 1T-token mid-training, so both are under-counted; the Qwen3.5 points
assume Qwen3's 36T tokens. Those choices flatter the competitors, not Limite. The scores are
mixed-source too: Qwen3.5 and Nemotron 3 Super from MathArena, the two NVIDIA Nemotron models from
NVIDIA, and Qwen3.5-4B's AIME 2026 is "IRT-estimated." Appendix F says "evaluation setups differ
across these sources." None of that is hidden, but none of it is on the chart.

My verdict: the sample-efficiency result is real. Distillation is a legitimate technique, and the
recipe after it plainly adds capability: Limite beats its teacher by 4.64 points on HMMT February
2026, 4.39 on the APEX Shortlist and 3.12 on HMMT 2025. A 1B reaching 94% on a 2026 competition is
remarkable whatever it cost. But "one to three orders of magnitude below the
models that match it" should read "below the models that match it, one of which taught it."

### The eval suite they released

The thread says "we also release the entire eval suite," and it is a serious piece of work: a
`limite-evals/` package in the GitHub repo that pins every dataset revision and every reference
model to a 40-character Hub commit, records template hashes and sampling settings in a manifest,
and refuses to compare runs whose manifests disagree. Every completion is scored on a ladder: the
official grader's exact rule, a strict boxed-answer rule, a lenient one and a permissive one, so
a format change cannot pass itself off as a capability change. It is the eval tooling I would want
from every release.

I read it for the three fairness questions that matter on this table. The first is token
budgets. The default completion budget is the model's own native context minus a
4,096-token prompt reserve (`cli.py`, `token_budget`). VibeThinker-3B and MiniCPM5-2B both declare
131,072 in their Hub configs, the same as Limite, so they got the same 126,976-token ceiling;
Qwen3.5-9B and Gemma-4-31B declare 262,144 and got more room, not less. Gemma, MUSE-Glimmer and Qwen3.6 are sampled at their own published temperature of
1.0; the rest at the harness default of 0.6, which is also Limite's. I found nothing in the
budgets that tilts the table towards Limite.

The second is statistical resolution. Every score is pass@1 averaged over 32 samples, and Table 10's
standard errors (0.54 for Limite on AIME 2026) measure sampling noise on a fixed problem set. They
do not measure the fact that AIME 2026 is 30 problems: 94.01 versus 93.85 is a twentieth of one
problem. Spread over a different 30 problems of the same difficulty, a 94% solver's score would
move by about four points. Read the top of the AIME columns as a tie, which is what the bolding
says.

The third is contamination. The report says pre-training data was decontaminated lexically and
semantically against every benchmark, and the same procedure was applied to all post-training
data. The corpus has a knowledge cutoff "around the final months of 2025," with "a small fraction"
of web text into early 2026. The evaluator's own comment on HMMT February 2026 says it was held
"after the training cutoff of every checkpoint currently under evaluation," which is true of the
pre-training corpus. It is less obviously true of post-training: the distillation teacher was
released in June 2026 and the SFT teachers in August, both after AIME 2026 and HMMT February 2026
were public. Their *solutions* went into Limite. Decontamination of the problems is the right
defence, and I have no evidence of a leak, but a teacher that has seen a competition can
distil its fluency on that competition's style even when the exact problems are filtered out.
ArXivMath, the freshest set, is where Limite does worst, which fits a model whose strength is the
competition format rather than leakage.

And one gap. The released suite, `math-extended`, contains AIME 2024, 2025 and 2026, MATH-500,
GSM8K, HMMT February 2025 and 2026, OlympiadBench, BeyondAIME and the APEX Shortlist. ArXivMath is
not in it. The report explains that MathArena's grader gave both false positives and false
negatives on ArXivMath, so Paradigma scored it with Math-Verify instead, and that scorer is not
what shipped. So the one headline column you cannot rerun with their tool is the one where Limite
trails. That cuts against them rather than for them, since it is their weakest number, but
"the entire eval suite" is six of the seven benchmarks. The suite also carries a caveat I liked:
four of the APEX Shortlist's 47 problems are AIME 2025 and HMMT 2025 problems, and the code
records that as a known double-count rather than trimming the set.

### What I would still ask for

The report answers most of what I asked for in September. What it does not give is still what
I asked for first: a serving measurement. The eval harness runs 320 concurrent requests per vLLM
replica by default, and a comment in its launcher notes that "the scarce resource is KV cache,
not parameter memory." It is the concurrency argument from the top of this article, written
by the people who built the model. Put a tokens-per-second number next to it.

The other thing I would want is a critic-off run of RL 3. The value-model machinery is the most
reusable idea in the report, the value checkpoint is released, and right now its contribution
and the SFT soup's are counted together.

<ChangeMyMind>
  <Falsifier claim="Neither the blog nor the README publishes any latency or throughput measurement.">
    A page on paradigma.inc, or a section of the technical report when it lands, carrying
    tokens/sec, time-to-first-token, or a concurrency sweep. My check is a grep of the rendered
    blog HTML and the GitHub README on 2026-09-22 for throughput, latency, tok/s, ms, batch and
    per second; the only hits are the marketing phrase itself. A number posted on X or Discord
    would also overturn it and I would not have seen it. *Update, 7 October:* the technical report
    has no throughput number either, and concedes the 48-layer depth gives "a relatively high
    per-token latency." The claim stands.
  </Falsifier>
  <Falsifier claim="The hybrid attention stack is the mechanism behind the throughput claim, worth 3.91x on the KV cache at 131K.">
    The factor is arithmetic on four config fields and is not in doubt. What would change my
    mind about it being *the* mechanism is a measured throughput number that does not improve
    with concurrency the way a KV-bound workload should — for instance if the custom plugin
    serialises something and the model is compute-bound at batch one, in which case the window
    is buying memory nobody is using. *Update, 7 October:* the report names KV-cache reduction as
    the purpose of the hybrid pattern, and the eval harness's launcher calls KV cache "the scarce
    resource". Still no measurement.
  </Falsifier>
  <Falsifier claim="Limite's wins are concentrated in competition-format benchmarks.">
    A second non-competition maths benchmark — MathArena's proof tasks, miniF2F, PutnamBench,
    or anything with a free-form answer — where the 1B places as highly as it does on AIME. One
    column is one column, and ArXivMath May 2026 is recent enough that I cannot rule out a
    contamination or harness quirk that depresses everyone's score unevenly. *Update, 7 October:*
    the report's stage tables strengthen this. At a 131K budget, every competition column gains
    2.76 to 16.29 points between the DPO checkpoint and the final model; ArXivMath gains 0.39.
  </Falsifier>
  <Falsifier claim="The compute-efficiency chart excludes the training stage that produces the headline capability.">
    The chart's own footnote says "RL excluded". If the technical report shows the RL stage is a
    small fraction of total FLOPs — plausible for a 1B — then including it would move Limite's
    point negligibly and the chart is fine as drawn. The report would settle this in one number.
    *Update, 7 October: overturned.* Limite's point includes distillation and SFT, and from the
    report's cohort counts I estimate RL at about 5% of the plotted compute. The chart's real
    omission is the distillation teacher, VibeThinker-3B, whose own training is not charged to
    Limite.
  </Falsifier>
  <Falsifier claim="The architecture is a NanoGPT-speedrun parts list rather than a novel design.">
    A technical report describing value embeddings, MuDD taps at 24 and 47, weightless norms and
    a 0.1 softmax scale as independently derived, or showing an ablation where the combination
    behaves differently from the sum of the published tricks. I am inferring lineage from field
    names and the blog's own one-line attribution to speedrun competitions; that is provenance,
    not a claim that nothing here is new. *Update, 7 October:* the report confirms the lineage
    ("inspired by modded nanogpt speedruns") and adds one adaptation of its own, the two-site MUDD
    mixer at 2.3% step-time cost, described without an ablation table.
  </Falsifier>
</ChangeMyMind>

## How I checked

The first version (22 September) read `config.json` and the safetensors header from the Hugging
Face repo, grepped the rendered blog and the GitHub README for throughput terms, and transcribed
the release table. The KV-cache figures are arithmetic on four config fields.

For the update I read the technical report PDF in full (37 pages, dated October 2026) and took its
figures from the PDF itself, cropped and flattened onto white. I read the author thread and the
quoted reaction through the fxtwitter mirror; the "graph" in that reaction is the report's
post-training pipeline figure. I shallow-cloned `paradigma-inc/limite-violetto` at commit
`7ddad34` and read the evaluator rather than running it: the suite list in `suites.py`, the token
budget in `cli.py`, the reference-model pins and decoding defaults in `reference_models.py`, the
concurrency default in `runner.py` and the launcher comment in `serve.py`. The native context
lengths of VibeThinker-3B, MiniCPM5-2B, Qwen3.5-9B and Gemma-4-31B come from their Hub configs, and
the base-checkpoint licence tags from the Hub API, both on 7 October.

Two numbers are my estimates rather than the report's. The RL compute assumes the cohort counts
read off Figures 9 to 11 at the kept checkpoints and answer lengths taken from Table 6; it ignores
rollouts generated after those checkpoints. The "under a day" for pre-training assumes 40% of
the H100's dense bf16 peak, which the report does not state. I did not run the model, the
evaluator or a serving benchmark.