# Limite 1B - Violetto: the throughput claim is in config.json, not the blog

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/limite-1b-violetto
> date: 2026-09-22
> tags: llm, architecture, kv-cache, long-context, math, benchmarks, small-models
Paradigma released [Limite 1B - Violetto](https://paradigma.inc/blog/limite-1b-violetto/)
yesterday, and it comes with two different taglines. The
[GitHub README](https://github.com/paradigma-inc/limite-violetto) says *"A model for
high-frequency mathematical intelligence."* The release blog says
*"high-throughput solutions of difficult mathematical problems"* and, in its meta description,
*"high-throughput mathematical reasoning."*

Neither page contains a tokens-per-second number, a latency figure, a batch size, a hardware
target, or the word "milliseconds". I grepped both. The serving instructions are a vLLM plugin
that requires `tensor_parallel_size=1`, and that is the whole of the deployment story.

So the interesting question is not whether the claim is true. It is whether the claim is
*attached to anything*. It is — just not in the prose. `config.json` is unusually explicit for a
model with no technical report yet, and one field in it does all the throughput work.

<ModelCard repo="paradigma-inc/limite-1b-violetto" />

## The stack, from the config

<LayerStack />

1,035,253,888 parameters, bf16, 2.07 GB on disk. 48 blocks, hidden 1280, intermediate 3328,
10 query heads over 2 KV heads at head\_dim 128, Qwen's tokenizer (`bos 151643`, vocab padded
to 151,680), `tie_word_embeddings: true`. Nothing surprising so far.

Then the rest of the file, which reads like a NanoGPT speedrun changelog rather than a model
card:

```json
{
  "global_every": 4,
  "global_layers": [3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43, 47],
  "global_nope": true,
  "sliding_window": 1024,
  "sliding_window_convention": "k >= q - sliding_window, inclusive of the query token",
  "ve_layers": [1, 4, 7, 10, 13, 16, 19, 22, 25, 28, 31, 34, 37, 40, 43, 46],
  "ve_dim": 128,
  "mudd": true, "mudd_layers": [24, 47], "mudd_taps": 3,
  "mudd_tap_idx": { "24": [0, 12, 24], "47": [0, 23, 47] },
  "attn_gate_applied": "per_head_before_o_proj",
  "qk_norm": "rms_pre_rope",
  "rms_norm_has_weight": false,
  "attention_softmax_scale": 0.1,
  "rope_base_global": 1024.0,
  "rope_frac": 0.5,
  "softcap_logits": { "kind": "sigmoid", "a": 23.0, "b": 5.0, "c": 7.5 }
}
```

Four of these are worth stopping on, because the blog's one sentence about architecture —
*"strongly inspired by recent advancements made by the community on pre-training speedrun
competitions"* — is the only explanation on offer, and the fields say more than the sentence.

**Value embeddings.** `model.value_embeds.weight` is a *second* embedding table, `151,680 × 256`,
38.8M parameters, injected at 16 of the 48 blocks. `ve_stored_heads: 2` and
`ve_head_slice: "first_num_key_value_heads"` mean it feeds the V path of the two KV heads
directly from the token id, before QK-norm. This is the speedrun's U-net value-embedding trick,
and it is 3.8% of the model spent on a lookup table that skips the residual stream entirely.

**MuDD.** Multi-way dynamic dense connections, at exactly two sites: block 24 taps the outputs
of blocks 0, 12 and 24, and block 47 taps 0, 23 and 47. The parameters are trivial —
`mudd.dense1` is `[32, 1280]`, `dense2` is `[48, 3, 32]` — so this is a routing structure, not a
capacity addition. Two deliberate long-range shortcuts at the half-way point and the exit.

**Norms with no weight.** `rms_norm_has_weight: false` throughout. Every RMSNorm is a pure
normalisation with no learnable gain. That is a speedrun-derived simplification and it removes a
hundred-odd thousand parameters, which is not the point; the point is that it removes a family of scale
degrees of freedom that usually have to be tuned.

**A pinned softmax scale.** `attention_softmax_scale: 0.1`, not `1/sqrt(128) = 0.0884`. Combined
with `rope_base_global: 1024.0` — three to four orders of magnitude below the 10⁶–10⁷ that a
131K-context model normally uses — and `rope_frac: 0.5`, so only half the head dimension is
rotated at all, with the 12 global blocks using no positional encoding whatsoever. The
positional scheme here is: local blocks get short-range RoPE on half their channels, global
blocks get nothing and rely on the causal mask. It is the NoPE-plus-local-RoPE arrangement,
committed to harder than most.

None of these four costs anything at inference. The fifth field does.

## The only derivable throughput claim

<KvBudget />

36 of 48 blocks attend over 1,025 keys regardless of how long the sequence is. Only 12 grow.

The arithmetic is four numbers from `config.json` and nothing else. KV bytes per token per
block = 2 (K and V) × `num_key_value_heads` (2) × `head_dim` (128) × 2 bytes = **1024 B**. A
global block at 131,072 tokens holds 134 MB; a windowed block stops at 1.05 MB. So the full
131K cache is `12 × 134 MB + 36 × 1.05 MB` = **1.65 GB**, against **6.44 GB** if all 48 blocks
were global. A factor of **3.91**.

That is the mechanism behind "high-throughput", and it is worth saying plainly what kind of
claim it is. It is not a speed claim — a windowed block and a global block cost roughly the same
per token at short context. It is a *concurrency* claim: on one 80 GB accelerator with the 2.07
GB checkpoint resident and 4 GB set aside for activations and runtime, the 73.9
GB left over holds 44 simultaneous 131K-token sequences instead of 11. For a workload that is thousands of independent competition problems
run in parallel, which is exactly what a maths model is for, that is the whole game.

It is also the ceiling, not a measurement. Nobody has published a real one.

<RepoCard repo="paradigma-inc/limite-violetto" />

<Callout type="note">
Paradigma ships a custom vLLM plugin (`VLLM_PLUGINS=limite`) pinned to vLLM 0.26.0, Python
3.12, PyTorch 2.11 on CUDA 13.0, and it currently requires TP and PP of one. A plugin exists
because `model_type: "limite"` is not a Transformers architecture — value embeddings, MuDD and
a weightless-norm stack are not expressible in a stock config. That is a real engineering cost
of the speedrun parts list, paid at deployment.
</Callout>

## The maths, and the one column that behaves differently

The headline is 94.01% on AIME 2026 at an estimated 1.71 × 10²¹ training FLOPs.

<Figure
  src="/articles/limite-1b-violetto/fig2.png"
  alt="A dark scatter chart titled 'AIME 2026 performance vs. estimated training compute'. The x-axis is estimated training compute in 6ND FLOPs on a log scale from 10 to the 21 to beyond 10 to the 24; the y-axis is AIME 2026 percentage from 40 to 100. A large purple dot marks Limite 1B - Violetto at 94.01% and 1.71 times 10 to the 21 FLOPs, far to the left. A dashed horizontal line at that score runs right across the chart to models that match it: MiniCPM5 2B near 10 to the 22, VibeThinker 3B and Nemotron Cascade 2 near 10 to the 23, Nemotron 3 Super and Qwen3.5 4B near 10 to the 24, and Qwen3.5 9B beyond it. OLMo 3 7B Think sits at about 69% and OLMo 3 7B Instruct at about 44%."
  caption="The release's headline figure: AIME 2026 against estimated pre-training compute, log x. The footnote is load-bearing — compute is 6ND with RL excluded, and the counted stages differ by model. (Paradigma, 'Limite 1B - Violetto', 2026-09-21.)"
/>

The 6ND estimate is checkable. `6 × 1.035e9 × 275e9 = 1.71e21`, so the x position implies about
275 billion training tokens — consistent with the blog's "less than 300B highly curated tokens",
and a genuine 1–3 orders of magnitude below the models on the same dashed line. Sample
efficiency is the real result here.

But read the axis footnote: **"RL excluded; counted training stages vary by model."** The blog
credits its competition-level maths to "a mix of synthetic data generation, curated SFT and RL
post-training", and the compute axis excludes the last of those for everyone. The chart is
therefore a pre-training-efficiency chart wearing a capability headline, and the gap it shows is
real but measured on the stage that is not the one doing the final work.

<Figure
  src="/articles/limite-1b-violetto/fig1.png"
  alt="A dense evaluation table of 17 models with columns for release date, total parameters, AIME 2026, HMMT February 2026, APEX Shortlist, BeyondAIME, AIME 2025, HMMT February 2025 and ArXivMath May 2026. The top row, highlighted, is Limite-1B-Violetto at 1B with 94.01, 83.62, 50.80, 74.25, 90.21, 91.35 and 25.08. Larger models below include MUSE-Glimmer-30B, Gemma-4-31B-IT, NVIDIA-Nemotron-3-Super at 120B and GLM 5.2 at 753B, which scores 99.20, 92.50, 68.09 and 56.67. A footnote marks daggered results as sourced from model cards or MathArena rather than rerun."
  caption="The full evaluation table as published. Seven benchmarks, 17 models, and a ragged grid — several large models have a score in only two or three columns. (Paradigma, 'Limite 1B - Violetto', 2026-09-21.)"
/>

Rank Limite in its own table, column by column:

<BenchSplit />

Six of the seven columns are competition-format short-answer maths — AIME, HMMT, BeyondAIME, an
olympiad shortlist — and Limite is first, third, third, fourth, fifth and sixth on them, against
models up to 730 times its size. That is a real result and the BeyondAIME number in particular
(74.25 against [MUSE-Glimmer-30B](/articles/muse-glimmer)'s 70.00, which the blog names) is not
a rounding artefact.

The seventh column is **ArXivMath May 2026**, and Limite is 7th of 12 at 25.08 — behind
VibeThinker-3B (29.77), Qwen3.5-9B (31.64), Gemma-4-31B-IT (35.78) and a long way behind GLM 5.2
(56.67). It is also behind Qwen3.5-4B (26.33), a model it beats by four points on AIME 2026.

That inversion is the most informative thing in the table. On every benchmark whose format is
"produce a boxed integer", the 1B is top-six. On the one benchmark whose format is not, the
ordering reverts to something much closer to parameter count. Paradigma does not hide this —
the column is right there in their own table, and the blog is unusually frank about the model's
limits elsewhere, printing four worked examples of Violetto reinterpreting "What is
photosynthesis?" as a cell-division recurrence and "What causes the seasons?" as a calendar
arithmetic problem. A model described as *"as lightly instruction-tuned as possible, to
challenge the assumption that models need to be embedded in an assistant persona"* is going to
do that, and saying so with receipts is the right call.

It does mean the honest summary is narrower than "mathematical intelligence." This is a
competition-problem solver. [MiniCPM5-2B](/articles/minicpm5-2b) appears in the same table two
rows down, and the comparison holds up: 90.21 to Limite's 94.01 on AIME 2026, at twice the
parameters — but 21.80 to Limite's 25.08 on ArXivMath, a much smaller gap.

## What is actually released

Four repositories under `paradigma-inc`: `limite-1b-violetto` (the RL'd model),
`limite-1b-base`, `limite-1b-base-soup` and `limite-1b-value-model` — the training-time value
model, which is an unusual and useful thing to release. The serving code is Apache-2.0, as are
the Violetto and value-model repos; the two base checkpoints carry no licence tag on the Hub at
the time of writing, which is presumably an oversight and is worth fixing before anyone builds
on them. The base and Violetto configs are byte-identical apart from the weights, so the
post-training changed no architecture.

What is not released: the technical report ("to be released soon"), the training data, the RL
recipe, the eval harness, and any performance measurement at all. For a model whose name is
"high-frequency", the last omission is the one to fix first, and it is an afternoon's work:
`vllm bench serve` against the plugin at a few concurrencies, on a named card, with the
prompt distribution the model is for.

<ChangeMyMind>
  <Falsifier claim="Neither the blog nor the README publishes any latency or throughput measurement.">
    A page on paradigma.inc, or a section of the technical report when it lands, carrying
    tokens/sec, time-to-first-token, or a concurrency sweep. My check is a grep of the rendered
    blog HTML and the GitHub README on 2026-09-22 for throughput, latency, tok/s, ms, batch and
    per second; the only hits are the marketing phrase itself. A number posted on X or Discord
    would also overturn it and I would not have seen it.
  </Falsifier>
  <Falsifier claim="The hybrid attention stack is the mechanism behind the throughput claim, worth 3.91x on the KV cache at 131K.">
    The factor is arithmetic on four config fields and is not in doubt. What would change my
    mind about it being *the* mechanism is a measured throughput number that does not improve
    with concurrency the way a KV-bound workload should — for instance if the custom plugin
    serialises something and the model is compute-bound at batch one, in which case the window
    is buying memory nobody is using.
  </Falsifier>
  <Falsifier claim="Limite's wins are concentrated in competition-format benchmarks.">
    A second non-competition maths benchmark — MathArena's proof tasks, miniF2F, PutnamBench,
    or anything with a free-form answer — where the 1B places as highly as it does on AIME. One
    column is one column, and ArXivMath May 2026 is recent enough that I cannot rule out a
    contamination or harness quirk that depresses everyone's score unevenly.
  </Falsifier>
  <Falsifier claim="The compute-efficiency chart excludes the training stage that produces the headline capability.">
    The chart's own footnote says "RL excluded". If the technical report shows the RL stage is a
    small fraction of total FLOPs — plausible for a 1B — then including it would move Limite's
    point negligibly and the chart is fine as drawn. The report would settle this in one number.
  </Falsifier>
  <Falsifier claim="The architecture is a NanoGPT-speedrun parts list rather than a novel design.">
    A technical report describing value embeddings, MuDD taps at 24 and 47, weightless norms and
    a 0.1 softmax scale as independently derived, or showing an ablation where the combination
    behaves differently from the sum of the published tricks. I am inferring lineage from field
    names and the blog's own one-line attribution to speedrun competitions; that is provenance,
    not a claim that nothing here is new.
  </Falsifier>
</ChangeMyMind>