2026-08-23 · 8 min · verifiers · rl · small-models · evaluation · explainer
Reasoning models improve by practising against a verifier — an automatic check that says whether an answer is right. Verifiers are everywhere in the current training stack and almost nobody asks how big they have to be, because the answer is assumed to be "as big as you can afford".
This project asks properly, and the answer is startling: on Countdown, 0.63 million parameters — a model that pretrains in about two minutes on one H100 — scores 0.85, within a few points of a 7B on the identical frozen test set.
The setup is worth the attention. Nineteen verifiers spanning almost five orders of magnitude, eleven of them pretrained from scratch specifically to probe below anything you can download. Three tasks chosen to get progressively harder to check. Every rung fine-tuned on the same data and scored on the same 1,200-example balanced test set, so chance is exactly 0.50 and the curves are genuinely comparable.
| Project | How small can a verifier be? · tw0zer0s |
| Ladder | 19 models, 0.07M–7B — 11 pretrained from scratch, 8 off-the-shelf (Monad, SmolLM2, Qwen2.5) |
| Tasks | Countdown and Maze (exactly checkable) · faithfulness judging (human labels, no checker) |
| Data | 12,000 balanced examples per task, 80/10/10, frozen shared test slice |
| Budget | 3.5B tokens, 30 GPU-hours total, on a single H100 |
| Floor | Countdown switches on at 0.63M, Maze at 1M, faithfulness at 1–2M |
| Headline | a 2M verifier scores 0.83 on faithfulness against zero-shot Gemini 2.5 Flash's 0.70 |
The three tasks
Read the three tells together and the switch-on ordering stops being arbitrary. It is not that Maze is a harder problem than Countdown — it is that Maze’s evidence sits at the far end of a hundred tokens of grid, and a model with a few hundred thousand parameters cannot carry a representation that far. Countdown’s check is local arithmetic on a short prompt, so it survives lowest.
Which suggests the floor is set by how far the evidence has to travel rather than by how hard the check is to perform — an architecture and context claim, not a reasoning one. That is a more tractable statement than “verification needs N parameters”, and it predicts that a task with a long prompt and a local tell should sit lower than its difficulty implies.
Flat, then on
Verification is flat, then on. Below a task-specific size the verifier sits at chance; above it, the curve jumps and then barely moves for three more orders of magnitude. Maze is the purest case — 0.928 at two million parameters, 0.932 at seven billion, a difference of four tenths of a point across a 3,500× size increase.
And flip to accuracy with Faithfulness on. The task with no exact checker, the one that supposedly needs judgement, produces the highest curve of the three — and every trained verifier from two million parameters up beats zero-shot Gemini 2.5 Flash on the same test set. Harder to solve is not the same as harder to check, though that particular result has a large asterisk that the project is admirably clear about.
The shape is the same on all three tasks and it is not a gentle scaling curve. Below a task-specific size the verifier sits at chance. Above it, the curve jumps and then stops moving. Maze is the cleanest case: 0.928 at two million parameters, 0.932 at seven billion. Four tenths of a point across a 3,500× increase in size.
Countdown is the honest exception, and the project's own framing is slightly generous here. It does climb — 0.83 at 2M to 0.934 at 3B — which is ten points, not a plateau. The right reading is not "size does nothing" but "size does an order of magnitude less than you would budget for": a model 5,000× smaller gets within ten points, and a model 100× smaller gets within seven.
The result I did not expect is the ordering. Faithfulness — the task with no exact checker, the one that supposedly needs judgement — produces the highest curve of the three. Every trained verifier from two million parameters up beats zero-shot Gemini 2.5 Flash on the same test set.
The bit below a million parameters
The switch-on points hide something more interesting than the switch.
The bottom panel is the finding I would take away from this whole project. A one-million-parameter model asked to reason first and then answer produces zero usable verdicts out of twelve hundred: it echoes the puzzle’s numbers correctly and then degenerates into pretraining babble. Remove the requirement to write a rationale, train on the single verdict token, and the same model on the same frozen test set reaches 0.826 and 0.934.
Chain of thought is usually framed as a capability you unlock. At this scale it is a format cost you impose, and the model pays for it with the only thing it was asked to produce. The discrimination was always there — the AUROC line says so three sizes earlier — it just could not survive being routed through a paragraph.
Two things are happening down there. The first is a measurement subtlety with a real lesson: accuracy reads the model's literal Final verdict: Yes/No, while AUROC reads the soft score P(Yes) / (P(Yes) + P(No)), which exists whether or not the model manages to write anything parseable. On Countdown there is a whole band — 0.15M to 0.34M — where AUROC climbs from 0.74 to 0.85 while accuracy sits at exactly chance. The model can rank correct answers above wrong ones and cannot say so.
The second is the finding I keep coming back to. At one million parameters under a chain-of-thought target, the model collapses: it echoes the puzzle's numbers correctly and then degenerates into pretraining babble, and not one of 1,200 outputs emits a parseable verdict. Train the same model on a verdict-only target and it reaches 0.826 on Countdown and 0.934 on Maze — matching the entire ladder above it.
Chain of thought is usually framed as something you unlock. At this scale it is a format cost you impose, and it destroys the only output that mattered. The discrimination was there all along; it could not survive being routed through a paragraph the model was too small to hold together.
The floors also differ by task, and the explanation is the one in the task diagram above: Maze sits at exactly chance across the whole sub-1M range with no latent signal at all because its tell lives at the far end of a hundred tokens of grid, while Countdown's local arithmetic on a short prompt carries signal down to 0.15M. Which makes the floor a statement about how far evidence has to travel, not about how hard the check is.
Why the plateau is flat
This is the part of the project I would point other people at, because it is the question most work of this shape does not ask.
A curve that stops moving admits two explanations. Either every model on the plateau fails a different slice of the data, with the error rates coinciding by accident, or they all fail the same examples. Those imply completely different things about whether to keep spending.
The flat plateau could have meant two very different things: every model failing a different slice, with the rates coinciding by accident, or every model failing the same examples. Counting how many of six plateau models get each item right settles it — joint failures run 4.6× to 13× above what independence predicts.
Which reframes the plateau entirely. It is not that bigger verifiers stop improving; it is that the residual difficulty lives in the data, not in the model. On Maze there is almost nothing left to win and a two-million-parameter verifier is genuinely finished. On faithfulness there are four points sitting in a contested quarter of the set, and the entire ladder from 1M to 7B walks past them — which is an argument for a different verifier, not a larger one.
It is the second, decisively. Taking six plateau models per task and counting how many get each of the 1,200 test items right: joint failures run 4.8× above chance on Countdown, 4.6× on faithfulness, and 13× on Maze. The plateau's height is set by the data's difficulty, not by each model rolling its own dice.
And the contested middle — the examples different models decide differently — varies enormously by task. Maze leaves 3% contested and a per-example oracle gains eleven thousandths over the best single model: nothing to win. Faithfulness leaves 26% contested and its oracle reaches 0.991 against 0.952, four points that the entire 1M-to-7B ladder walks straight past. That is an argument for a different verifier, not a bigger one.
The methodology, which is most of the value
A lot of the work here is in the dataset construction, and every piece of it closes a specific hole through which a verifier could score well without ever doing the check.
Toggle any of the four off and read what you would get instead. Each closes a specific hole through which a verifier can score well without ever doing the check — and none of them is a step you would take unless you had thought about how the measurement could lie to you.
This is where most of the project’s value sits, and it is why the headline is worth taking seriously. The claim is not “a small model gets 0.85”; a small model can get 0.85 on a badly built set trivially. The claim is 0.85 on a balanced set whose negatives are one edit from correct and whose positives include problems the generator could not solve, on a frozen slice shared with every larger model in the comparison.
One more safeguard that does not fit the diagram: LoRA was run alongside full fine-tuning as a cross-check, agreeing within 1–4 points wherever both were run, so the size trend does not depend on how the verifier was adapted.
The eleven from-scratch models copy Monad's architecture and tokenizer — Llama-style, 8,192-token vocabulary, tied embeddings, head dim 64, 3× MLP — pretrained on PleIAs/SYNTH only. The 10M model trains in about ten minutes on one H100; the 630K one in about two.
Where I would push back
The project caught two shortcuts and says so; the question is how many it didn't. On Maze, model-generated wrong path lengths are longer than correct ones (≈15.2 steps against ≈6.6), so the rule "answer ≤ 10 → correct" already scores well. On faithfulness, the verifiers learn HaluEval's corruption style. Both are named in the limitations. But the natural conclusion is that a verifier scoring 0.93 on a task where a one-line heuristic scores well is not evidence that verification is cheap — it is evidence that this instance of the task is. The plateau might be the difficulty ceiling of the proxy, not of the check.
Monad-56M is the strangest point on every curve and goes unremarked. It scores below the 2M–10M from-scratch models on all three tasks — 0.775 on Countdown against 0.828, 0.878 on Maze against 0.928, 0.818 on faithfulness against 0.873. A 56M model pretrained on 200B tokens losing to a 5M model pretrained on 250M tokens is either a fine-tuning artifact or a genuinely interesting statement about what general pretraining does to a narrow classifier, and it deserves a sentence either way.
The Qwen2.5-7B faithfulness row is byte-identical to SmolLM2-135M's — same accuracy, same CI, same AUROC — and sits well below the 1.5B and 3B rows around it. That looks like a transcription error in the appendix rather than a finding, and it is the one number I would not build on.
"Small verifier" and "cheap verification" are not the same claim. The motivation is that running a verifier over training-scale corpora is a throughput bottleneck. But the models on the plateau are being fine-tuned per task on 9,600 labelled examples, which for the checkable tasks requires a generator and an exact checker — and if you have an exact checker, you do not need a verifier. The load-bearing case is faithfulness, where there is no checker, and that is precisely the case where the result is an in-distribution classifier.
What it changes
The practical takeaway is not "use a 2M verifier". It is that verifier size should be a measured parameter rather than an assumed one, and that measuring it is cheap — this entire study is 3.5B tokens and 30 GPU-hours.
If you are verifying at training scale, or filtering reasoning branches at inference time, or running a fleet of agents that each need their outputs checked, the difference between a 7B judge and a 5M one is the difference between verification being the bottleneck and being free. The project's own next step is the right one: let a model optimize against a small verifier and see whether it learns to fool it, and whether smaller verifiers get fooled sooner. Everything above says small verifiers are accurate. Nothing above says they are robust to something trying to break them.