# How small can a verifier be? Six hundred thousand parameters
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/verifier-frontier
> date: 2026-08-23
> tags: verifiers, rl, small-models, evaluation, explainer
Reasoning models improve by practising against a verifier — an automatic check that says whether an answer is right. Verifiers are everywhere in the current training stack and almost nobody asks how big they have to be, because the answer is assumed to be "as big as you can afford".
[This project](https://www.twozeros.eu/projects/verifier-frontier) asks properly, and the answer is startling: on Countdown, **0.63 million parameters** — a model that pretrains in about two minutes on one H100 — scores 0.85, within a few points of a 7B on the identical frozen test set.
The setup is worth the attention. Nineteen verifiers spanning almost five orders of magnitude, eleven of them pretrained from scratch specifically to probe below anything you can download. Three tasks chosen to get progressively harder to check. Every rung fine-tuned on the same data and scored on the same 1,200-example balanced test set, so chance is exactly 0.50 and the curves are genuinely comparable.
| | |
|---|---|
| Project | [How small can a verifier be?](https://www.twozeros.eu/projects/verifier-frontier) · tw0zer0s |
| Ladder | **19 models**, 0.07M–7B — 11 pretrained from scratch, 8 off-the-shelf (Monad, SmolLM2, Qwen2.5) |
| Tasks | Countdown and Maze (exactly checkable) · faithfulness judging (human labels, no checker) |
| Data | 12,000 balanced examples per task, 80/10/10, frozen shared test slice |
| Budget | **3.5B tokens, 30 GPU-hours** total, on a single H100 |
| Floor | Countdown switches on at **0.63M**, Maze at **1M**, faithfulness at **1–2M** |
| Headline | a **2M** verifier scores 0.83 on faithfulness against zero-shot Gemini 2.5 Flash's 0.70 |
## The three tasks
## Flat, then on
The shape is the same on all three tasks and it is not a gentle scaling curve. Below a task-specific size the verifier sits at chance. Above it, the curve jumps and then **stops moving**. Maze is the cleanest case: 0.928 at two million parameters, 0.932 at seven billion. Four tenths of a point across a 3,500× increase in size.
Countdown is the honest exception, and the project's own framing is slightly generous here. It does climb — 0.83 at 2M to 0.934 at 3B — which is ten points, not a plateau. The right reading is not "size does nothing" but "size does an order of magnitude less than you would budget for": a model 5,000× smaller gets within ten points, and a model 100× smaller gets within seven.
The result I did not expect is the ordering. **Faithfulness — the task with no exact checker, the one that supposedly needs judgement — produces the highest curve of the three.** Every trained verifier from two million parameters up beats zero-shot Gemini 2.5 Flash on the same test set.
That headline needs its asterisk, and the project supplies it without being asked. The trained verifiers see HaluEval's training split, so they learn *that dataset's hallucination signatures* — the stylistic tells of a synthetically corrupted answer: over-specific names, added detail, subtle contradiction. Gemini judges cold against a stricter and more general notion of "supported". So the comparison measures in-distribution fine-tuning against zero-shot transfer, and in-distribution wins decisively. That is a real and useful result. It is not "a 2M model is a better faithfulness judge than Gemini".
## The bit below a million parameters
The switch-on points hide something more interesting than the switch.
Two things are happening down there. The first is a measurement subtlety with a real lesson: accuracy reads the model's literal `Final verdict: Yes/No`, while AUROC reads the soft score `P(Yes) / (P(Yes) + P(No))`, which exists whether or not the model manages to *write* anything parseable. On Countdown there is a whole band — 0.15M to 0.34M — where AUROC climbs from 0.74 to 0.85 while accuracy sits at exactly chance. The model can rank correct answers above wrong ones and cannot say so.
The second is the finding I keep coming back to. At one million parameters under a chain-of-thought target, the model **collapses**: it echoes the puzzle's numbers correctly and then degenerates into pretraining babble, and not one of 1,200 outputs emits a parseable verdict. Train the *same model* on a verdict-only target and it reaches 0.826 on Countdown and 0.934 on Maze — matching the entire ladder above it.
Chain of thought is usually framed as something you unlock. At this scale it is a format cost you impose, and it destroys the only output that mattered. The discrimination was there all along; it could not survive being routed through a paragraph the model was too small to hold together.
The floors also differ by task, and the explanation is the one in the task diagram above: Maze sits at exactly chance across the whole sub-1M range with no latent signal at all because its tell lives at the far end of a hundred tokens of grid, while Countdown's local arithmetic on a short prompt carries signal down to 0.15M. Which makes the floor a statement about how far evidence has to travel, not about how hard the check is.
## Why the plateau is flat
This is the part of the project I would point other people at, because it is the question most work of this shape does not ask.
A curve that stops moving admits two explanations. Either every model on the plateau fails a *different* slice of the data, with the error rates coinciding by accident, or they all fail the *same* examples. Those imply completely different things about whether to keep spending.
It is the second, decisively. Taking six plateau models per task and counting how many get each of the 1,200 test items right: joint failures run **4.8× above chance on Countdown, 4.6× on faithfulness, and 13× on Maze**. The plateau's height is set by the data's difficulty, not by each model rolling its own dice.
And the contested middle — the examples different models decide differently — varies enormously by task. Maze leaves 3% contested and a per-example oracle gains eleven thousandths over the best single model: nothing to win. Faithfulness leaves 26% contested and its oracle reaches 0.991 against 0.952, four points that the entire 1M-to-7B ladder walks straight past. That is an argument for a *different* verifier, not a bigger one.
## The methodology, which is most of the value
A lot of the work here is in the dataset construction, and every piece of it closes a specific hole through which a verifier could score well without ever doing the check.
One more safeguard that does not fit the diagram: **LoRA was run alongside full fine-tuning** as a cross-check, agreeing within 1–4 points wherever both were run, so the size trend does not depend on how the verifier was adapted.
The eleven from-scratch models copy Monad's architecture and tokenizer — Llama-style, 8,192-token vocabulary, tied embeddings, head dim 64, 3× MLP — pretrained on PleIAs/SYNTH only. The 10M model trains in about ten minutes on one H100; the 630K one in about two.
## Where I would push back
**The project caught two shortcuts and says so; the question is how many it didn't.** On Maze, model-generated wrong path lengths are longer than correct ones (≈15.2 steps against ≈6.6), so the rule "answer ≤ 10 → correct" already scores well. On faithfulness, the verifiers learn HaluEval's corruption style. Both are named in the limitations. But the natural conclusion is that a verifier scoring 0.93 on a task where a one-line heuristic scores well is not evidence that *verification* is cheap — it is evidence that this instance of the task is. The plateau might be the difficulty ceiling of the proxy, not of the check.
**Monad-56M is the strangest point on every curve and goes unremarked.** It scores *below* the 2M–10M from-scratch models on all three tasks — 0.775 on Countdown against 0.828, 0.878 on Maze against 0.928, 0.818 on faithfulness against 0.873. A 56M model pretrained on 200B tokens losing to a 5M model pretrained on 250M tokens is either a fine-tuning artifact or a genuinely interesting statement about what general pretraining does to a narrow classifier, and it deserves a sentence either way.
**The Qwen2.5-7B faithfulness row is byte-identical to SmolLM2-135M's** — same accuracy, same CI, same AUROC — and sits well below the 1.5B and 3B rows around it. That looks like a transcription error in the appendix rather than a finding, and it is the one number I would not build on.
**"Small verifier" and "cheap verification" are not the same claim.** The motivation is that running a verifier over training-scale corpora is a throughput bottleneck. But the models on the plateau are being *fine-tuned per task* on 9,600 labelled examples, which for the checkable tasks requires a generator and an exact checker — and if you have an exact checker, you do not need a verifier. The load-bearing case is faithfulness, where there is no checker, and that is precisely the case where the result is an in-distribution classifier.
## What it changes
The practical takeaway is not "use a 2M verifier". It is that **verifier size should be a measured parameter rather than an assumed one**, and that measuring it is cheap — this entire study is 3.5B tokens and 30 GPU-hours.
If you are verifying at training scale, or filtering reasoning branches at inference time, or running a fleet of agents that each need their outputs checked, the difference between a 7B judge and a 5M one is the difference between verification being the bottleneck and being free. The project's own next step is the right one: let a model optimize against a small verifier and see whether it learns to fool it, and whether smaller verifiers get fooled sooner. Everything above says small verifiers are accurate. Nothing above says they are robust to something trying to break them.