# Reproducing Jev, from a source I could not find
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/reproducing-jev
> date: 2026-09-22
> tags: explainer, llm, benchmarks, calibration, architecture
I was handed a block of text. An independent team spent several days trying to
reproduce Jev, then trying to break it, then trying to improve it, and wrote
down five things they learned. The findings are sharp, specific and numerate —
the kind of write-up I would normally link to three times in an article.
There is no link. No author, no repository, no benchmark name, no denominators.
So the first half of this piece is the attribution attempt, and it failed. Every
number attributed to that team is marked **Reported-unattributed** and appears
nowhere in this article as a fact about the world. The second half is the part
that is worth your time: I took the five claims as *hypotheses* and went looking
for public evidence that bears on each. Three of the five are independently
corroborated, one is corroborated in kind but not in magnitude, and one is a
mechanism claim that nobody has run the experiment for.
Here is the ledger before the detail.
| # | the claim, as reported | can I check it? | what public evidence says |
|---|---|---|---|
| 1 | bounded answer space ⇒ don't decode token by token; 1–2 orders of magnitude of inference work disappears | mechanism yes, constant no | Corroborated. The mechanism is `max_new_tokens: 0` on an endpoint that shipped in June 2025. The factor is a token-count ratio, and one published release measures the scored path **3.23x slower**. |
| 2 | RLCD is not the moat; the data might be | yes | Corroborated twice. Laya publishes the whole RLCD loop in a Kaggle notebook and none of its data. An independent RLCD study holding the algorithm fixed concludes "the secret sauce is probably more in the data than in the training algorithm". |
| 3 | "open source already beat Jev" is a benchmark illusion; same checkpoint, 0.769 in-distribution, 0.541 out | not their numbers; yes the claim | **Corroborated, and understated.** On a third-party suite Laya sits 88 items of 220 behind the model it beat by 3.9 points at home. |
| 4 | compute-optimal ≠ learnability-optimal; moving state out of the problem sequence cost 27 points; six other ideas failed; only more data worked | no | Corroborated in kind. The closest public analogue is a marker-token ablation that "does not learn… at any learning rate", and a study where only data coverage moved the out-of-distribution number. |
| 5 | a decoder with zero additional training beat every trained small encoder | yes | **Corroborated on public numbers.** Two untrained decoders sit above every trained small encoder on JevBench's hard tier — and an untrained *encoder*, the control, does not. |
## First, the part I could not do
The corpus has been bitten by exactly this once already. Laya's `BENCHMARKS.md`
reports Jev's option-order instability at 0.13, with no citation, in a file whose
own opening line says it never had TypeSafe API access.
[What a decision model cannot do](/articles/what-decision-models-cannot-do) had
to flag that number as the single most load-bearing unverified figure in the
category. I am not going to add five more.
What I searched, so the failure is reproducible. The exact strings
`"compute-optimal ≠ learnability-optimal"`, `"learnability-optimal"`,
`"learnability optimal"`, each of `0.769`, `0.541`, `0.5498`, `0.4069`, `0.5671`,
`0.5628`, `0.5411`, `123,475`, and the phrases *"if the answer space is bounded"*,
*"search cheap, branch wide"*, *"RLCD is not the moat"*. Where: general web
search; Hacker News via the Algolia index (`learnability-optimal`: zero hits);
Hugging Face full-text search across model cards, dataset cards and Space
sources (`learnability-optimal`: zero hits in all three); the arXiv API
(`all:"learnability-optimal"`: zero results); four community `awesome-jev`
indexes; the 52-system JevBench roster; and the `jev-reproductions-tracker`
Space's `data/index.json`, where `0.4069` and `0.5671` do occur and belong to an
Arabic-English sarcasm track and a Brier score respectively, which is what
coincidence looks like.
The nearest misses are worth naming, because someone will think they are the
answer. `0.768` is meraGPT's Decider 1 on the `LocalLLaMA/typed-decisions`
leaderboard, which is close to `0.769` and is a proprietary hosted model with no
out-of-distribution figure. `1,200` is exactly the size of that benchmark's train
split — 1,200 cases, 6,000 typed decisions — which is suggestive enough that I
would bet the team started there, and a bet is not a citation.
**Label for the whole of this article: Reported-unattributed.** Every figure in
the five findings — the 27 points, the 0.769/0.541 pair, the 1,200 → 123,475
scaling, the 0.5671 / 0.5628 / 0.5411 / 0.4069 spread — arrived as text with no
author, no benchmark name and no denominator. I could not establish whether
"0.541" is accuracy, macro-F1 or agreement, or what it is 0.541 *of*. Do not cite
them from here. If you are the team: send the link and I will rewrite this piece
around it, because with the artifacts it is a much better article than this one.
## Finding 1 · the answer space is bounded, so stop decoding
> "If the answer space is bounded, don't generate the answer token by token."
> Removing autoregressive decoding removes one to two orders of magnitude of
> inference work. — *reported, unattributed*
The mechanism is right and it is not theirs, or anyone's. SGLang's `/v1/score`
has done exactly this since **2025-06-21**, PR #7399, *"[OAI refactor] Add rerank
and score serving"* — fifteen months before the category existed. The request it
builds carries `sampling_params={"max_new_tokens": 0}` and `return_logprob=True`,
so the model prefills once, generates nothing, and the reply carries the
logprobs of the token ids you named. I took that handler apart in
[Any model can be Jev](/articles/any-model-can-be-jev); there is nothing to add
except that the finding is a rediscovery of a serving flag.
The constant is where I would push back. "One to two orders of magnitude" is not
a property of the method, it is a property of *how many tokens you made the
baseline type*. Fitting the five committed L40S rows of the one release that
publishes its raw cases gives the autoregressive path as
**95.4 ms + 15.21 ms × generated_tokens, R² = 0.983**. The 62.91x case generated
226 tokens. The 1.09x case generated 13. Same method, same hardware, 58x apart in
reported speedup, and the only thing that changed was the length of the JSON the
baseline had to emit.
The row that should be quoted alongside every speedup claim in this category is
in the same release's README: with 255 candidates on an M2 Max, the scored path
is **3.23x slower** than generating. Scoring `n` candidates is `O(n)` work in the
branch dimension; decoding one answer is not. There is a crossover and the
release publishes where it is.
So: correct, old, and workload-dependent by roughly two orders of magnitude in
its own right.
## Finding 2 · RLCD is not the moat; the data might be
This one I can check almost completely, and it holds.
[Laya](https://github.com/NandhaKishorM/laya) is Apache-2.0: three checkpoints on
Hugging Face, the model code in `laya/common.py`, a `BENCHMARKS.md`, and a
fine-tuning notebook that its own README describes as *"the whole loop: build the
dataset, train with RLCD (proper-scoring-rule rewards, GRPO-style policy
gradient), fit calibration temperatures, evaluate, and push the result to the
Hub."* RLCD — the thing the category is named after — ships as a Kaggle notebook
you can run on two T4s.
**Measured:** I pulled that notebook and read it. Its header says it fine-tunes
Laya *"on the **1,200 training cases (6,000 typed decisions)** of the
`LocalLLaMA/typed-decisions` benchmark"*. The string `synthetic` appears zero
times in it. `load_dataset` appears four times. The notebook consumes a public
dataset; it does not generate one. Nothing in the repository generates the data
the *base* checkpoint was trained on, and nothing publishes it.
TypeSafe say the same thing from the other side, and say it plainly. From the
launch post's collapsed FAQ: *"TypeSafe is primarily a data research lab, which
is how the biggest results in AI get made. We make all the data ourselves. …if
you want to find out more, we'd have to hire you."* A company whose method
acronym is public and whose data policy is a hiring pitch has told you which half
it thinks is defensible.
And there is now a public experiment that isolates it.
[`googao/decision-head-rlcd`](https://github.com/googao/decision-head-rlcd) runs
RLCD on Qwen3.5-4B with a 4.9M-parameter LoRA over 32,000 typed decisions, three
seeds, base model and algorithm and hyperparameters held constant, for 9.0 GPU-hours
on one H200 — about \$42. Its conclusion is the finding, in its authors' words:
*"the secret sauce is probably more in the data than in the training algorithm."*
Its sharper result is the one I would put next to it: *"generalisation tracks
training-data coverage exactly, in both directions,"* and where the training
stream did not cover a structure, *"RL does not merely fail to help — it lands
below the untrained starting point."*
Corroborated. The RL is a notebook. The data is the product.
## Finding 3 · "open source already beat Jev" is a benchmark illusion
This is the most consequential claim in the batch, and the one I could check
hardest, because the specific numbers are unverifiable but the claim is not.
Every open-beats-Jev headline this site has covered was measured on the
challenger's own data:
- **cua-s1-forms**, 99.7% against Jev's 83.6% on 196 real form decisions — trained
on 150,000 rows from the generator that produced the eval.
- **Laya**, 0.766 against Jev's 0.727 on 2,000 typed decisions — from a checkpoint
fine-tuned on that benchmark's own 1,200-case train split.
- **Bespoke Nimble**, 90.12% against Jev's 93.21% on a 324-row contrastive holdout
— in-domain, and Nimble loses anyway.
- **Chopra's fifteen-task suite** — all three models run through their native APIs,
which is fairer, on items nobody else can see.
Since [Jev scores zero](/articles/jev-scores-zero) noted that pattern, the field
has produced the control: a third-party suite that ran the whole roster on
identical items. JevBench v1.3.0, from Benchmark Heaven, scores 52 systems on 534
decisions; its hard tier is 220 items, 111 public and 109 held out, frozen
2026-09-19. Nobody in the table trained on it.
Laya beat Jev by 3.9 points on the benchmark Laya fine-tuned against. On 220
items chosen by someone else, Jev is 163/220 and Laya is 75/220. Eighty-eight
items. The ordering does not narrow, it inverts.
I want to be careful about what that chart is and is not. The two columns are
different tasks, so the *slope* of a line is not a measured degradation and must
not be read as one — this is not a before-and-after of the same test. What is
readable is the ordering inside each column, and the ordering between Jev and
every challenger reverses between them. Jev is the control: cold on both, and
flat.
**Receipts.** Every open-model-beats-Jev headline this site has covered was measured on the challenger's own data. Where the same system has since been run on a third-party suite, the ordering does not survive. Where it has not been run, nobody knows.
| system | its own published number | what kind of number | JevBench hard, 220 items | would the headline survive? |
| :--- | :--- | :--- | ---: | :--- |
| Jev 1.13.0 | 0.727 | generalist, cold — had never seen the benchmark | 163/220 · 0.741 | control. The only row whose two numbers agree. |
| Laya (ModernBERT-large 421M) | 0.766 | specialist — fine-tuned on that benchmark's own 1,200-case train split | 75/220 · 0.341 | No. Ahead of Jev by 3.9 points at home, behind by 40.0 away. |
| Bespoke Nimble 9B | 0.9012 | specialist — LoRA on 2,676 rows from the generator that made the eval | 144/220 · 0.655 | Yes, directionally. Nimble already lost its own head-to-head (Jev 0.9321), and it stays second here. |
| open-jev-deberta-v3-large | 0.85 in-domain, 0.69 out | specialist — and the only one that built a held-out-question split itself | 80/220 · 0.364 | Its own OOD number already said so. The card publishes both. |
| cua-s1-forms (706K params) | 0.997 vs Jev's 0.836 | specialist — 150,000 training rows from the generator that made the 196-case eval | not entered | Unknown, and the source release says no checkpoint claim is established by it. |
| kev family (0.5B–8B) | Jev 0.857 vs kev-8b about 0.78 | already out-of-domain — 764 records from six public sources kev never trained on | 104/220 · 0.473 (kev 8B) | Yes. It was an OOD comparison when it was published, and Jev won it. |
| AgentJev-0.6B | 79.25 vs 77.00, reported | unsourced — I could not find the primary measurement | not listed under that name | Unknown. Flagging it rather than repeating it. |
JevBench is one benchmark, by one author, and half its hard tier is public — at least one entrant records using the public items as a development gate. Read it as a second opinion, not as ground truth. The point of the table is not that JevBench is right; it is that a number measured on the challenger's own generator and a number measured by somebody else are different kinds of number, and only one of them was ever published for most of these.
> method: For each published head-to-head, the left number is the one its authors published on data they chose. The right number is the same system's hard-tier accuracy on JevBench v1.3.0, a third-party suite of 220 items (111 public, 109 held out) frozen 2026-09-19, which none of these systems trained on. The two columns are different tasks, so the levels are not comparable; the ordering between two systems in the same column is.
> source: https://benchmarkheaven.com/api/jevbench/v1.2
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/reproducing-jev/data/home-turf.json (7 rows)
### The benchmark that says it best is the one Laya won
`LocalLLaMA/typed-decisions` — the 0.766-vs-0.727 benchmark — spends a section of
its own dataset card on this, and it is better than anything I was going to
write:
**"Train a specialist on these four workflows, score it on them, and you have measured architecture… It is not a comparison against a general System One model, which has never seen these workflows."** And, two paragraphs later: **"A specialist number sitting next to a generalist number, unlabelled, misleads the reader."**
Its leaderboard tags every row `specialist` or `general` for exactly this reason.
It also publishes three reference points on the same 1,600 cases: always
guessing the commonest label scores **0.520**, a model fitted to the latent
factors that generated each case scores **0.704**, and a fresh teacher sample
scored against gold built from the other samples — the noise floor of the
labelling process — scores **0.735**. On the test split itself, a prior that
ignores the input entirely scores 0.470. The card's own reading of the last one:
*"Scoring far above it means predicting the teacher's quirks rather than the
task."*
Jev scores 0.727 against that 0.735 floor. A fine-tuned checkpoint scoring 0.766
is not 3.9 points better than Jev at the task; it is 3.1 points above the point
at which the card says the number stops measuring the task.
### So which published comparisons survive?
Working through them, because this is the practical consequence:
**Does not survive.** cua-s1-forms' 99.7%, and Laya's 0.766. Both are specialist
numbers printed beside a cold generalist. cua-s1's own source release already
says *"No checkpoint performance claim is established by this source-only
release"*, which is more honesty than the headline got.
**Survives.** Bespoke's, twice over — the in-domain row already went against them,
and they then shipped `docs/PUBLIC_BENCHMARKS.md`, thirteen human-labelled public
datasets and 3,880 records, on which Jev has the lower ECE on 11 of 13 subsets.
`kev`'s too: 764 records from six public sources kev never trained on, Brier 0.211
for Jev against 0.339 for kev-8b. Both of those were out-of-domain comparisons
the day they were published, and both lost.
**Survives, weakly.** Chopra's fifteen tasks. Nobody's home turf, which is the
right design; the items are unpublished, so nobody can re-run it.
**Unknown.** AgentJev-0.6B's reported 79.25 against 77.00. I could not find the
primary measurement, and no checkpoint under that name appears in JevBench's
52-system roster. I am flagging it rather than repeating it, which is the same
rule I am applying to the five findings themselves.
The honest summary is not "the open models are bad". Several of them are very
good at the thing they were fitted to, which is the real thesis of small
specialist models and worth demonstrating. It is that **the sentence "X beats Jev
on typed decisions" has meant, in every case this site has covered, "X beats Jev
on X's data"** — and the one time somebody ran the whole field on shared items,
the ranking changed almost completely.
## Finding 4 · compute-optimal is not learnability-optimal
> Moving state outside the problem sequence, to save compute, cost 27 points —
> attributed to discarding MLM pretraining priors. Six further ideas in one night
> (anchor selection, continuity smoothing, asymmetric windows, bucketed
> temperature, two-stage retrieval, full RLCD) all failed or failed to
> generalise. Only more data worked: 1,200 → 123,475 examples took the
> out-of-distribution number from 0.4069 to 0.5498. — *reported, unattributed*
I cannot check any of the numbers. The shape of the finding has good public
company.
The closest analogue is an ablation in `kotoba-lang/typed-decisions`, stated as a
negative result in the source rather than a hunch. Scoring a *fresh* `[OPT]`
marker token's hidden state, instead of the mean of the option's own text tokens,
**"does not learn"** — in its docstring's words, *"at any learning rate (1e-5…1e-4),
head lr (3e-5, 1e-3), Brier weight (0, 1, 3), autocast on/off, sdpa/eager,
`reference_compile` on/off, on base and on large: every run ends at the label
prior."* The same loop overfits 16 states to loss 0.000 in 50 steps, so the
mechanics were fine. A fresh token has no pretrained structure to score; the
option's words do.
That is the same class of failure the 27 points is being attributed to: an
architectural change that is cheaper and that quietly discards what pretraining
put in the representation. And the "only more data worked" half has a public
match in the RLCD study above — where coverage moved the out-of-distribution
number and nothing else did, and where the same repository's ablations found
augmentation (a data intervention) bought 2.6 points of out-of-domain accuracy
for the same \$0.26 while a symmetric-KL consistency loss (a loss intervention)
bought nothing and cost double.
That repository also publishes the ladder those ablations sit on.
### The mechanism claim is falsifiable, so here is the experiment
"We saved FLOPs while throwing away MLM pretraining priors" is a real claim with
at least three competing explanations, and one 27-point drop cannot distinguish
them:
1. **The priors explanation.** Moving state out of the sequence means the state's
tokens no longer attend to, and are no longer attended by, the question's
tokens, so the joint representation masked-language-model pretraining built is
never formed.
2. **The capacity explanation.** Whatever now carries the state — a pooled vector,
a cached prefix, a separate tower — is simply a narrower channel. Nothing to do
with pretraining; a from-scratch model would lose the same points.
3. **The optimisation explanation.** The new path has an untrained interface at the
join, and it fails the way a fresh `[OPT]` marker fails: it never gets off the
label prior in the budget available.
The experiment that separates them is one training matrix, and it is cheap at this
scale. Run the same architecture change four ways: on the MLM-pretrained encoder,
on a randomly-initialised copy of the identical architecture, on the pretrained
encoder with the new interface warm-started from the old one, and on the
pretrained encoder trained for three times as long.
- If the randomly-initialised model loses the *same* 27 points, the priors are
irrelevant and explanation 2 is right — you bought a narrower channel.
- If the drop is much smaller from random init, the pretrained model is losing
something it had, and explanation 1 survives.
- If warm-starting the interface, or simply training longer, recovers most of the
27, it was explanation 3 and the architecture is fine.
Nobody has published this. It is the cheapest useful experiment in the whole
finding, and the claim as stated is the one thing in the five I would most like
to see an artifact for.
## Finding 5 · an untrained decoder beat every trained small encoder
> 0.5671 against 0.5628 / 0.5411 / 0.4069. — *reported, unattributed*
Those four numbers I cannot check. The claim behind them is checkable, and it is
true on public data.
On JevBench's 220 hard items, two decoders that nobody trained for the decision —
`jqv`, whose entry reads *"Qwen/Qwen3-32B, bf16, read as a direct-logit classifier
(no fine-tuning)"*, and SemIf, whose entry reads *"Qwen/Qwen3.5-4B (frozen,
BF16)"* — score 142/220 and 131/220. Every trained small encoder in the field
scores between 75 and 84: openJev Verdict 84, jeff 83, open-jev-deberta-v3-large
80, Laya 75. SemIf is **ranked second** on the suite's
composite, behind only the closed model it was built to imitate.
The control is what makes this more than a size effect. `OpenDecision` is a
ModernBERT-large zero-shot NLI engine — an encoder of roughly the same size as
Laya's and jeff's, with nothing trained for the decision either — and it scores
73/220, at the bottom with the trained encoders rather than with the untrained
decoders. **It is not untrainedness that wins. It is the decoder.**
**Receipts.** On the one public benchmark that ran all of them on the same items, two untrained decoders read through their own logits beat every trained small encoder in the field — and a zero-shot encoder, the control, lands with the trained encoders rather than with the decoders. The thing that wins is the decoder, not the absence of training.
| system | readout family | trained for the decision? | hard tier | calibration axis |
| :--- | :--- | :--- | ---: | ---: |
| Jev 1.13.0 | closed | yes, and nobody outside TypeSafe knows how | 163/220 | 82.7 |
| jqv (Qwen3-32B) | decoder, direct-logit readout | no — "read as a direct-logit classifier (no fine-tuning)" | 142/220 | 79.0 |
| Bespoke Nimble 9B | decoder, allowed-token readout | yes — LoRA on 2,676 contrastive rows | 144/220 | 65.3 |
| SemIf / openjev (Qwen3.5-4B) | decoder, option-letter readout | no — "Qwen/Qwen3.5-4B (frozen, BF16)" | 131/220 | 72.6 |
| kev 8B | decoder, LoRA + pointer head | yes | 104/220 | 44.2 |
| kev 4B | decoder, LoRA + pointer head | yes | 93/220 | 42.0 |
| kev 0.6B | decoder, LoRA + pointer head | yes | 88/220 | 51.1 |
| openJev Verdict (151M) | encoder, GLiClass head | yes — "gliclass-modern-base-v2.0 fine-tuned" | 84/220 | 51.3 |
| jeff (GLiFormer 400M) | encoder, scoring head | yes, head only; backbone frozen | 83/220 | 64.6 |
| open-jev-deberta-v3-large | encoder, per-option scalar head | yes — CE + Brier on 42,000 questions | 80/220 | 66.4 |
| Laya (ModernBERT-large 421M) | encoder, option-marker head | yes — "RLCD-trained" | 75/220 | 62.5 |
| OpenDecision (ModernBERT-large) | encoder, zero-shot NLI | no — the control for "untrained" | 73/220 | 56.1 |
Hard-tier items do not all have the same option count, so raw accuracy is not chance-corrected; the suite's separate Intelligence axis is. The ordering here is the same under both. Encoder entries were served on CPU and decoder entries on GPU, which moves the speed axis and nothing in these two columns.
> method: JevBench v1.3.0 hard tier: 220 items (129 choice, 77 noul, 14 score), 111 public and 109 held out, frozen 2026-09-19. Accuracy is raw correct-out-of-220 on identical items. Calibration is the suite's own 0-100 axis, higher is better. The training column is the benchmark's own `underlying` field, quoted, not my inference.
> source: https://benchmarkheaven.com/api/jevbench/v1.2
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/reproducing-jev/data/readouts.json (12 rows)
### This is what the serving-feature thesis predicts
[Any model can be Jev](/articles/any-model-can-be-jev) argued that the readout is
a **serving feature, not a model feature** — that there is nothing in a checkpoint
that makes it a decision model, only a way of calling it, and that the way of
calling it has been sitting in SGLang since June 2025. An untrained decoder
beating trained encoders is that thesis's own prediction, and this is the first
time I have seen it come out of a benchmark somebody else ran.
The second panel is where the thesis gets its limit back. The same article
insisted that **calibration is the one property you cannot get from the serving
layer**, because measuring it costs labelled outcomes. On JevBench's calibration
axis the untrained decoders are 79.0 and 72.6, ahead of every trained small model
in the table — and Jev is 82.7, ahead of all of them. The free readout gets you
most of the accuracy and most of the calibration a small trained model gets you,
and neither route gets you the closed model. What it does not do is what a
training run was supposed to buy: the three `kev` checkpoints in that table,
which do fit a head *and* a temperature, score 42.0, 44.2 and 51.1 — worse
calibrated than either untrained readout above them. That is the GPT-4 report's
own finding (pre-train ECE 0.007, post-PPO 0.074) arriving in a different
category by a different route.
### But which decoder readout?
The brief I was given asks whether their decoder result is the same mechanism.
From one accuracy number it is not possible to say, and the difference matters
enough that I would not accept the result without knowing.
There are two untrained decoder readouts in the wild and they fail in opposite
directions:
- **Single-position vocabulary readout.** Options go in the prompt as lettered
text; you slice the rows for `A`, `B`, `C` out of one logit vector. That is
openjev's `direct.py`, which is the public 131/220, and it is what SGLang's
`/v1/score` serves when the options go in the prompt and `label_token_ids` are
the letters. It requires every candidate to be one token, and it puts every
option at a *position* — which is why the same code changes its mind on
**10 of 36** reversed questions, and why slot A is worth **+1.71 logits** on
average.
- **Per-candidate likelihood.** Each candidate gets its own forward pass or its own
branch of a shared prefill, and the score is the summed log-probability of its
own tokens. No position, no letter, no order sensitivity — and a length bias
instead, which one repo's committed rows put at r = −0.004 within a decision,
so theoretically live and empirically inert there.
If the reported 0.5671 came out of a letter readout, some unknown fraction of it
is a slot prior and re-running with shuffled option orders will move it. If it
came out of per-candidate likelihood, it will not. One reversed list settles it in
an afternoon, and it is the first thing I would ask them for.
Worth noting which way the public data falls: the untrained readout that is
beating the trained encoders is the *letter* one, the family that carries the
known instability. It is winning on accuracy while holding a defect the encoders
do not have.
## The branching argument is right about the shape and wrong about the agent
The architectural claim attached to the five findings is the most interesting
thing in them, and I think it is half true in a way worth spelling out.
> Agents often do not have a reasoning problem but a **branching problem**. Every
> action creates more possible next states, the tree explodes over long horizons,
> and autoregressive reasoning is an expensive way to search it. So "search cheap,
> branch wide, reason only when necessary" should get more valuable as horizons
> lengthen. — *reported, unattributed*
And then, to their credit, immediately afterwards: **long-horizon planning is
where Jev has impressed them least.**
Those two sentences are usually read as a theory and an embarrassment. They are
the same observation, and
[what a decision model cannot do](/articles/what-decision-models-cannot-do)
supplies the missing step: a decision model cannot plan, because planning needs a
place to write things down and a bounded output has none. Planning lives in the
harness.
A search needs three jobs done. Somebody **enumerates** the successors of a state,
somebody **prices** them, and somebody **sequences** — decides what to expand next,
when to stop, when to back up. A decision model does exactly one of the three.
- **Enumeration is the caller's.** WindTunnel rebuilds its action menu every step
from the tools the page is exposing right now, caps it, and throws on a
duplicate id. jev-semgrep enumerates thirty lines times however many meanings you
passed. json-render enumerates seventeen component types plus per-request value
candidates. In each case the branching factor is a number the harness chose.
"Branch wide" is a property of the enumerator's budget, not of the model.
- **Pricing is the model's, and it is the whole of the model's job.** A 255-option
Choice carries $\log_2 255 \approx 7.99$ bits and nothing else comes back — no
tokens, no text, no state.
- **Sequencing is nowhere.** There is no scratchpad and nothing carries between
calls; the caller reassembles `state` each time. Chopra's WikiRouter is
Wikipedia navigation on a ten-hop budget, and Jev reached the destination on
**1 of 120 routes**, stopping on 118. Not lost — stopped. Deciding to stop is a
decision; knowing you are three hops in with a target still to reach is state.
Which gives the economics, and this is the part I have not seen said. Width is
almost free: the state is prefilled once and the options fan out over a shared
prefix, so pricing `b` branches costs about one state read. Depth is not free at
all: every level is a fresh call that re-reads the state, because the contract has
nowhere to keep it. So the cost of a search is roughly (one state read) × (depth),
independent of width. **The shape rewards wide and punishes deep** — and a long
horizon is made of depth.
That split is the third gate in
[the procedure this site derived from eighteen deployments](/articles/where-to-use-jev),
reached from the other end: the questions have to be independent, and when they
are not, the relation gets built in code before the model sees a pair.
**Reasoned:** "search cheap, branch wide" does get more valuable as horizons
lengthen, and the value accrues to whoever wrote the search, not to the model. A
decision model dropped into a long-horizon task without a harness that enumerates
and sequences will look worst exactly where a search would have helped most. Which
is what they observed.
One measurement supports the split rather than the pessimism, and it is worth
having. JevBench's hard tier has a `multi_hop` family — multi-hop reasoning
*inside a single decision* — and Jev scores **30 of 35** on it. Its worst family
on the whole tier is `temporal_numeric` at **8 of 30**, which is the "reads dates
as text" failure TypeSafe documents itself. So the thing that is weak is not hops;
it is episodes. No single-decision benchmark tests planning, because planning is
not a decision, and that is the gap the harness people are filling.
## What I would ask them for
If the team reads this: four artifacts turn all five findings into something the
rest of us can build on, and none of them is a model release.
1. **The benchmark name and the split definition** behind 0.769/0.541. What is the
out-of-distribution set held out *from*? A held-out question schema and a
held-out workflow are different experiments with different implications.
2. **The readout.** Letter logits or per-candidate likelihood, and the option-order
flip rate at the same configuration.
3. **The 27-point run's matrix** — or just the random-init control, which is one
extra training run and settles the mechanism claim on its own.
4. **The six failures, with their numbers.** Negative results at this specificity
are rarer than checkpoints and considerably more useful. Six ideas in one night
that did not generalise is, published, a better contribution than a seventh
that did.
Until then: five interesting hypotheses, three of which the public record already
supports, and none of which should be cited from this page.
A link. Any of: a post, a repository, a gist, a model card, a paper. My search is itemised in the first section — exact phrases and figures across general web search, HN's Algolia index, Hugging Face full-text search over models, datasets and Spaces, the arXiv API, four `awesome-jev` indexes, the 52-system JevBench roster and the `jev-reproductions-tracker` dataset. It ran on 2026-09-22 and it found nothing. A search failing is weak evidence at the best of times; if the work is on a platform none of those index, or went up after that date, this claim expires immediately and the article should be rewritten around the primary source.
JevBench is one benchmark by one author, and 111 of its 220 hard items are public — at least one entrant records using the public set as a development gate, which is precisely the contamination this claim is about. Run any two of these systems on a third suite nobody has optimised against — Bespoke's thirteen public datasets would do, or a fresh split of `LocalLLaMA/typed-decisions` scored with the specialist and generalist rows kept apart. If Laya lands anywhere near Jev there, the inversion is a JevBench artifact and my reading is wrong.
My control is a single row: `OpenDecision`, a zero-shot ModernBERT-large, at 73/220 against the untrained decoders' 131 and 142. One control is thin. Add two more untrained encoders of comparable size — a zero-shot NLI model and a stock cross-encoder reranker read as a scorer — and an untrained decoder near 400M rather than 4B. If a well-chosen untrained encoder reaches the 130s, the split is capability and scale rather than architecture. If a 400M untrained decoder collapses to the encoders' band, the result is about model size and I have described it wrongly.
Read from JevBench's calibration axis, which is one suite's composite and not an ECE. Compute raw ECE and Brier for the frozen base model and for each trained checkpoint on the *same* rows, with and without a fitted temperature. If the trained models' raw ECE is better than their base's and only the suite's axis disagrees, the axis is doing the work and this paragraph is wrong. `kev` sitting at 42.0 to 51.1 on that axis while the project's own model card reports ECE 0.065, and 0.031 after a fitted temperature, is exactly the kind of discrepancy that would show up.
Build a long-horizon task where the harness does nothing but pass the model's previous choice back in `state`, and compare it against the same task with a real search in the harness — beam, backtracking, an explicit frontier. If the bare loop performs comparably, the model is carrying more of the search than I think it can, and "sequencing is nowhere" is too strong. WikiRouter at 1 of 120 is one task from one author on unpublished items and is not enough to rest this on.
The fit is five points, which is five points. Run one harness across ten schemas with generation lengths from 10 to 400 tokens and plot speedup against generated tokens. A straight line says the ratio is arithmetic; a bend says something in the method is doing work I have not accounted for.
---
*Public artifacts used here, all fetched 2026-09-22: JevBench v1.3.0 via [benchmarkheaven.com/api/jevbench/v1.2](https://benchmarkheaven.com/api/jevbench/v1.2) (52 systems, 534 decisions; hard tier 220 items frozen 2026-09-19, `dataset_hash_all` `ec200ccd…`), the [`LocalLLaMA/typed-decisions`](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) dataset card and its baseline table, [`NandhaKishorM/laya`](https://github.com/NandhaKishorM/laya) including `notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb` read as JSON, [`googao/decision-head-rlcd`](https://github.com/googao/decision-head-rlcd), and [`kotoba-lang/typed-decisions`](https://github.com/kotoba-lang/typed-decisions). Companion pieces, which carry the measurements I am re-using rather than re-deriving: the architecture and training in [RLCD is not constrained decoding](/articles/rlcd-calibrated-decisions), the two readout families in [A System One model in 706,048 parameters](/articles/cua-s1-forms), the serving-flag thesis and the speedup regression in [Any model can be Jev](/articles/any-model-can-be-jev), the relational-choice zero in [Jev scores zero](/articles/jev-scores-zero), and the planning argument in [What a decision model cannot do](/articles/what-decision-models-cannot-do). The same five findings are read from the deployment side, as four ordered gates, in [The test is whether one option can read another](/articles/where-to-use-jev). On figures: the five findings arrived as plain text and carry none — there is no post, repository or card to take one from, which is the whole of the first section — so both figures above belong to `decision-head-rlcd`, the public experiment that corroborates findings 2 and 4, and are reproduced under its Apache-2.0 terms with the details in [NOTICE.txt](/articles/reproducing-jev/NOTICE.txt). Laya's own comparison sheets are the other candidate and are already mirrored on this site, in [Jev in the browser](/articles/jev-in-the-browser) and [Jev scores zero](/articles/jev-scores-zero); `kotoba-lang/typed-decisions` publishes no image at all. None of the three repositories ships a video, so there is none here.*