Reproducing Jev, from a source I could not find

I was handed a block of text. An independent team spent several days trying to reproduce Jev, then trying to break it, then trying to improve it, and wrote down five things they learned. The findings are sharp, specific and numerate — the kind of write-up I would normally link to three times in an article.

There is no link. No author, no repository, no benchmark name, no denominators.

So the first half of this piece is the attribution attempt, and it failed. Every number attributed to that team is marked Reported-unattributed and appears nowhere in this article as a fact about the world. The second half is the part that is worth your time: I took the five claims as hypotheses and went looking for public evidence that bears on each. Three of the five are independently corroborated, one is corroborated in kind but not in magnitude, and one is a mechanism claim that nobody has run the experiment for.

Here is the ledger before the detail.

First, the part I could not do

The corpus has been bitten by exactly this once already. Laya's BENCHMARKS.md reports Jev's option-order instability at 0.13, with no citation, in a file whose own opening line says it never had TypeSafe API access. What a decision model cannot do had to flag that number as the single most load-bearing unverified figure in the category. I am not going to add five more.

What I searched, so the failure is reproducible. The exact strings "compute-optimal ≠ learnability-optimal", "learnability-optimal", "learnability optimal", each of 0.769, 0.541, 0.5498, 0.4069, 0.5671, 0.5628, 0.5411, 123,475, and the phrases "if the answer space is bounded", "search cheap, branch wide", "RLCD is not the moat". Where: general web search; Hacker News via the Algolia index (learnability-optimal: zero hits); Hugging Face full-text search across model cards, dataset cards and Space sources (learnability-optimal: zero hits in all three); the arXiv API (all:"learnability-optimal": zero results); four community awesome-jev indexes; the 52-system JevBench roster; and the jev-reproductions-tracker Space's data/index.json, where 0.4069 and 0.5671 do occur and belong to an Arabic-English sarcasm track and a Brier score respectively, which is what coincidence looks like.

The nearest misses are worth naming, because someone will think they are the answer. 0.768 is meraGPT's Decider 1 on the LocalLLaMA/typed-decisions leaderboard, which is close to 0.769 and is a proprietary hosted model with no out-of-distribution figure. 1,200 is exactly the size of that benchmark's train split — 1,200 cases, 6,000 typed decisions — which is suggestive enough that I would bet the team started there, and a bet is not a citation.

Finding 1 · the answer space is bounded, so stop decoding

"If the answer space is bounded, don't generate the answer token by token." Removing autoregressive decoding removes one to two orders of magnitude of inference work. — reported, unattributed

The mechanism is right and it is not theirs, or anyone's. SGLang's /v1/score has done exactly this since 2025-06-21, PR #7399, "[OAI refactor] Add rerank and score serving" — fifteen months before the category existed. The request it builds carries sampling_params={"max_new_tokens": 0} and return_logprob=True, so the model prefills once, generates nothing, and the reply carries the logprobs of the token ids you named. I took that handler apart in Any model can be Jev; there is nothing to add except that the finding is a rediscovery of a serving flag.

The constant is where I would push back. "One to two orders of magnitude" is not a property of the method, it is a property of how many tokens you made the baseline type. Fitting the five committed L40S rows of the one release that publishes its raw cases gives the autoregressive path as 95.4 ms + 15.21 ms × generated_tokens, R² = 0.983. The 62.91x case generated 226 tokens. The 1.09x case generated 13. Same method, same hardware, 58x apart in reported speedup, and the only thing that changed was the length of the JSON the baseline had to emit.

The row that should be quoted alongside every speedup claim in this category is in the same release's README: with 255 candidates on an M2 Max, the scored path is 3.23x slower than generating. Scoring n candidates is O(n) work in the branch dimension; decoding one answer is not. There is a crossover and the release publishes where it is.

So: correct, old, and workload-dependent by roughly two orders of magnitude in its own right.

Finding 2 · RLCD is not the moat; the data might be

This one I can check almost completely, and it holds.

Laya is Apache-2.0: three checkpoints on Hugging Face, the model code in laya/common.py, a BENCHMARKS.md, and a fine-tuning notebook that its own README describes as "the whole loop: build the dataset, train with RLCD (proper-scoring-rule rewards, GRPO-style policy gradient), fit calibration temperatures, evaluate, and push the result to the Hub." RLCD — the thing the category is named after — ships as a Kaggle notebook you can run on two T4s.

Measured: I pulled that notebook and read it. Its header says it fine-tunes Laya "on the 1,200 training cases (6,000 typed decisions) of the LocalLLaMA/typed-decisions benchmark". The string synthetic appears zero times in it. load_dataset appears four times. The notebook consumes a public dataset; it does not generate one. Nothing in the repository generates the data the base checkpoint was trained on, and nothing publishes it.

TypeSafe say the same thing from the other side, and say it plainly. From the launch post's collapsed FAQ: "TypeSafe is primarily a data research lab, which is how the biggest results in AI get made. We make all the data ourselves. …if you want to find out more, we'd have to hire you." A company whose method acronym is public and whose data policy is a hiring pitch has told you which half it thinks is defensible.

And there is now a public experiment that isolates it. googao/decision-head-rlcd runs RLCD on Qwen3.5-4B with a 4.9M-parameter LoRA over 32,000 typed decisions, three seeds, base model and algorithm and hyperparameters held constant, for 9.0 GPU-hours on one H200 — about $42. Its conclusion is the finding, in its authors' words: "the secret sauce is probably more in the data than in the training algorithm." Its sharper result is the one I would put next to it: "generalisation tracks training-data coverage exactly, in both directions," and where the training stream did not cover a structure, "RL does not merely fail to help — it lands below the untrained starting point."

expand

Corroborated. The RL is a notebook. The data is the product.

Finding 3 · "open source already beat Jev" is a benchmark illusion

This is the most consequential claim in the batch, and the one I could check hardest, because the specific numbers are unverifiable but the claim is not.

Every open-beats-Jev headline this site has covered was measured on the challenger's own data:

• cua-s1-forms, 99.7% against Jev's 83.6% on 196 real form decisions — trained on 150,000 rows from the generator that produced the eval.
• Laya, 0.766 against Jev's 0.727 on 2,000 typed decisions — from a checkpoint fine-tuned on that benchmark's own 1,200-case train split.
• Bespoke Nimble, 90.12% against Jev's 93.21% on a 324-row contrastive holdout — in-domain, and Nimble loses anyway.
• Chopra's fifteen-task suite — all three models run through their native APIs, which is fairer, on items nobody else can see.

Since Jev scores zero noted that pattern, the field has produced the control: a third-party suite that ran the whole roster on identical items. JevBench v1.3.0, from Benchmark Heaven, scores 52 systems on 534 decisions; its hard tier is 220 items, 111 public and 109 held out, frozen 2026-09-19. Nobody in the table trained on it.

Laya beat Jev by 3.9 points on the benchmark Laya fine-tuned against. On 220 items chosen by someone else, Jev is 163/220 and Laya is 75/220. Eighty-eight items. The ordering does not narrow, it inverts.

I want to be careful about what that chart is and is not. The two columns are different tasks, so the slope of a line is not a measured degradation and must not be read as one — this is not a before-and-after of the same test. What is readable is the ordering inside each column, and the ordering between Jev and every challenger reverses between them. Jev is the control: cold on both, and flat.

The benchmark that says it best is the one Laya won

LocalLLaMA/typed-decisions — the 0.766-vs-0.727 benchmark — spends a section of its own dataset card on this, and it is better than anything I was going to write:

Its leaderboard tags every row specialist or general for exactly this reason. It also publishes three reference points on the same 1,600 cases: always guessing the commonest label scores 0.520, a model fitted to the latent factors that generated each case scores 0.704, and a fresh teacher sample scored against gold built from the other samples — the noise floor of the labelling process — scores 0.735. On the test split itself, a prior that ignores the input entirely scores 0.470. The card's own reading of the last one: "Scoring far above it means predicting the teacher's quirks rather than the task."

Jev scores 0.727 against that 0.735 floor. A fine-tuned checkpoint scoring 0.766 is not 3.9 points better than Jev at the task; it is 3.1 points above the point at which the card says the number stops measuring the task.

So which published comparisons survive?

Working through them, because this is the practical consequence:

Does not survive. cua-s1-forms' 99.7%, and Laya's 0.766. Both are specialist numbers printed beside a cold generalist. cua-s1's own source release already says "No checkpoint performance claim is established by this source-only release", which is more honesty than the headline got.

Survives. Bespoke's, twice over — the in-domain row already went against them, and they then shipped docs/PUBLIC_BENCHMARKS.md, thirteen human-labelled public datasets and 3,880 records, on which Jev has the lower ECE on 11 of 13 subsets. kev's too: 764 records from six public sources kev never trained on, Brier 0.211 for Jev against 0.339 for kev-8b. Both of those were out-of-domain comparisons the day they were published, and both lost.

Survives, weakly. Chopra's fifteen tasks. Nobody's home turf, which is the right design; the items are unpublished, so nobody can re-run it.

Unknown. AgentJev-0.6B's reported 79.25 against 77.00. I could not find the primary measurement, and no checkpoint under that name appears in JevBench's 52-system roster. I am flagging it rather than repeating it, which is the same rule I am applying to the five findings themselves.

The honest summary is not "the open models are bad". Several of them are very good at the thing they were fitted to, which is the real thesis of small specialist models and worth demonstrating. It is that the sentence "X beats Jev on typed decisions" has meant, in every case this site has covered, "X beats Jev on X's data" — and the one time somebody ran the whole field on shared items, the ranking changed almost completely.

Finding 4 · compute-optimal is not learnability-optimal

Moving state outside the problem sequence, to save compute, cost 27 points — attributed to discarding MLM pretraining priors. Six further ideas in one night (anchor selection, continuity smoothing, asymmetric windows, bucketed temperature, two-stage retrieval, full RLCD) all failed or failed to generalise. Only more data worked: 1,200 → 123,475 examples took the out-of-distribution number from 0.4069 to 0.5498. — reported, unattributed

I cannot check any of the numbers. The shape of the finding has good public company.

The closest analogue is an ablation in kotoba-lang/typed-decisions, stated as a negative result in the source rather than a hunch. Scoring a fresh [OPT] marker token's hidden state, instead of the mean of the option's own text tokens, "does not learn" — in its docstring's words, "at any learning rate (1e-5…1e-4), head lr (3e-5, 1e-3), Brier weight (0, 1, 3), autocast on/off, sdpa/eager, reference_compile on/off, on base and on large: every run ends at the label prior." The same loop overfits 16 states to loss 0.000 in 50 steps, so the mechanics were fine. A fresh token has no pretrained structure to score; the option's words do.

That is the same class of failure the 27 points is being attributed to: an architectural change that is cheaper and that quietly discards what pretraining put in the representation. And the "only more data worked" half has a public match in the RLCD study above — where coverage moved the out-of-distribution number and nothing else did, and where the same repository's ablations found augmentation (a data intervention) bought 2.6 points of out-of-domain accuracy for the same $0.26 while a symmetric-KL consistency loss (a loss intervention) bought nothing and cost double.

That repository also publishes the ladder those ablations sit on.

expand

The mechanism claim is falsifiable, so here is the experiment

"We saved FLOPs while throwing away MLM pretraining priors" is a real claim with at least three competing explanations, and one 27-point drop cannot distinguish them:

1. The priors explanation. Moving state out of the sequence means the state's tokens no longer attend to, and are no longer attended by, the question's tokens, so the joint representation masked-language-model pretraining built is never formed.
2. The capacity explanation. Whatever now carries the state — a pooled vector, a cached prefix, a separate tower — is simply a narrower channel. Nothing to do with pretraining; a from-scratch model would lose the same points.
3. The optimisation explanation. The new path has an untrained interface at the join, and it fails the way a fresh [OPT] marker fails: it never gets off the label prior in the budget available.

The experiment that separates them is one training matrix, and it is cheap at this scale. Run the same architecture change four ways: on the MLM-pretrained encoder, on a randomly-initialised copy of the identical architecture, on the pretrained encoder with the new interface warm-started from the old one, and on the pretrained encoder trained for three times as long.

• If the randomly-initialised model loses the same 27 points, the priors are irrelevant and explanation 2 is right — you bought a narrower channel.
• If the drop is much smaller from random init, the pretrained model is losing something it had, and explanation 1 survives.
• If warm-starting the interface, or simply training longer, recovers most of the 27, it was explanation 3 and the architecture is fine.

Nobody has published this. It is the cheapest useful experiment in the whole finding, and the claim as stated is the one thing in the five I would most like to see an artifact for.

Finding 5 · an untrained decoder beat every trained small encoder

0.5671 against 0.5628 / 0.5411 / 0.4069. — reported, unattributed

Those four numbers I cannot check. The claim behind them is checkable, and it is true on public data.

On JevBench's 220 hard items, two decoders that nobody trained for the decision — jqv, whose entry reads "Qwen/Qwen3-32B, bf16, read as a direct-logit classifier (no fine-tuning)", and SemIf, whose entry reads "Qwen/Qwen3.5-4B (frozen, BF16)" — score 142/220 and 131/220. Every trained small encoder in the field scores between 75 and 84: openJev Verdict 84, jeff 83, open-jev-deberta-v3-large 80, Laya 75. SemIf is ranked second on the suite's composite, behind only the closed model it was built to imitate.

The control is what makes this more than a size effect. OpenDecision is a ModernBERT-large zero-shot NLI engine — an encoder of roughly the same size as Laya's and jeff's, with nothing trained for the decision either — and it scores 73/220, at the bottom with the trained encoders rather than with the untrained decoders. It is not untrainedness that wins. It is the decoder.

This is what the serving-feature thesis predicts

Any model can be Jev argued that the readout is a serving feature, not a model feature — that there is nothing in a checkpoint that makes it a decision model, only a way of calling it, and that the way of calling it has been sitting in SGLang since June 2025. An untrained decoder beating trained encoders is that thesis's own prediction, and this is the first time I have seen it come out of a benchmark somebody else ran.

The second panel is where the thesis gets its limit back. The same article insisted that calibration is the one property you cannot get from the serving layer, because measuring it costs labelled outcomes. On JevBench's calibration axis the untrained decoders are 79.0 and 72.6, ahead of every trained small model in the table — and Jev is 82.7, ahead of all of them. The free readout gets you most of the accuracy and most of the calibration a small trained model gets you, and neither route gets you the closed model. What it does not do is what a training run was supposed to buy: the three kev checkpoints in that table, which do fit a head and a temperature, score 42.0, 44.2 and 51.1 — worse calibrated than either untrained readout above them. That is the GPT-4 report's own finding (pre-train ECE 0.007, post-PPO 0.074) arriving in a different category by a different route.

But which decoder readout?

The brief I was given asks whether their decoder result is the same mechanism. From one accuracy number it is not possible to say, and the difference matters enough that I would not accept the result without knowing.

There are two untrained decoder readouts in the wild and they fail in opposite directions:

• Single-position vocabulary readout. Options go in the prompt as lettered text; you slice the rows for A, B, C out of one logit vector. That is openjev's direct.py, which is the public 131/220, and it is what SGLang's /v1/score serves when the options go in the prompt and label_token_ids are the letters. It requires every candidate to be one token, and it puts every option at a position — which is why the same code changes its mind on 10 of 36 reversed questions, and why slot A is worth +1.71 logits on average.
• Per-candidate likelihood. Each candidate gets its own forward pass or its own branch of a shared prefill, and the score is the summed log-probability of its own tokens. No position, no letter, no order sensitivity — and a length bias instead, which one repo's committed rows put at r = −0.004 within a decision, so theoretically live and empirically inert there.

If the reported 0.5671 came out of a letter readout, some unknown fraction of it is a slot prior and re-running with shuffled option orders will move it. If it came out of per-candidate likelihood, it will not. One reversed list settles it in an afternoon, and it is the first thing I would ask them for.

Worth noting which way the public data falls: the untrained readout that is beating the trained encoders is the letter one, the family that carries the known instability. It is winning on accuracy while holding a defect the encoders do not have.

The branching argument is right about the shape and wrong about the agent

The architectural claim attached to the five findings is the most interesting thing in them, and I think it is half true in a way worth spelling out.

Agents often do not have a reasoning problem but a branching problem. Every action creates more possible next states, the tree explodes over long horizons, and autoregressive reasoning is an expensive way to search it. So "search cheap, branch wide, reason only when necessary" should get more valuable as horizons lengthen. — reported, unattributed

And then, to their credit, immediately afterwards: long-horizon planning is where Jev has impressed them least.

Those two sentences are usually read as a theory and an embarrassment. They are the same observation, and what a decision model cannot do supplies the missing step: a decision model cannot plan, because planning needs a place to write things down and a bounded output has none. Planning lives in the harness.

A search needs three jobs done. Somebody enumerates the successors of a state, somebody prices them, and somebody sequences — decides what to expand next, when to stop, when to back up. A decision model does exactly one of the three.

• Enumeration is the caller's. WindTunnel rebuilds its action menu every step from the tools the page is exposing right now, caps it, and throws on a duplicate id. jev-semgrep enumerates thirty lines times however many meanings you passed. json-render enumerates seventeen component types plus per-request value candidates. In each case the branching factor is a number the harness chose. "Branch wide" is a property of the enumerator's budget, not of the model.
• Pricing is the model's, and it is the whole of the model's job. A 255-option Choice carries $\log_2 255 \approx 7.99$ bits and nothing else comes back — no tokens, no text, no state.
• Sequencing is nowhere. There is no scratchpad and nothing carries between calls; the caller reassembles state each time. Chopra's WikiRouter is Wikipedia navigation on a ten-hop budget, and Jev reached the destination on 1 of 120 routes, stopping on 118. Not lost — stopped. Deciding to stop is a decision; knowing you are three hops in with a target still to reach is state.

Which gives the economics, and this is the part I have not seen said. Width is almost free: the state is prefilled once and the options fan out over a shared prefix, so pricing b branches costs about one state read. Depth is not free at all: every level is a fresh call that re-reads the state, because the contract has nowhere to keep it. So the cost of a search is roughly (one state read) × (depth), independent of width. The shape rewards wide and punishes deep — and a long horizon is made of depth.

That split is the third gate in the procedure this site derived from eighteen deployments, reached from the other end: the questions have to be independent, and when they are not, the relation gets built in code before the model sees a pair.

Reasoned: "search cheap, branch wide" does get more valuable as horizons lengthen, and the value accrues to whoever wrote the search, not to the model. A decision model dropped into a long-horizon task without a harness that enumerates and sequences will look worst exactly where a search would have helped most. Which is what they observed.

One measurement supports the split rather than the pessimism, and it is worth having. JevBench's hard tier has a multi_hop family — multi-hop reasoning inside a single decision — and Jev scores 30 of 35 on it. Its worst family on the whole tier is temporal_numeric at 8 of 30, which is the "reads dates as text" failure TypeSafe documents itself. So the thing that is weak is not hops; it is episodes. No single-decision benchmark tests planning, because planning is not a decision, and that is the gap the harness people are filling.

What I would ask them for

If the team reads this: four artifacts turn all five findings into something the rest of us can build on, and none of them is a model release.

1. The benchmark name and the split definition behind 0.769/0.541. What is the out-of-distribution set held out from? A held-out question schema and a held-out workflow are different experiments with different implications.
2. The readout. Letter logits or per-candidate likelihood, and the option-order flip rate at the same configuration.
3. The 27-point run's matrix — or just the random-init control, which is one extra training run and settles the mechanism claim on its own.
4. The six failures, with their numbers. Negative results at this specificity are rarer than checkpoints and considerably more useful. Six ideas in one night that did not generalise is, published, a better contribution than a seventh that did.

Until then: five interesting hypotheses, three of which the public record already supports, and none of which should be cited from this page.

Public artifacts used here, all fetched 2026-09-22: JevBench v1.3.0 via benchmarkheaven.com/api/jevbench/v1.2 (52 systems, 534 decisions; hard tier 220 items frozen 2026-09-19, dataset_hash_all ec200ccd…), the LocalLLaMA/typed-decisions dataset card and its baseline table, NandhaKishorM/laya including notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb read as JSON, googao/decision-head-rlcd, and kotoba-lang/typed-decisions. Companion pieces, which carry the measurements I am re-using rather than re-deriving: the architecture and training in RLCD is not constrained decoding, the two readout families in A System One model in 706,048 parameters, the serving-flag thesis and the speedup regression in Any model can be Jev, the relational-choice zero in Jev scores zero, and the planning argument in What a decision model cannot do. The same five findings are read from the deployment side, as four ordered gates, in The test is whether one option can read another. On figures: the five findings arrived as plain text and carry none — there is no post, repository or card to take one from, which is the whole of the first section — so both figures above belong to decision-head-rlcd, the public experiment that corroborates findings 2 and 4, and are reproduced under its Apache-2.0 terms with the details in NOTICE.txt. Laya's own comparison sheets are the other candidate and are already mirrored on this site, in Jev in the browser and Jev scores zero; kotoba-lang/typed-decisions publishes no image at all. None of the three repositories ships a video, so there is none here.