Jev alternatives, week two: the numbers arrive, and most sit on public items

mdjsonmcpcopy↗ai

2026-09-26 · 24 min · explainer · llm · ecosystem · benchmarks · calibration · architecture

Week one was a census of the code people wrote against Jev in its first nine days. It had one shape: a cheap, typed classifier in front of something slower or more expensive. Four of the fourteen projects I could read reported a measured benchmark.

Week two is different in kind. The eleven posts in this roundup are seven open-weight decision models, a local server that speaks Jev's wire format, a closed classifier, a benchmark that went live, and a recipe for training data; a twelfth, a Core ML port of Kev, arrived late. This time there are numbers, several of them against Jev.

Most of those comparisons stand on the same ground: the 231 public items of JevBench, Benchmark Heaven's third-party decision benchmark. Three of the new releases report a score on them. None puts its number beside the others', and only one puts it beside Jev's, in a reformatted test. A fourth, Open-Jev, ran them on purpose, against Jev. JevBench has since added 308 sealed items nobody outside can see, and on those a public lead over Jev turns into a deficit.

Two releases did the thing this site keeps asking for. Lev publishes a head-to-head it loses. Open-Jev ran somebody else's items and published the loss.

Parameter counts below are safetensors header sums, not the cards' round numbers. Code was read, never run. Measured means read from a file or recomputed from published rows; reported means the author's claim, which I could not check; reasoned means my inference. All checked on 2026-09-26.

Which family each one is

The first question to ask of a decision model is not its score. It is how it turns a list of options into probabilities, because that decides what can go wrong. This site has used three families since cua-s1-forms and Kev:

• Letter (vocabulary) readout. Options go into the prompt as letters, and the answer is the letter tokens' logits. A and B carry different priors, so order matters, and the alphabet caps the option count.
• Per-option scorer. Each option is encoded alone and scored to one number; the softmax is over those numbers. No option sees another, so order cannot matter.
• Pointer. Options are spans in one shared sequence and a head points at one. Options see each other, and order matters again. This is Kev's family.

From each release's code (measured): XOR, Cua-S1-4B-0.2 and Solomon are letter readouts. Open-Jev is a per-option scorer. NeoHorse-Jev-4B and Lumma-fev are pointers, and both credit Kev. localjev is none of them: the model writes its probabilities into JSON. Span-01 is closed.

Lev is both. Mode A is a letter readout over single-token label codes, read in two option orders and averaged. Mode B takes over when the options outrun those codes: a per-option head whose candidates attend to each other through a set_attention layer before scoring, so it is not isolated. It is the only release this week that does both.

The yardstick nobody names

JevBench is one author's benchmark, run from a server in Germany one request at a time; version 1.4.2, scored 24 September, has 93 systems. Everyone who reported on its 231 public items this week:

For scale, JevBench's own run puts reflex-4b at 183 and Kev-4B at 153. A self-run on a different harness is not a rank.

Public items are fine for development and weak for claims. Anyone can tune against them or generate training data that resembles them. Nobody here says they did, and none of the numbers above can tell you. That is what v1.4's 308 sealed items are for: only the benchmark sees them.

What the sealed tier shows (measured; I read 14 of the board's 93 rows):

• Every decision model I read loses a lot. Jev goes from 86.6% to 36.7%, a 49.9-point drop. decider-4b v2 goes from 83.5% to 34.7%, Cygnet from 87.9% to 33.8%, XOR's row from 89.6% to 33.8%, Open-Jev 9B from 77.5% to 29.9%, and Open-Jev 2B from 64.5% to 26.3%. The median public-minus-sealed gap across the 89 ranked systems is 45.8 points.
• GPT-6 Luna, a frontier chat model, goes from 99.6% to 95.5%: 4.1 points.

The obvious reading of that drop is wrong. It is not evidence that anybody trained on the public items: Jev drops 49.9 points, and Jev is the control. The sealed families are much harder for these decision models than for a frontier chat model (reasoned, from Luna's 4.1). What the drop does show is that a public lead of a few points says nothing about the sealed ordering. XOR is 3.0 points ahead of Jev on public items and 2.9 behind it on sealed ones. The chart's first view is the ranking a release shows you. The second is the one that holds.

The two that publish their losses

Lev: a head-to-head it loses, on a harness that reproduces Jev

The card's headline is "68.9% on all 13 S1Bench subsets" (reported). S1Bench is 13 human-labelled public subsets, 3,880 items, pinned by Bespoke's Nimble manifests, the set cua-s1-forms covered.

expand

The arithmetic checks (measured). The 13-subset macro is 0.689 for Lev and 0.761 for Jev, 7.2 points apart. Jev wins 10 subsets. Lev wins 2, multinli (0.890 against 0.836) and helpsteer2 (0.386 against 0.341), and summeval-relevance ties at 0.358. The big gaps are paws (−12.4), vitaminc (−13.3) and summeval-consistency (−54.1). By the footnote's own bar, Lev's two wins, 5.4 and 4.5 points, are at or under the noise (reasoned).

The findings file is as candid (reported): its harness reproduces TypeSafe's published Jev figures within 0.8 points per subset, and fine-tuning caused the summeval-consistency collapse, since the frozen backbone scores 0.826 there. Mean ECE from their table is 0.115 for Lev and 0.091 for Jev (measured, recomputed).

The levbench code, pinned subset files and per-subset logs are public. Per-item predictions are not, so you can check the arithmetic but not an item. And if you search JevBench: its lev-350m is an unrelated project by Franck Verrot.

A 4B model 7.2 points behind Jev is not the news. The news is a release that measured Jev on identical items and printed the loss in its model card.

Open-Jev: somebody else's items, and the loss in the same table

This is Zefan-Cai/Open-Jev, not week one's pngwn/open-jev: MIT code, adapters for 2B and 9B, a 27B v1.1 published after the post, and a CC0 dataset of 79,116 rows (measured).

It is a per-option scorer, and the head is the interesting part (measured, jev/model.py). Each candidate becomes its own chat-templated sequence ending Is this proposed answer correct? Answer Yes or No. The last token's hidden state goes through nn.Linear(hidden, 1), and the softmax is over the K scalars. The head is initialised from lm_head[Yes] − lm_head[No], so at step zero it is the base model's Yes-minus-No readout, and training starts from a working reader rather than from noise. The loss adds 0.1 times the Brier score to soft cross-entropy.

expand

The report behind the chart (measured, docs/jevbench-public.md, audited replays committed):

Benchmark Heaven's independent run agrees on the public set and adds the sealed one: the 9B scores 77.5% public and 29.9% sealed, rank 75 of 93.

The internal numbers are home turf, since the test sets come from the project's own generators (reasoned): 97.54% test and 91.97% out-of-distribution for the 9B. On a 512-row OOD check the 2B's accuracy rose from 59.30% to 87.81% while its calibrated ECE got worse, 0.0512 to 0.0785, and the authors flag it.

Two things to know before you use it:

• Order. Candidate position never reaches the model. The caller's key name does, as Proposed answer: A: …. On JF100's three option rotations, where the letters A–D stay put and the content rotates under them, the pilot checkpoints held the same answer on 74%, 69% and 75% of items (2B, 9B, 27B), against Jev's published 85%. I attribute the gap to the visible letter (reasoned). The released checkpoints have no rotation result yet.
• Determinism. Re-running a pilot on identical inputs at batch size 1 instead of 16 changed 11, 2 and 12 of 288 answers (measured): the same class of problem Jev is not deterministic found in the hosted model.

Three items behind Jev with the 27B, 21 behind with the released 9B, ECE 0.086 against 0.032, all in the same table as the author's own model. That is the standard.

The rest, claim first

XOR (Juspay): a lead that exists only on public items

The post: "An open source, multimodal Jev-like decision model", with a "Much bigger 260k context window".

What it is (measured): juspay/jev-one redirects to juspay/xor, and the 35,107,181,936 parameters are Qwen3.6-35B-A3B's minus its 19 multi-token-prediction tensors. In sampled byte spans the attention projections and shared-expert MLP changed, while the routed experts, embeddings, LM head and the entire vision tower are byte-identical to the base: a merged LoRA (reasoned). The multimodality is Qwen3.6's untouched ViT, and the 260k is the base's max_position_embeddings of 262,144.

It reads one token's log-probabilities over A–Z (measured, from the serving tarball, read and not run), so a hard cap of 26 options where Jev takes 255. Every question runs forward and reversed, merged. One quirk: with three or more options the server applies a hard-coded T=1.5, so the documented per-type temperatures, 2.2 for Choice and 2.7 for Score, only reach two-option questions.

The README never mentions Jev. JevBench had already measured it as "JevOne (Juspay, repo juspay/jev-one)": 207 against Jev's 200 on public items, 165 against 163 on the 220-item hard tier, and 33.8% against 36.7% sealed. Its hard-tier ECE, 0.030 against Jev's 0.061, is the one axis where it is clearly better. Its composite rank is 50 of 89, dragged down mostly by the formula's penalties, one of them on an estimated cost. The row pins no revision, so I cannot prove it is tag xor-v1; the name, base and method match.

Cua-S1-4B-0.2 (trycua): the same team's opposite bet

The post: "the first multimodal decision model trained with RLOO on live computer-use tasks, using task-completion rewards."

Measured: two LoRA adapters on Qwen3.5-4B, r=16, no head tensor. Text is 21,233,664 parameters; multimodal is 25,403,392, 4,169,728 of them on the vision tower, the only release this week that trains vision weights. It is a letter readout over A–Z (MAX_OPTIONS = 26) in caller order. The same team's cua-s1-forms was a 706,048-parameter per-option scorer: same team, opposite family.

The RLOO is real (measured, training/train_4b_rl.py). The "live computer-use tasks" are six single-widget web pages (a button, an icon, a colour picker, a spreadsheet cell, a toggle, a text input) in Playwright under a simulated provider. Held-out evaluation is three episodes per page: N=18, where it reports 0.944 text and 0.722 multimodal success against djev's 0.889 and 0.667. The bench README adds, for one hard split, 0.929 multimodal with ECE 0.069 (N=168); the harness is public, but no result files are committed.

expand

The Jev comparison is that right panel, general_decision: JevBench's 231 public items recast so every label is a "Button" to click or skip, chance 0.5 (measured). Cua scores 0.887, Jev 0.667, which is 154 of 231. JevBench's own harness gives Jev 200 of the same 231, so the recast penalises Jev's API (reasoned). The model card also says 0.2 was "not measured" on five tasks the bench README reports.

NeoHorse-Jev-4B (TokenRhythm): Kev's head, to the parameter

The post: "Scores 77.70 across six text decision benchmark groups, ranking first among the four open-weight models with complete results in the comparison."

Measured: the pointer head is 1,311,232 parameters, exactly Kev-4B's as counted in Kev, and it is credited: the config's runtime_origin is github.com/jaredpalmer/kev, and the vendored NOTICE says model.py and schema.py "derive from Jared Palmer's Kev". The text tensors match TokenRhythm's NeoHorse-1-4B chat model with the projections changed, a merged LoRA (reasoned). The vision tensors are byte-identical to Qwen3.5-4B's, and the config says vision_finetuning: false.

The 77.70 recomputes exactly. The four models are Open-Jev-9B (75.67), Kev-4B (74.25), Laya English (58.24) and NeoHorse. Jev is not in the comparison. One group is Kev's own splits, which, with no training-data section, cannot be checked for overlap. No eval code is published, and the card says "NLL, Brier, and ECE calibration results have not been reported."

Kev-0.8B on Core ML (FluidInference): same answers, a different baseline

FluidInference's claim (reported): Kev-0.8B on Core ML plays "Guess Who" over 80 Wikipedia bios on an M5 Pro in 37 ms per bio against 1.07 s for the original PyTorch, 3.0 s against 99 s for all 80, in about 2 GB of memory against 6.6 GB, from 1.45 GB of weights against 1.79 GB, with the same accuracy.

A port, not a new model: Kev's LoRA folded into Qwen3.5-0.8B-Base, plus Kev's pointer head, as fp16 packages for the GPU (measured: config.json says fp16, and the embedding file is exactly 248,320 × 1,024 × 2 bytes). The fused package and embeddings the benchmark loads are 1,508,651,136 bytes, 1.51 GB or 1.41 GiB, either side of the report's 1.449 GB. The whole repository, with row fallbacks up to 1,024 tokens and 80 options, is 3.55 GB. The "about 2 GB" is a reported 0.67 GB footprint plus those weights, mapped from disk.

The report names the baseline: Kev's own loader in PyTorch on MPS, the Apple GPU, as in the port GLiNER2.5-Decide examined. The 1.07 s and 6.6 GB are its bf16 run; Kev's fp32 default is 1,287 ms and 9.5 GB. It puts most of that time in transformers' reference Gated DeltaNet code, and against Kev's own MLX server the gap narrows to 37.8 against 51.5 ms for six questions. Its Neural Engine run is slower still, 740 ms a call against 30.8 ms on the GPU (all reported).

"Same accuracy" is agreement: the Guess Who questions "have no gold labels". Core ML matches fp32 PyTorch on all 960 answers and bf16 differs on 3, in a report that gives counts, not per-item answers. Accuracy lives in Kev's dev-split reports, which I read: 0.8252 against fp32's 0.8267 on decision-v7, with ECE 0.0326 against 0.0329 (measured).

Lumma-fev (FrontiersMind): Laya's Jev column

The post: "Lumma-fev is beating Jev on several benchmarks", with 4B and 9B "tomorrow".

Measured: the 0.1B is 154,102,848 parameters (positions to 2,048), the 0.6B 649,282,476 (positions to 12,288), both on FrontiersMind's own backbones with no LM head, Kev-style pointer heads, a NOTICE crediting Jared Palmer, and up to 255 options. The 0.6B's shared_kv: true is the K = V ablation its own paper calls "not a proposed system", as Grouped Value Attention covered. The 4B and 9B are not on Hugging Face as of this check.

The benchmarks, which average as stated (measured):

The Jev and Laya columns are Laya's model card, truncated: Laya's card has Jev at 0.870, 0.480, 0.910 and 0.727, and says Jev's Banking77 figure was measured on 72 labels, not 77. FrontiersMind did not re-measure Jev (reasoned). On typed-decisions, the only decision-native set, Lumma scores 0.2 and 0.3 against 0.72. No n, no eval code, no calibration. Laya vs Jev has the background on Laya.

Solomon 27B (Doccy): a careful card, an unnamed opponent

The post: "the best open-weight alternative to Jev", which "roughly matches the intelligence of its base-model", is "Natively Multimodal" and has a "265k Context Window". It had no link. The repository is DoccyHealth/Solomon, created 2 hours 38 minutes before the post; the poster, Archer Hume, is a co-founder of Doccy and wrote the architecture analysis cited in Jev, TypeSafe's System One model.

Measured: an adapter, not weights. A LoRA of r=64 on layers 32–63 only (217,579,520 parameters), ten-slot letter heads and an evidence MLP for Qwen3.8-27B. No vision keys: images go through the base's ViT. The base's max_position_embeddings is 262,144, which "265k" misstates; the card claims no context length. Options are A–J, two to ten, enforced, through heads initialised from the LM head's rows for those letters: a letter readout with the rows retrained (reasoned).

One idea is worth copying: the adapter is off while the document is prefilled and on from the question onward, so every question branches from one cached base-model prefix, as in AgentJev-0.6B.

The claims (reported, the card; arithmetic measured):

• On an in-house panel of 802 questions over 54 documents: Solomon 706 (88.0%), an "external reference" 690 (86.0%), +2.0 points. The reference is "a commercial structured-decision API", never named. The card's opening box says v1.1's gain over v1.0 is "not statistically established".
• On 800 MMLU and MMLU-Pro items: Solomon 72.9%, its base 71.8%, the reference 87.1%, −14.2 points. "Roughly matches its base" holds.
• The labels are AI-generated and "have not been reviewed by a human". Fitted temperatures made held-out ECE worse in 8 of 10 cells, so it ships at T=1.

localjev (GitHub Next): the wire format, without the readout

The post calls it "a poor man's Jev". It is MIT TypeScript exposing POST /v1/systemone, so the official SDK works against it (measured).

It is not a readout. One chat completion per chunk of up to 16 questions, strict JSON at temperature 0: the model writes its probabilities. The README says so: "wire-compatible, but not mathematically equivalent to OpenJev's logit read. The probabilities are generated/self-reported." That is the line Any model can be Jev drew between reading a probability off the logits and asking for one.

The bake-off is five models on 1,200 requests on an M5 Max, with macro accuracy from 45.8% to 76.7% and ECE from 0.032 to 0.544 (measured). There is no Jev row: "this is not a Jev-vs-model benchmark". The per-row file is the useful part. On BoolQ, DiffusionGemma put at least 0.999 on its answer for all 40 items, and each of its 5 wrong answers gave the truth probability 0. Qwen did the same on 37 of 40. On yes/no, these models return 0 or 1, not probabilities (reasoned), and the repo says it first: "Do not treat these outputs as calibrated probabilities."

Span-01 (Respan): relative percentages and a borrowed ratio

Closed: no weights, no disclosed size or architecture, a waitlisted API at $0.02 per million input tokens with free output (measured). Its post's claims, against Respan's own chart data, read from the page's JavaScript bundle:

The benchmark was published two days before launch, labelled where two frontier models agree; its card says "These are model labels, not ground truth." Span-01's RLAIF source is undisclosed, so whether this is home turf is not checkable.

Caps, windows and borrowed eyes

Option caps. Jev's Choice takes up to 255 options. A letter readout stops at its alphabet: XOR and Cua at 26, Solomon at 10. Lumma's pointer takes 255, and Lev switches family when its codes run out. A routing table with 40 destinations cannot be asked of three of these at all.

Context. XOR's "260k" and Solomon's "265k" are both their bases' max_position_embeddings, 262,144 (measured). NeoHorse inherits that window and caps the state at 2,048 tokens and the total at 32,768.

Multimodal. XOR's vision tower is byte-identical to its base's in every span sampled, NeoHorse's to Qwen3.5-4B's, and Solomon's adapter has no vision keys. All three take images because their bases do. Cua's 4,169,728 vision parameters are the week's only trained ones.

ImageJevBench, "Coming soon" on 23 September, is live as v0.1: 684 scored items, 228 public and 456 sealed, 12 systems (measured). Jev-Omni leads at 73.10. Jev cannot enter: "no native image field". Cua-S1-4B-0.2 and NeoHorse are "requested, not yet evaluated", so no multimodal claim this week has a third-party image number yet.

The recipe's fixed slot

@neural_avb posted "Simply DSPy code to generate choice-based JEV decision training data from any raw text source", saying you can "produce 50K high quality examples in ~5-10$". The code exists only as a screenshot. It asks gpt-6-luna for questions with one correct answer and 2–10 distractors, and writes each passage as a Kev fine-tuning row with to_kev:

# @neural_avb's to_kev(), transcribed from the post's screenshot (trimmed)
for index, item in enumerate(generation):
    options = [item.correct_answer, *item.distractors]
    keys = [option.answer_key for option in options]
    random.Random(42).shuffle(options)
    criteria = {option.answer_key: option.answer for option in options}

random.Random(42) builds a fresh generator with the same seed for every question, and the same seed shuffles a list of the same length the same way. The correct answer starts at index 0, so it always lands in the same place.

With CPython 3.11 (measured): 3 options put the gold at B, 4 at D, 5 and 6 at E, 7 at F, 8 at G, 9 at H, 10 and 11 at I. criteria keeps the shuffled order, so the gold's position is a function of the option count alone. A model that sees option order can learn that table instead of the passage (reasoned). That includes Kev at its defaults, --option_isolation 0 and --perm_kl 0.0 (measured, kev/train.py), the format to_kev writes. It is harmless for an isolated per-option scorer like Open-Jev. The fix is to seed once, outside the loop.

On cost (reasoned, at OpenRouter's $0.10 in and $0.50 out per million tokens for GPT-6 Luna, a 1,000-token passage and five questions each): 50K single questions come to about $6.55, and 50K to_kev rows, which are passages, about $32.75. The claim holds only if an "example" is one question and the chain of thought stays short.

If you are choosing this week

All reasoned, from the evidence above:

• A number you can check: Lev (per-subset logs, pinned items) and Open-Jev (audited per-item replays). Both lose to Jev, and both say so.
• Order invariance: Open-Jev, the only per-option scorer. Position cannot reach it; the key name can, so do not letter your keys.
• More than 26 options: not XOR or Cua. Solomon stops at 10.
• A sealed-item score: of this week's releases, XOR and Open-Jev are the ones I found on JevBench's sealed tier, and both sit below Jev there.
• Images: only Cua trained vision weights, and none has an independent image score.

A correction to an earlier piece

Reproducing Jev said of JevBench v1.3.0 "Nobody in the table trained on it." On v1.4.2, first place goes to decider-4b v2, which says "8,000 of the v2 LoRA rows come from generators written from the published names of the ten sealed families": not training on the items, but training aimed at them. That article now carries a dated note.

The take

Week one had code and almost no numbers. Week two has numbers, mostly on items anyone can read. The biggest change is in the evaluation layer: JevBench sealed 308 items, and on those a 3-point public lead became a 2.9-point deficit.

The releases worth trusting measured Jev themselves, on the same items, and printed the result when it went against them. This week that was two posts of twelve.

Sources, read on 2026-09-26: interfaze-ai/lev; juspay/xor; TokenRhythm/NeoHorse-Jev-4B; FrontiersMind's Lumma collection; DoccyHealth/Solomon; cua-ai/cua-s1-4b-0.2 and trycua/cua; Zefan-Cai/Open-Jev and its dataset; githubnext/localjev; Respan's launch post and benchmark; the JevBench board and Image JevBench. The DSPy recipe is the screenshot attached to @neural_avb's post. No third-party code was executed. Figures are reproduced for commentary from Lev's model card (Apache-2.0), Open-Jev's repository (MIT) and trycua's cua-bench-s1 (MIT); the three interactives are original.