# Jev alternatives, week two: the numbers arrive, and most sit on public items

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/jev-alternatives-week-two
> date: 2026-09-26
> tags: explainer, llm, ecosystem, benchmarks, calibration, architecture

[Week one](/articles/jev-ecosystem) was a census of the code people wrote
against Jev in its first nine days. It had one shape: a cheap, typed
classifier in front of something slower or more expensive. Four of the
fourteen projects I could read reported a measured benchmark.

Week two is different in kind. The eleven posts in this roundup are seven
open-weight decision models, a local server that speaks Jev's wire format, a
closed classifier, a benchmark that went live, and a recipe for training data;
a twelfth, a Core ML port of Kev, arrived late. This time there are numbers,
several of them against Jev.

Most of those comparisons stand on the same ground: the **231 public items** of
JevBench, Benchmark Heaven's third-party decision benchmark. Three of the new
releases report a score on them. None puts its number beside the others', and
only one puts it beside Jev's, in a reformatted test. A fourth, Open-Jev, ran
them on purpose, against Jev. JevBench has since added **308 sealed items**
nobody outside can see, and on those a public lead over Jev turns into a
deficit.

Two releases did the thing this site keeps asking for. **Lev publishes a
head-to-head it loses. Open-Jev ran somebody else's items and published the
loss.**

Parameter counts below are safetensors header sums, not the cards' round
numbers. Code was read, never run. **Measured** means read from a file or
recomputed from published rows; **reported** means the author's claim, which I
could not check; **reasoned** means my inference. All checked on 2026-09-26.

## Which family each one is

The first question to ask of a decision model is not its score. It is how it
turns a list of options into probabilities, because that decides what can go
wrong. This site has used three families since
[cua-s1-forms](/articles/cua-s1-forms) and [Kev](/articles/kev):

- **Letter (vocabulary) readout.** Options go into the prompt as letters, and
  the answer is the letter tokens' logits. `A` and `B` carry different priors,
  so order matters, and the alphabet caps the option count.
- **Per-option scorer.** Each option is encoded alone and scored to one number;
  the softmax is over those numbers. No option sees another, so order cannot
  matter.
- **Pointer.** Options are spans in one shared sequence and a head points at
  one. Options see each other, and order matters again. This is Kev's family.

From each release's code (measured): **XOR**, **Cua-S1-4B-0.2** and **Solomon**
are letter readouts. **Open-Jev** is a per-option scorer. **NeoHorse-Jev-4B**
and **Lumma-fev** are pointers, and both credit Kev. **localjev** is none of
them: the model writes its probabilities into JSON. **Span-01** is closed.

**Lev** is both. Mode A is a letter readout over single-token label codes, read
in two option orders and averaged. Mode B takes over when the options outrun
those codes: a per-option head whose candidates attend to each other through a
`set_attention` layer before scoring, so it is not isolated. It is the only
release this week that does both.

<ReleaseMatrix />

## The yardstick nobody names

JevBench is one author's benchmark, run from a server in Germany one request at
a time; version 1.4.2, scored 24 September, has 93 systems. Everyone who
reported on its 231 public items this week:

| System | Correct of 231 | Who ran it |
|---|---|---|
| Jev 1.13.0 | 200 | JevBench |
| XOR | 207 | JevBench, as the row "JevOne" |
| XOR | 204 | Juspay: 48/48 Easy, 70/72 Original, 86/111 Hard-public |
| Cua-S1-4B-0.2 | about 205 (0.887) | trycua, recast as click-or-skip; Jev scores 154 in that format |
| NeoHorse-Jev-4B | 174 | TokenRhythm's own harness |
| Open-Jev 27B v1.1 | 197 | Zefan Cai, audited replays |
| Open-Jev 9B | 179 | Zefan Cai, audited replays |
| Open-Jev 2B | 150 | Zefan Cai, audited replays |

For scale, JevBench's own run puts reflex-4b at 183 and Kev-4B at 153. A
self-run on a different harness is not a rank.

Public items are fine for development and weak for claims. Anyone can tune
against them or generate training data that resembles them. Nobody here says
they did, and none of the numbers above can tell you. That is what v1.4's
**308 sealed items** are for: only the benchmark sees them.

<PublicSealed />

What the sealed tier shows (measured; I read 14 of the board's 93 rows):

- **Every decision model I read loses a lot.** Jev goes from 86.6% to 36.7%, a
  49.9-point drop. decider-4b v2 goes from 83.5% to 34.7%, Cygnet from 87.9% to
  33.8%, XOR's row from 89.6% to 33.8%, Open-Jev 9B from 77.5% to 29.9%, and
  Open-Jev 2B from 64.5% to 26.3%. The median public-minus-sealed gap across
  the 89 ranked systems is 45.8 points.
- **GPT-6 Luna, a frontier chat model, goes from 99.6% to 95.5%**: 4.1 points.

The obvious reading of that drop is wrong. It is not evidence that anybody
trained on the public items: Jev drops 49.9 points, and Jev is the control. The
sealed families are much harder for these decision models than for a frontier
chat model (reasoned, from Luna's 4.1). What the drop does show is that a public
lead of a few points says nothing about the sealed ordering. **XOR is 3.0 points
ahead of Jev on public items and 2.9 behind it on sealed ones.** The chart's
first view is the ranking a release shows you. The second is the one that holds.

## The two that publish their losses

### Lev: a head-to-head it loses, on a harness that reproduces Jev

The card's headline is **"68.9% on all 13 S1Bench subsets"** (reported).
S1Bench is 13 human-labelled public subsets, 3,880 items, pinned by Bespoke's
Nimble manifests, the set [cua-s1-forms](/articles/cua-s1-forms) covered.

<ModelCard repo="interfaze-ai/lev" note="A LoRA (r=32, 42,467,328 parameters) plus a 3,673,600-parameter candidate head for Qwen3.5-4B, counted from the safetensors header and from a pickletools disassembly of mode_b_head.pt, which was not loaded. The 4B is the base: 4,659,865,088 parameters." />

<Figure
  src="/articles/jev-alternatives-week-two/fig1.png"
  alt="Horizontal bar chart titled Accuracy by subset, Lev in blue against Jev in orange on all 13 S1Bench subsets, 3,880 task items, same harness, same laptop. Jev's bar is longer on most rows: vitaminc-dev 66.8 against 80.1 percent, paws 77.6 against 90.0, boolq 82.7 against 89.3. Lev is ahead on multinli, 89.0 against 83.6, and helpsteer2, 38.6 against 34.1. summeval-relevance is tied at 35.8. The largest gap is summeval-consistency, where Lev scores 27.1 percent and Jev 81.2. A footnote says a per-subset difference needs roughly 5 to 9 points to clear sampling noise, and that on civil_comments, helpsteer2 and both summeval subsets always answering the most common label beats both models."
  caption="Lev against Jev on all 13 S1Bench subsets, same harness, same laptop. Jev is ahead on 10, and the chart's own footnote says a per-subset difference needs roughly 5–9 points to clear sampling noise (interfaze-ai/lev model card, 'Accuracy by subset')."
/>

The arithmetic checks (measured). The 13-subset macro is **0.689 for Lev and
0.761 for Jev**, 7.2 points apart. Jev wins 10 subsets. Lev wins 2, multinli
(0.890 against 0.836) and helpsteer2 (0.386 against 0.341), and
summeval-relevance ties at 0.358. The big gaps are paws (−12.4), vitaminc
(−13.3) and summeval-consistency (−54.1). By the footnote's own bar, Lev's two
wins, 5.4 and 4.5 points, are at or under the noise (reasoned).

The findings file is as candid (reported): its harness reproduces TypeSafe's
published Jev figures within 0.8 points per subset, and fine-tuning caused the
summeval-consistency collapse, since the frozen backbone scores 0.826 there. Mean ECE from their
table is 0.115 for Lev and 0.091 for Jev (measured, recomputed).

The `levbench` code, pinned subset files and per-subset logs are public.
Per-item predictions are not, so you can check the arithmetic but not an item.
And if you search JevBench: its `lev-350m` is an unrelated project by Franck
Verrot.

A 4B model 7.2 points behind Jev is not the news. The news is a release that
measured Jev on identical items and printed the loss in its model card.

### Open-Jev: somebody else's items, and the loss in the same table

This is `Zefan-Cai/Open-Jev`, not week one's `pngwn/open-jev`: MIT code,
adapters for 2B and 9B, a 27B v1.1 published after the post, and a CC0 dataset
of 79,116 rows (measured).

It is a per-option scorer, and the head is the interesting part (measured,
`jev/model.py`). Each candidate becomes its own chat-templated sequence ending
`Is this proposed answer correct? Answer Yes or No.` The last token's hidden
state goes through `nn.Linear(hidden, 1)`, and the softmax is over the K
scalars. **The head is initialised from `lm_head[Yes] − lm_head[No]`**, so at step
zero it *is* the base model's Yes-minus-No readout, and training starts from a
working reader rather than from noise. The loss adds 0.1 times the Brier score
to soft cross-entropy.

<ModelCard repo="ZefanCai/Open-Jev-9B" note="A rank-8 LoRA (5,898,240 parameters, fp32) and a 4,097-parameter head, Linear(hidden, 1), initialised from lm_head[Yes] minus lm_head[No]. The 2B's adapter is 2,506,752 and the 27B v1.1's 15,466,496. Base weights are not included; the Qwen revisions are pinned." />

<Figure
  src="/articles/jev-alternatives-week-two/fig2.png"
  alt="Dark poster headed Open-Jev 27B v1.1, One hard answer behind Jev. Two bar panels. All public tasks: Open-Jev-2B 150 of 231, Open-Jev-9B 179 of 231, Open-Jev 27B v1.1 197 of 231, Jev 1.13.0 200 of 231. Public hard tier: 46, 66, 80 and 81 of 111. Footer: public subset of 231 tasks including 111 Hard; the full JevBench has 534 tasks; quality scores only, model sizes and training differ, no speed or cost comparison."
  caption="Open-Jev's JevBench public run, as its author charts it: every Open-Jev bar is shorter than Jev's. The report behind it also lists GPT-5.6 Luna at 206 and GPT-6 Astra at 231 of 231, which the chart leaves out (Zefan-Cai/Open-Jev, site/v1-1/benchmark-poster.png)."
/>

The report behind the chart (measured, `docs/jevbench-public.md`, audited
replays committed):

| System | Correct / 231 | Hard / 111 | Brier | ECE |
|---|---|---|---|---|
| Open-Jev 2B | 150 | 46 | 0.475 | 0.127 |
| Open-Jev 9B | 179 | 66 | 0.322 | 0.086 |
| Open-Jev 27B v1.1 | 197 | 80 | 0.242 | not exported |
| Jev 1.13.0 | 200 | 81 | 0.181 | 0.032 |

Benchmark Heaven's independent run agrees on the public set and adds the sealed
one: the 9B scores 77.5% public and 29.9% sealed, rank 75 of 93.

The internal numbers are home turf, since the test sets come from the project's
own generators (reasoned): 97.54% test and 91.97% out-of-distribution for the
9B. On a 512-row OOD check the 2B's accuracy rose from 59.30% to 87.81% while its
calibrated **ECE got worse, 0.0512 to 0.0785**, and the authors flag it.

Two things to know before you use it:

- **Order.** Candidate position never reaches the model. The caller's key name
  does, as `Proposed answer: A: …`. On JF100's three option rotations, where the
  letters A–D stay put and the content rotates under them, the *pilot*
  checkpoints held the same answer on 74%, 69% and 75% of items (2B, 9B, 27B),
  against Jev's published 85%. I attribute the gap to the visible letter
  (reasoned). The released checkpoints have no rotation result yet.
- **Determinism.** Re-running a pilot on identical inputs at batch size 1
  instead of 16 changed 11, 2 and 12 of 288 answers (measured): the same class of
  problem [Jev is not deterministic](/articles/jev-is-not-deterministic) found
  in the hosted model.

Three items behind Jev with the 27B, 21 behind with the released 9B, ECE 0.086
against 0.032, all in the same table as the author's own model. That is the
standard.

## The rest, claim first

### XOR (Juspay): a lead that exists only on public items

The post: "An open source, multimodal Jev-like decision model", with a "Much
bigger 260k context window".

<ModelCard repo="juspay/xor" note="Full merged BF16 weights in two shards, 35,107,181,936 parameters: Qwen3.6-35B-A3B's tensors minus its 19 multi-token-prediction tensors, with no new head tensor. juspay/jev-one redirects here." />

What it is (measured): `juspay/jev-one` redirects to `juspay/xor`, and the
35,107,181,936 parameters are Qwen3.6-35B-A3B's minus its 19 multi-token-prediction
tensors. In sampled byte spans the attention projections and shared-expert MLP
changed, while the routed experts, embeddings, LM head and **the entire vision
tower** are byte-identical to the base: a merged LoRA (reasoned). The
multimodality is Qwen3.6's untouched ViT, and the 260k is the base's
`max_position_embeddings` of 262,144.

It reads one token's log-probabilities over `A`–`Z` (measured, from the serving
tarball, read and not run), so **a hard cap of 26 options where Jev takes 255**.
Every question runs forward and reversed, merged. One quirk: with three or more
options the server applies a hard-coded T=1.5, so the documented per-type
temperatures, 2.2 for Choice and 2.7 for Score, only reach two-option questions.

The README never mentions Jev. JevBench had already measured it as "JevOne
(Juspay, repo juspay/jev-one)": 207 against Jev's 200 on public items, 165
against 163 on the 220-item hard tier, and **33.8% against 36.7% sealed**. Its
hard-tier ECE, **0.030 against Jev's 0.061**, is the one axis where it is clearly
better. Its composite rank is 50 of 89, dragged down mostly by the formula's
penalties, one of them on an estimated cost. The row pins no revision, so I cannot prove it is tag `xor-v1`; the
name, base and method match.

### Cua-S1-4B-0.2 (trycua): the same team's opposite bet

The post: "the first multimodal decision model trained with RLOO on live
computer-use tasks, using task-completion rewards."

Measured: two LoRA adapters on Qwen3.5-4B, r=16, no head tensor. Text is
21,233,664 parameters; multimodal is 25,403,392, **4,169,728 of them on the
vision tower**, the only release this week that trains vision weights. It is a
letter readout over `A`–`Z` (`MAX_OPTIONS = 26`) in caller order. The same
team's cua-s1-forms was a 706,048-parameter per-option scorer: same team,
opposite family.

The RLOO is real (measured, `training/train_4b_rl.py`). The "live computer-use tasks" are **six single-widget web pages** (a button, an
icon, a colour picker, a spreadsheet cell, a toggle, a text input) in Playwright
under a `simulated` provider. Held-out evaluation is three episodes per page:
N=18, where it reports 0.944 text and 0.722 multimodal success against djev's
0.889 and 0.667. The bench README adds, for one hard split, 0.929 multimodal
with ECE 0.069 (N=168); the harness is public, but no result files are
committed.

<Figure
  src="/articles/jev-alternatives-week-two/fig3.png"
  alt="Two-panel bar chart. Left, 6 core GUI families, text, hard cross-dataset split, N=615: consent_checkbox, form_filling, login_auth, multi_step_submit, pagination and search_filter, with bars for jev, djev, semif, cua-s1-nano-0.1, cua-s1-4b-0.1 and cua-s1-4b-0.2; cua-s1-4b-0.2 is tallest on five of six, and jev's grey bars are near zero on most. Right, general_decision, external jevbench, N=231: jev about 0.67, djev about 0.62, semif about 0.56, cua-s1-4b-0.1 about 0.63, cua-s1-4b-0.2 about 0.89."
  caption="trycua's results chart. The right panel is not a JevBench score: it is JevBench's 231 public items recast into cua's click-or-skip format, where Jev scores 0.667; on JevBench's own harness Jev scores 200 of 231 on the same items (trycua/cua, libs/cua-bench-s1 results_chart.png)."
/>

The Jev comparison is that right panel, `general_decision`: JevBench's 231
public items recast so every label is a "Button" to click or skip, chance 0.5
(measured). Cua scores 0.887, Jev 0.667, which is 154 of 231. JevBench's own
harness gives Jev 200 of the same 231, so the recast penalises Jev's API
(reasoned). The model card also says 0.2 was "not measured" on five tasks the
bench README reports.

### NeoHorse-Jev-4B (TokenRhythm): Kev's head, to the parameter

The post: "Scores 77.70 across six text decision benchmark groups, ranking
first among the four open-weight models with complete results in the
comparison."

<ModelCard repo="TokenRhythm/NeoHorse-Jev-4B" note="4,540,576,768 parameters: a Qwen3.5-4B-derived backbone with no LM head, an untouched Qwen3.5-4B vision tower (333,514,240) and a 1,311,232-parameter pointer head, the same size as Kev-4B's." />

Measured: the pointer head is **1,311,232 parameters, exactly Kev-4B's** as
counted in [Kev](/articles/kev), and it is credited: the config's
`runtime_origin` is `github.com/jaredpalmer/kev`, and the vendored NOTICE says
`model.py` and `schema.py` "derive from Jared Palmer's Kev". The text tensors match
TokenRhythm's NeoHorse-1-4B chat model with the projections changed, a merged
LoRA (reasoned). The vision tensors are byte-identical to Qwen3.5-4B's, and the
config says `vision_finetuning: false`.

The 77.70 recomputes exactly. The four models are Open-Jev-9B (75.67), Kev-4B
(74.25), Laya English (58.24) and NeoHorse. **Jev is not in the comparison.**
One group is Kev's own splits, which, with no training-data section, cannot be
checked for overlap. No eval code is published, and the card says
"NLL, Brier, and ECE calibration results have not been reported."

### Kev-0.8B on Core ML (FluidInference): same answers, a different baseline

FluidInference's claim (reported): [Kev](/articles/kev)-0.8B on Core ML plays
"Guess Who" over 80 Wikipedia bios on an M5 Pro in 37 ms per bio against 1.07 s
for the original PyTorch, 3.0 s against 99 s for all 80, in about 2 GB of memory
against 6.6 GB, from 1.45 GB of weights against 1.79 GB, with the same accuracy.

A port, not a new model: Kev's LoRA folded into Qwen3.5-0.8B-Base, plus
Kev's pointer head, as fp16 packages for the GPU (measured: `config.json` says
fp16, and the embedding file is exactly 248,320 × 1,024 × 2 bytes). The fused
package and embeddings the benchmark loads are 1,508,651,136 bytes, 1.51 GB or
1.41 GiB, either side of the report's 1.449 GB. The whole repository, with row
fallbacks up to 1,024 tokens and 80 options, is 3.55 GB. The "about 2 GB" is a
reported 0.67 GB footprint plus those weights, mapped from disk.

The report names the baseline: Kev's own loader in PyTorch **on MPS**, the Apple
GPU, as in the port [GLiNER2.5-Decide](/articles/gliner-2-5-decide) examined.
The 1.07 s and 6.6 GB are its bf16 run; Kev's fp32 default is 1,287 ms and
9.5 GB. It puts most of that time in transformers' reference Gated DeltaNet
code, and against Kev's own MLX server the gap narrows to 37.8 against 51.5 ms
for six questions. Its Neural Engine run is slower still, 740 ms a call against
30.8 ms on the GPU (all reported).

"Same accuracy" is agreement: the Guess Who questions "have no gold labels".
Core ML matches fp32 PyTorch on all 960 answers and bf16 differs on 3, in a
report that gives counts, not per-item answers. Accuracy lives in Kev's
dev-split reports, which I read: 0.8252 against fp32's 0.8267 on decision-v7,
with ECE 0.0326 against 0.0329 (measured).

### Lumma-fev (FrontiersMind): Laya's Jev column

The post: "Lumma-fev is beating Jev on several benchmarks", with 4B and 9B
"tomorrow".

Measured: the 0.1B is 154,102,848 parameters (positions to 2,048), the 0.6B
649,282,476 (positions to 12,288), both on FrontiersMind's own backbones with
no LM head, Kev-style pointer heads, a NOTICE crediting Jared Palmer, and up to
255 options. The 0.6B's `shared_kv: true` is the `K = V` ablation its own paper
calls "not a proposed system", as
[Grouped Value Attention](/articles/grouped-value-attention) covered. The 4B
and 9B are not on Hugging Face as of this check.

The benchmarks, which average as stated (measured):

| | Jev | Laya | 0.15B | 0.6B |
|---|---|---|---|---|
| Banking77 | 0.87 | 0.425 | 0.47 | 0.90 |
| DAIR Emotion | 0.48 | 0.595 | 0.68 | 0.89 |
| AG News | 0.91 | 0.95 | 0.89 | 0.85 |
| typed-decisions | 0.72 | 0.76 | 0.2 | 0.3 |
| average | 0.745 | 0.683 | 0.56 | 0.735 |

**The Jev and Laya columns are Laya's model card**, truncated: Laya's card has
Jev at 0.870, 0.480, 0.910 and 0.727, and says Jev's Banking77 figure was
measured on 72 labels, not 77. FrontiersMind did not re-measure Jev (reasoned).
On typed-decisions, the only decision-native set, Lumma scores 0.2 and 0.3
against 0.72. No n, no eval code, no calibration. [Laya vs Jev](/articles/laya-vs-jev)
has the background on Laya.

### Solomon 27B (Doccy): a careful card, an unnamed opponent

The post: "the best open-weight alternative to Jev", which "roughly matches the
intelligence of its base-model", is "Natively Multimodal" and has a "265k
Context Window". It had no link. The repository is `DoccyHealth/Solomon`,
created 2 hours 38 minutes before the post; the poster, Archer Hume, is a
co-founder of Doccy and wrote the architecture analysis cited in
[Jev, TypeSafe's System One model](/articles/jev-system-one-models).

Measured: an adapter, not weights. A LoRA of r=64 on **layers 32–63 only**
(217,579,520 parameters), ten-slot letter heads and an evidence MLP for
Qwen3.8-27B. No vision keys: images go through the base's ViT. The base's `max_position_embeddings` is 262,144, which "265k" misstates;
the card claims no context length. Options are `A`–`J`, **two to ten,
enforced**, through heads initialised from the LM head's rows for those letters:
a letter readout with the rows retrained (reasoned).

One idea is worth copying: the adapter is **off while the document is
prefilled and on from the question onward**, so every question branches from
one cached base-model prefix, as in [AgentJev-0.6B](/articles/agent-jev-06b).

The claims (reported, the card; arithmetic measured):

- On an in-house panel of 802 questions over 54 documents: Solomon 706 (88.0%),
  an "external reference" 690 (86.0%), **+2.0 points**. The reference is "a
  commercial structured-decision API", never named. The card's opening box says
  v1.1's gain over v1.0 is "not statistically established".
- On 800 MMLU and MMLU-Pro items: Solomon 72.9%, its base 71.8%, the reference
  87.1%, **−14.2 points**. "Roughly matches its base" holds.
- The labels are AI-generated and "have not been reviewed by a human". Fitted
  temperatures made held-out ECE worse in 8 of 10 cells, so it ships at T=1.

### localjev (GitHub Next): the wire format, without the readout

The post calls it "a poor man's Jev". It is MIT TypeScript exposing
`POST /v1/systemone`, so the official SDK works against it (measured).

It is not a readout. One chat completion per chunk of up to 16 questions,
strict JSON at temperature 0: the model **writes** its probabilities. The
README says so: "wire-compatible, but not mathematically equivalent to OpenJev's
logit read. The probabilities are generated/self-reported." That is the line
[Any model can be Jev](/articles/any-model-can-be-jev) drew between reading a
probability off the logits and asking for one.

The bake-off is five models on 1,200 requests on an M5 Max, with macro
accuracy from 45.8% to 76.7% and ECE from 0.032 to 0.544 (measured). There is
no Jev row: "this is not a Jev-vs-model benchmark".
The per-row file is the useful part. On BoolQ, **DiffusionGemma put at least
0.999 on its answer for all 40 items, and each of its 5 wrong answers gave the
truth probability 0.** Qwen did the same on 37 of 40. On yes/no, these models
return 0 or 1, not probabilities (reasoned), and the repo says it first: "Do not
treat these outputs as calibrated probabilities."

### Span-01 (Respan): relative percentages and a borrowed ratio

Closed: no weights, no disclosed size or architecture, a waitlisted API at
\$0.02 per million input tokens with free output (measured). Its post's claims,
against Respan's own chart data, read from the page's JavaScript bundle:

| Claim | Respan's own numbers |
|---|---|
| "2x cheaper, 18% better than Jev" | F1 0.843 against 0.715: 18% relative, **12.8 points**. On Respan's production table, 0.806 against 0.716. The 2x is \$0.042 against \$0.02 per input token, not per decision. |
| "700x cheaper, 4% better than GPT-6 Luna" | The chart prices Luna at 0.6, so **30x**. 700x is GPT-5.6 Terra's ratio, 14 over 0.02. Against Luna the gap is 2.8 points, 3.4% relative. |
| "#1 on Behavior Benchmark" | True on Respan's benchmark. On its production table GPT-6 Sol leads, 0.885 against 0.806. |

The benchmark was published two days before launch, labelled where two
frontier models agree; its card says "These are model labels, not ground
truth." Span-01's RLAIF source is undisclosed, so whether this is home turf is
not checkable.

## Caps, windows and borrowed eyes

**Option caps.** Jev's Choice takes up to 255 options. A letter readout stops at
its alphabet: XOR and Cua at 26, Solomon at 10. Lumma's pointer takes 255, and
Lev switches family when its codes run out. A routing table with 40
destinations cannot be asked of three of these at all.

**Context.** XOR's "260k" and Solomon's "265k" are both their bases'
`max_position_embeddings`, 262,144 (measured). NeoHorse inherits that window and
caps the state at 2,048 tokens and the total at 32,768.

**Multimodal.** XOR's vision tower is byte-identical to its base's in every span
sampled, NeoHorse's to Qwen3.5-4B's, and Solomon's adapter has no vision keys.
All three take images because their bases do. Cua's 4,169,728 vision parameters
are the week's only trained ones.

ImageJevBench, "Coming soon" on 23 September, is live as v0.1: 684 scored items,
228 public and 456 sealed, 12 systems (measured). [Jev-Omni](/articles/jev-omni)
leads at 73.10. **Jev cannot enter**: "no native image field". Cua-S1-4B-0.2 and
NeoHorse are "requested, not yet evaluated", so no multimodal claim this week
has a third-party image number yet.

## The recipe's fixed slot

@neural_avb posted "Simply DSPy code to generate choice-based JEV decision
training data from any raw text source", saying you can "produce 50K high
quality examples in ~5-10\$". The code exists only as a screenshot. It asks
`gpt-6-luna` for questions with one correct answer and 2–10 distractors, and
writes each passage as a Kev fine-tuning row with `to_kev`:

```python
# @neural_avb's to_kev(), transcribed from the post's screenshot (trimmed)
for index, item in enumerate(generation):
    options = [item.correct_answer, *item.distractors]
    keys = [option.answer_key for option in options]
    random.Random(42).shuffle(options)
    criteria = {option.answer_key: option.answer for option in options}
```

`random.Random(42)` builds a fresh generator with the same seed for every
question, and the same seed shuffles a list of the same length the same way.
The correct answer starts at index 0, so it always lands in the same place.

<FixedSlot />

With CPython 3.11 (measured): 3 options put the gold at B, 4 at D, 5 and 6 at E,
7 at F, 8 at G, 9 at H, 10 and 11 at I. `criteria` keeps the shuffled order, so
the gold's position is a function of the option count alone. A model that sees
option order can learn that table instead of the passage (reasoned). That
includes Kev at its defaults, `--option_isolation 0` and `--perm_kl 0.0`
(measured, `kev/train.py`), the format `to_kev` writes. It is harmless for an
isolated per-option scorer like Open-Jev. The fix is to seed once, outside the
loop.

On cost (reasoned, at OpenRouter's \$0.10 in and \$0.50 out per million tokens
for GPT-6 Luna, a 1,000-token passage and five questions each): 50K single
questions come to about \$6.55, and 50K `to_kev` rows, which are passages, about
\$32.75. The claim holds only if an "example" is one question and the chain of
thought stays short.

## If you are choosing this week

All reasoned, from the evidence above:

- **A number you can check:** Lev (per-subset logs, pinned items) and Open-Jev
  (audited per-item replays). Both lose to Jev, and both say so.
- **Order invariance:** Open-Jev, the only per-option scorer. Position cannot
  reach it; the key name can, so do not letter your keys.
- **More than 26 options:** not XOR or Cua. Solomon stops at 10.
- **A sealed-item score:** of this week's releases, XOR and Open-Jev are the
  ones I found on JevBench's sealed tier, and both sit below Jev there.
- **Images:** only Cua trained vision weights, and none has an independent
  image score.

## A correction to an earlier piece

[Reproducing Jev](/articles/reproducing-jev) said of JevBench v1.3.0 "Nobody
in the table trained on it." On v1.4.2, first place goes to decider-4b v2, which
says "8,000 of the v2 LoRA rows come from generators written from the published
names of the ten sealed families": not training on the items, but training
aimed at them. That article now carries a dated note.

## The take

Week one had code and almost no numbers. Week two has numbers, mostly on items
anyone can read. The biggest change is in the evaluation layer: JevBench sealed
308 items, and on those a 3-point public lead became a 2.9-point deficit.

The releases worth trusting measured Jev themselves, on the same items, and
printed the result when it went against them. This week that was two posts of
twelve.

---

*Sources, read on 2026-09-26: [interfaze-ai/lev](https://huggingface.co/interfaze-ai/lev); [juspay/xor](https://huggingface.co/juspay/xor); [TokenRhythm/NeoHorse-Jev-4B](https://huggingface.co/TokenRhythm/NeoHorse-Jev-4B); [FrontiersMind's Lumma collection](https://huggingface.co/collections/FrontiersMind/lumma-decision-models); [DoccyHealth/Solomon](https://huggingface.co/DoccyHealth/Solomon); [cua-ai/cua-s1-4b-0.2](https://huggingface.co/cua-ai/cua-s1-4b-0.2) and [trycua/cua](https://github.com/trycua/cua); [Zefan-Cai/Open-Jev](https://github.com/Zefan-Cai/Open-Jev) and [its dataset](https://huggingface.co/datasets/ZefanCai/Open-Jev); [githubnext/localjev](https://github.com/githubnext/localjev); Respan's [launch post](https://www.respan.ai/blog/introducing-span-1) and [benchmark](https://huggingface.co/datasets/respanai/behavior-benchmark); the [JevBench board](https://benchmarkheaven.com/jev-models) and [Image JevBench](https://benchmarkheaven.com/image-jev-bench). The DSPy recipe is the screenshot attached to @neural_avb's post. No third-party code was executed. Figures are reproduced for commentary from Lev's model card (Apache-2.0), Open-Jev's repository (MIT) and trycua's cua-bench-s1 (MIT); the three interactives are original.*