# Jev scores zero, and zero is the informative number
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/jev-scores-zero
> date: 2026-09-19
> tags: explainer, llm, architecture, calibration
[Paras Chopra benchmarked](https://gist.github.com/paraschopra/48c69b7edb99cd15137524379cceba3a)
Jev against Laya and a 4-bit Qwen3-4B prototype across fifteen tasks — 120 navigation
routes and 4,400 decision cases. Jev takes thirteen of the fifteen, often by a lot.
Then there is row eight.
**Zero of one hundred.** On a suite where the same model scores 100% on rule/evidence
judgment, 100% on ordinal scoring, and 98% on ARC-Challenge.
## Why zero is worth more than any other number in the table
A model that finds a task hard scores *badly*, and badly has a shape. On a multiple-choice
task it floors out around chance — a third, a fifth, something — and scatters. Getting
**none** right across a hundred attempts is not the bottom of that scatter. It is a
different event.
Put the arithmetic on it. If the task is three-way and the model is guessing, the number
of correct answers is $\mathrm{Binomial}(100, 1/3)$: mean 33.3, standard deviation 4.71.
The probability of scoring exactly zero is $(2/3)^{100}$, which is
$2.46 \times 10^{-18}$ — about one draw in $4 \times 10^{17}$. The option count per case
is not published, so that 1/3 is an assumption, and it is not load-bearing: at four
options it is $3.2 \times 10^{-13}$, at five $2.0 \times 10^{-10}$. There is no plausible
option count that makes 0/100 a bad day.
So the model is not guessing, and it is not guessing *badly* either. Zero out of a hundred
is what you get from a **deterministic rule that is orthogonal to the answer key** — or,
if the benchmark is built so the relationally-correct option is the one that looks least
convincing on its own, actively anti-correlated with it. Which is exactly how you would
build a test of relational reading: make the locally-plausible option the wrong one. A
scorer that ranks options on local plausibility then lands on the same wrong answer every
single time. Not 33 minus noise. Zero, reliably, by construction.
That makes it the most architecturally informative cell in the whole benchmark, and it
is the reason this piece exists. Every other row measures how good Jev is. This one
measures what Jev *is*.
The task, in the author's description, "uses information in one option to select
another." Here is what that does to a scorer that never lets options see each other:
If each option is encoded and scored on its own — a forward pass per `(question, option)`
pair, a scalar out, softmax afterwards — then an option that refers to another option is
referring to something that is not in its context. Not down-weighted. Absent. The scorer
is being asked to rate a sentence whose subject it cannot see, a hundred times, and it
rates it on whatever is left.
## The mechanism, in three readouts
Three readouts sat in that table, and all three shapes are readable in open code.
[Laya](https://github.com/NandhaKishorM/laya) ships the model that scored its 8.
[openjev](https://github.com/TheoLeeCJ/openjev) is a different project against the same
target, and it implements both of the other two readouts side by side — a per-option
reranker and a single-prompt letter head — which makes it the cleanest place to read the
difference. Chopra's prototype is not openjev; it is a separate 4-bit Qwen3-4B that uses
the letter readout ("a single forward pass scores answer-label probabilities; it does not
generate reasoning text"). Nobody outside TypeSafe can read Jev's own code, which is why
its benchmark row has to do the talking.
Here is the isolation, in source. This is openjev's reranker readout — the one that
follows Qwen3-Reranker's native yes/no contract, and the one whose method note says, in
the author's own words, that *"each candidate answer becomes a separate query/document
relevance proposition."*
```python
# openjev · src/semif_phase1/reranker.py @ ca3ba65
def _encode(tokenizer, row, option, max_tokens):
experiment = row.get("provenance", {}).get("experiment")
instruction = RETRIEVAL_INSTRUCTION if experiment in {"code-rag", "company-brain"} else DECISION_INSTRUCTION
body = (
f": {instruction}\n"
f": Question: {row['question']}\nCandidate answer: {option['description']}\n" # <- one option
f": {row['state']}"
)
text = PREFIX + body + SUFFIX
ids = tokenizer.encode(text, add_special_tokens=False)
if not ids or len(ids) > max_tokens:
raise ValueError(f"Row {row['id']}: {len(ids)} tokens exceed limit {max_tokens}")
return ids, digest(text)
```
`_encode` takes **one** `option`. `row["options"]` is never passed. Whatever the other
options say is not in `text`, is not in `ids`, and is therefore not in any activation the
model computes for this option. That is the line where the relation dies.
The loop above it just repeats that encoding once per option and normalises at the end:
```python
# openjev · src/semif_phase1/reranker.py @ ca3ba65
def score(model, tokenizer, row, metadata, max_tokens=4096):
validate_row(row)
scored, timing = score_pair_batch(
model, tokenizer, [(row, option) for option in row["options"]], max_tokens
) # one independent sequence per option
log_odds = [item["log_odds"] for item in scored]
return {
"option_logits": log_odds,
"probabilities": softmax(log_odds), # the options meet here, and only here
...
}
```
`score_pair_batch` does stack the per-option sequences into one tensor, which looks like
sharing and is not: the batch axis carries no attention. The options meet for the first
time in `softmax(log_odds)`, four scalars long, after every forward pass is finished. By
then there is nothing left of option D except one number.
Laya is the counterexample in the same table, and its code is the reason. Every option
goes into **one** sequence, each preceded by a `[MASK]` marker whose hidden state is what
gets scored:
```python
# laya · laya/common.py @ 6a58191
def build_sequence(tok, state, q, max_len=512, head_max_len=192, option_order=None, truncate_left=False):
"""Format: [CLS] instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]."""
ids = [tok.cls_token_id] + head_ids + [tok.sep_token_id]
markers = []
for o in opt_ids:
markers.append(len(ids)) # where option i's scalar will be read from
ids.extend(o)
ids.append(tok.sep_token_id)
...
return ids[:max_len], [m for m in markers if m < max_len]
# ... and in DecisionModel.forward, a single shared head reads every marker:
idx = marker_pos.clamp(min=0)[:, :, None].expand(-1, -1, h.size(-1))
m = torch.gather(h, 1, idx) # [batch, n_options, d]
logits = self.scorer(m).squeeze(-1).float() # LayerNorm -> Linear(d,d) -> GELU -> Linear(d,1)
```
The encoder is ModernBERT — bidirectional — so `opt1`'s marker attends to `opt3`'s tokens
by construction. Laya *can* represent a relational question. It still scores 8, which on
the same binomial is $2.1 \times 10^{-9}$ and just as far below chance. That is a 421M
model with a 192-token head budget and 48 tokens per option getting the task badly wrong
from inside a window that contains the answer. It is a training and capacity problem.
Jev's zero is not.
### One case, spelled out
Chopra does not publish the suite's items, so this is a case I wrote in openjev's row
schema — the shape `validate_row` accepts — not a benchmark row. The template it is
rendered through is real.
```json
{
"id": "relational-0001",
"state": "Order 88301, customer #4412. Address on file: 9 Kestrel Way, Boise. The customer called this morning and asked for the order to go to their Reno address instead.",
"question": "Which instruction should the fulfilment system follow? Choose the instruction, not the raw address.",
"options": [
{ "id": "A", "description": "Ship to the address on file" },
{ "id": "B", "description": "Ship to the address in option D" },
{ "id": "C", "description": "Hold for pickup at the Boise depot" },
{ "id": "D", "description": "14 Almond Row, Reno" }
]
}
```
B is the answer. You can only know that by reading D, because "the address in option D"
is Reno and Reno is what the customer asked for. Now run it through `_encode` and look at
what each forward pass actually contains:
```text
pass B ────────────────────────────────────────────────────────────────────────
: Given evidence and one possible answer to a question, determine whether the
evidence supports that answer under the question's criterion. Use only the supplied evidence.
: Question: Which instruction should the fulfilment system follow? Choose the
instruction, not the raw address.
Candidate answer: Ship to the address in option D
: Order 88301, customer #4412. Address on file: 9 Kestrel Way, Boise. The
customer called this morning and asked for the order to go to their Reno address instead.
the other three passes are byte-identical except for one line:
pass A Candidate answer: Ship to the address on file
pass C Candidate answer: Hold for pickup at the Boise depot
pass D Candidate answer: 14 Almond Row, Reno
```
The string `14 Almond Row, Reno` appears in exactly one of the four windows, and it is not
B's. Pass B is asked whether the evidence supports "ship to the address in option D" while
holding no option D. The best it can do is notice that the evidence mentions Reno and that
the candidate mentions a pointer, and the pointer resolves to nothing. Pass A, meanwhile,
reads as a clean, well-formed shipping instruction that the evidence partly supports — the
address on file *is* the address on file. A is the locally plausible option. A wins every
time, and A is wrong every time.
That is the whole of 0/100. Not a hundred hard questions. One rule, applied a hundred
times, to a window that never contained the answer.
## This is the claim the last piece could only argue
[The RLCD article](/articles/rlcd-calibrated-decisions) reasoned, from two sentences in
TypeSafe's docs, that Jev must be a per-option scalar scorer rather than a vocabulary
readout. The evidence was verbal: *"Every level is evaluated separately. The model
doesn't see a level's number or its neighbours"*, and the note that high-cardinality
choices run "a 2 stage-system of scoring independently then making an explicit choice."
I labelled it **Reasoned** and attached a falsifier, because that is all it was.
A 0/100 on relational choice is the behavioural signature that reasoning predicts. It is
not proof — a model could fail this way for other reasons — but it is the outcome the
independent-scoring hypothesis requires, and it is a strange outcome under any
architecture where the options share a context.
It also sharpens the parallelism claim. The launch post says questions are processed in
parallel. This result suggests the options within a question are too, which is a stronger
statement about the shape of the thing, and it comes with a real cost: a whole class of
question is not merely answered badly but is unanswerable by construction.
## The size estimate, and how far to trust it
The same benchmark reports Jev at **79.75%** on MMLU-Pro against the 4-bit Qwen3-4B
prototype's **45.00%**, with median latency of **338.6 ms** remote. From that, Chopra
infers Jev is "in the 30Bn range" and concludes it is "mostly a standard modern model
with specific fast-inference related tradeoffs."
The latency half of that is weak evidence: 338.6 ms is a network round trip to someone
else's datacentre, and it bounds nothing about the model. The MMLU-Pro half is the real
argument, and it is a decent one — a 35-point gap over a 4B is a large gap — but
benchmark score maps onto parameter count only loosely and only within a family.
Architecture, training data and post-training move that curve enough that reading a
parameter count off one benchmark is an estimate with a wide interval, not a
measurement. Worth saying because TypeSafe's only published claim about size is
"[Jev is] neither small nor an LLM", and a credible outside estimate of *large* is
genuinely new information, even a loose one.
## Laya: the most complete open System One, and a timeline that does not hold
[Laya](https://laya.convaiinnovations.com) is a 421M ModernBERT-large decision model,
Apache 2.0, with the same three primitives Jev exposes — choice, score and noul — in a
single non-autoregressive forward pass, plus multilingual routing. It is the most
complete open thing in this space, and unlike CUA-S1 or Nimble it publishes the two
numbers everybody else omits: a calibration error, **0.081 after temperature fitting**,
and an option-order stability figure.
It is also being described as having existed months before Jev. That is not what its
artefacts say.
**Receipts.** Laya is widely described as an open System One model that existed months before Jev. Its public artefacts do not support that. Every independent timestamp puts Laya three days AFTER Jev's launch. What does predate Jev is its author's earlier published work on confidence-aware routing — which is real, is genuinely early, and is not the thing the claim says it is.
| artefact | timestamp | source | vs Jev launch |
| :--- | :--- | :--- | :--- |
| arXiv 2503.23303 — SalesRLAgent | 2025-03-30 | arXiv abstract page | ~18 months before |
| arXiv 2510.01237 — Confidence-Aware Routing | 2025-09-23 | arXiv abstract page | ~12 months before |
| TypeSafe Jev launch | 2026-09-15 | launch post | — |
| laya 0.1.0 on PyPI | 2026-09-18 04:38:51 | PyPI upload_time | 3 days after |
| convaiinnovations/laya on HF | 2026-09-18 05:05:55 | HF commits API | 3 days after |
| NandhaKishorM/laya on GitHub | 2026-09-18 | first commit (history rewritten) | 3 days after |
The two arXiv papers are real and predate Jev by a year and by roughly twelve months. The second is described in the author's write-up as 'the exact framework for schema-based decisions guided by reinforcement learning'; its actual title is about pre-generation hallucination mitigation by confidence-aware routing. Adjacent problem, genuinely early, but not a typed-decision model and not RLCD.
> method: PyPI upload times are server-side and not settable by the uploader. Hugging Face commit dates come from the repo's own commit API. The GitHub history was rewritten (one commit message says so), so it is the weakest of the three and is listed last. arXiv submission dates are from the abstract pages.
> captured: 2026-09-19
> data: https://ai.thesatyajit.com/articles/jev-scores-zero/data/timeline.json (6 rows)
PyPI upload times are set server-side. Hugging Face commit dates come from the repo's own
API. Both put Laya's first public release at **2026-09-18**, three days *after* Jev
launched on the 15th. The GitHub history agrees, though it is the weakest of the three
because it has been rewritten — one of its own commit messages says so.
What *is* real is the prior work. The author points to two arXiv papers, and both exist
and both predate Jev:
- [arXiv 2503.23303](https://arxiv.org/abs/2503.23303), *SalesRLAgent*, 30 March 2025 —
RL for real-time sales conversion prediction.
- [arXiv 2510.01237](https://arxiv.org/abs/2510.01237), *Confidence-Aware Routing for
Large Language Model Reliability Enhancement*, 23 September 2025 — assessing model
uncertainty **before** generation and routing on it.
The second is a year ahead of Jev's launch and is genuinely in the neighbourhood:
pre-generation confidence, used to route. That is the idea behind confidence-gated
routing, published early, by someone who then shipped a model. Worth crediting properly.
It is also not what the write-up says it is. It is described as "the exact framework for
schema-based decisions guided by reinforcement learning"; it is a hallucination-mitigation
paper about routing on confidence signals. Adjacent, early, and not a typed-decision
model. The honest version of the claim — *this author was working on confidence-aware
routing a year before Jev launched, and released Laya three days after it* — is a better
story than the overclaim, because it is checkable.
## Credit where the benchmarks are honest
Two things in these releases are worth more than any number in them.
Laya's `BENCHMARKS.md` opens by disarming its own comparison: **"Jev figures are
third-party published, never measured here — no TypeSafe API access — so sample sizes and
prompts differ; treat them as indicative."** The `README` goes further in its own *Honest
limits* section: the base checkpoints score near chance zero-shot — **0.362 and 0.352
against a 0.318 random baseline, and below the 0.461 majority-class baseline** — so the
0.766 headline "comes from the checkpoint fine-tuned on that benchmark's own training
split," and you should "treat Laya as a fast base to specialise, not as a zero-shot
decision engine." That is a project publishing the least flattering framing of its own
model, in its own README, above the fold.
And Chopra published row eight. A benchmark author whose headline is "Jev is a standard
model with tradeoffs" had every incentive to bury a row where Jev scores zero as an
outlier or a bug. Publishing it is what makes the rest of the table worth reading.
## One correction to the earlier piece
The RLCD article said there is no published calibration number for Jev "anywhere", and
that the only number in existence was one outsider's MMLU probe. The first half stands —
TypeSafe still publishes none. The second half is now out of date. Third parties have
measured Jev's ECE at **0.246** and its option-order instability at **0.13**, and there is
now a fifteen-task benchmark. Nobody at TypeSafe published any of it, which was the point,
but "nobody has one" is no longer true and the sentence should not have implied it would
stay that way.
## The pattern, three for three
Every open System One release so far reports beating Jev on its own evaluation:
| release | its benchmark | its score | Jev |
|---|---|---|---|
| cua-s1-forms | 196 real form decisions | 99.7% | 83.6% |
| Bespoke Nimble | 324 contrastive holdout | 90.12% | **93.21%** |
| Laya (fine-tuned) | 2,000 typed decisions | 0.766 | 0.727 |
In every case the open model is measured in-domain — on data from the generator it was
trained against — and Jev is measured cold, having never seen the distribution. That is
the correct way to demonstrate *a specialist beats a generalist on its own turf*, which
is the real thesis of small specialist models and is worth demonstrating. It is not
evidence that Jev is worse at what Jev does.
Nimble is the one that breaks the pattern, and Bespoke published it anyway.
Build the smallest possible probe: a two-option question where option B says "the value in option A" and A carries the value. Run it twenty times with the content swapped between positions. A per-option scorer should be at or near zero regardless of arrangement. If Jev solves it whenever the referent happens to precede the referrer, the cause is ordering or truncation, not isolation, and the architectural reading is wrong.
This is an inference from one benchmark and I would not defend it hard. A 35-point MMLU-Pro gap over a 4-bit 4B is consistent with a much smaller model trained differently, or a mixture where active parameters are far below total. Any disclosure of parameter count, or a careful scaling comparison against models of known size on the same harness, settles it.
PyPI upload times and Hugging Face commit dates are both server-side. If an earlier artefact exists under another name — a package, a model repo, a tagged release before 2026-09-15 — it overturns this immediately, and the claim of priority becomes straightforwardly true rather than a matter of which artefact you count.
Run any of the three on someone else's benchmark. Nimble's contrastive holdout, cua-s1's real-form eval and Laya's typed-decisions set are all published; cross-evaluating them takes an afternoon and would say far more than three separate in-domain wins.