Jev scores zero, and zero is the informative number

Paras Chopra benchmarked Jev against Laya and a 4-bit Qwen3-4B prototype across fifteen tasks — 120 navigation routes and 4,400 decision cases. Jev takes thirteen of the fifteen, often by a lot.

Then there is row eight.

Zero of one hundred. On a suite where the same model scores 100% on rule/evidence judgment, 100% on ordinal scoring, and 98% on ARC-Challenge.

Why zero is worth more than any other number in the table

A model that finds a task hard scores badly, and badly has a shape. On a multiple-choice task it floors out around chance — a third, a fifth, something — and scatters. Getting none right across a hundred attempts is not the bottom of that scatter. It is a different event.

Put the arithmetic on it. If the task is three-way and the model is guessing, the number of correct answers is $\mathrm{Binomial}(100, 1/3)$: mean 33.3, standard deviation 4.71. The probability of scoring exactly zero is $(2/3)^{100}$, which is $2.46 \times 10^{-18}$ — about one draw in $4 \times 10^{17}$. The option count per case is not published, so that 1/3 is an assumption, and it is not load-bearing: at four options it is $3.2 \times 10^{-13}$, at five $2.0 \times 10^{-10}$. There is no plausible option count that makes 0/100 a bad day.

So the model is not guessing, and it is not guessing badly either. Zero out of a hundred is what you get from a deterministic rule that is orthogonal to the answer key — or, if the benchmark is built so the relationally-correct option is the one that looks least convincing on its own, actively anti-correlated with it. Which is exactly how you would build a test of relational reading: make the locally-plausible option the wrong one. A scorer that ranks options on local plausibility then lands on the same wrong answer every single time. Not 33 minus noise. Zero, reliably, by construction.

That makes it the most architecturally informative cell in the whole benchmark, and it is the reason this piece exists. Every other row measures how good Jev is. This one measures what Jev is.

The task, in the author's description, "uses information in one option to select another." Here is what that does to a scorer that never lets options see each other:

If each option is encoded and scored on its own — a forward pass per (question, option) pair, a scalar out, softmax afterwards — then an option that refers to another option is referring to something that is not in its context. Not down-weighted. Absent. The scorer is being asked to rate a sentence whose subject it cannot see, a hundred times, and it rates it on whatever is left.

The mechanism, in three readouts

Three readouts sat in that table, and all three shapes are readable in open code. Laya ships the model that scored its 8. openjev is a different project against the same target, and it implements both of the other two readouts side by side — a per-option reranker and a single-prompt letter head — which makes it the cleanest place to read the difference. Chopra's prototype is not openjev; it is a separate 4-bit Qwen3-4B that uses the letter readout ("a single forward pass scores answer-label probabilities; it does not generate reasoning text"). Nobody outside TypeSafe can read Jev's own code, which is why its benchmark row has to do the talking.

Here is the isolation, in source. This is openjev's reranker readout — the one that follows Qwen3-Reranker's native yes/no contract, and the one whose method note says, in the author's own words, that "each candidate answer becomes a separate query/document relevance proposition."

# openjev · src/semif_phase1/reranker.py @ ca3ba65
def _encode(tokenizer, row, option, max_tokens):
    experiment = row.get("provenance", {}).get("experiment")
    instruction = RETRIEVAL_INSTRUCTION if experiment in {"code-rag", "company-brain"} else DECISION_INSTRUCTION
    body = (
        f"<Instruct>: {instruction}\n"
        f"<Query>: Question: {row['question']}\nCandidate answer: {option['description']}\n"   # <- one option
        f"<Document>: {row['state']}"
    )
    text = PREFIX + body + SUFFIX
    ids = tokenizer.encode(text, add_special_tokens=False)
    if not ids or len(ids) > max_tokens:
        raise ValueError(f"Row {row['id']}: {len(ids)} tokens exceed limit {max_tokens}")
    return ids, digest(text)

_encode takes one option. row["options"] is never passed. Whatever the other options say is not in text, is not in ids, and is therefore not in any activation the model computes for this option. That is the line where the relation dies.

The loop above it just repeats that encoding once per option and normalises at the end:

# openjev · src/semif_phase1/reranker.py @ ca3ba65
def score(model, tokenizer, row, metadata, max_tokens=4096):
    validate_row(row)
    scored, timing = score_pair_batch(
        model, tokenizer, [(row, option) for option in row["options"]], max_tokens
    )                                # one independent sequence per option
    log_odds = [item["log_odds"] for item in scored]
    return {
        "option_logits": log_odds,
        "probabilities": softmax(log_odds),   # the options meet here, and only here
        ...
    }

score_pair_batch does stack the per-option sequences into one tensor, which looks like sharing and is not: the batch axis carries no attention. The options meet for the first time in softmax(log_odds), four scalars long, after every forward pass is finished. By then there is nothing left of option D except one number.

Laya is the counterexample in the same table, and its code is the reason. Every option goes into one sequence, each preceded by a [MASK] marker whose hidden state is what gets scored:

# laya · laya/common.py @ 6a58191
def build_sequence(tok, state, q, max_len=512, head_max_len=192, option_order=None, truncate_left=False):
    """Format: [CLS] <type> instructions [SEP] [MASK] opt0 [MASK] opt1 ... [SEP] state [SEP]."""
    ids = [tok.cls_token_id] + head_ids + [tok.sep_token_id]
    markers = []
    for o in opt_ids:
        markers.append(len(ids))     # where option i's scalar will be read from
        ids.extend(o)
    ids.append(tok.sep_token_id)
    ...
    return ids[:max_len], [m for m in markers if m < max_len]
 
# ... and in DecisionModel.forward, a single shared head reads every marker:
idx = marker_pos.clamp(min=0)[:, :, None].expand(-1, -1, h.size(-1))
m = torch.gather(h, 1, idx)                       # [batch, n_options, d]
logits = self.scorer(m).squeeze(-1).float()       # LayerNorm -> Linear(d,d) -> GELU -> Linear(d,1)

The encoder is ModernBERT — bidirectional — so opt1's marker attends to opt3's tokens by construction. Laya can represent a relational question. It still scores 8, which on the same binomial is $2.1 \times 10^{-9}$ and just as far below chance. That is a 421M model with a 192-token head budget and 48 tokens per option getting the task badly wrong from inside a window that contains the answer. It is a training and capacity problem. Jev's zero is not.

One case, spelled out

Chopra does not publish the suite's items, so this is a case I wrote in openjev's row schema — the shape validate_row accepts — not a benchmark row. The template it is rendered through is real.

{
  "id": "relational-0001",
  "state": "Order 88301, customer #4412. Address on file: 9 Kestrel Way, Boise. The customer called this morning and asked for the order to go to their Reno address instead.",
  "question": "Which instruction should the fulfilment system follow? Choose the instruction, not the raw address.",
  "options": [
    { "id": "A", "description": "Ship to the address on file" },
    { "id": "B", "description": "Ship to the address in option D" },
    { "id": "C", "description": "Hold for pickup at the Boise depot" },
    { "id": "D", "description": "14 Almond Row, Reno" }
  ]
}

B is the answer. You can only know that by reading D, because "the address in option D" is Reno and Reno is what the customer asked for. Now run it through _encode and look at what each forward pass actually contains:

pass B  ────────────────────────────────────────────────────────────────────────
<Instruct>: Given evidence and one possible answer to a question, determine whether the
evidence supports that answer under the question's criterion. Use only the supplied evidence.
<Query>: Question: Which instruction should the fulfilment system follow? Choose the
instruction, not the raw address.
Candidate answer: Ship to the address in option D
<Document>: Order 88301, customer #4412. Address on file: 9 Kestrel Way, Boise. The
customer called this morning and asked for the order to go to their Reno address instead.
 
the other three passes are byte-identical except for one line:
  pass A    Candidate answer: Ship to the address on file
  pass C    Candidate answer: Hold for pickup at the Boise depot
  pass D    Candidate answer: 14 Almond Row, Reno

The string 14 Almond Row, Reno appears in exactly one of the four windows, and it is not B's. Pass B is asked whether the evidence supports "ship to the address in option D" while holding no option D. The best it can do is notice that the evidence mentions Reno and that the candidate mentions a pointer, and the pointer resolves to nothing. Pass A, meanwhile, reads as a clean, well-formed shipping instruction that the evidence partly supports — the address on file is the address on file. A is the locally plausible option. A wins every time, and A is wrong every time.

That is the whole of 0/100. Not a hundred hard questions. One rule, applied a hundred times, to a window that never contained the answer.

This is the claim the last piece could only argue

The RLCD article reasoned, from two sentences in TypeSafe's docs, that Jev must be a per-option scalar scorer rather than a vocabulary readout. The evidence was verbal: "Every level is evaluated separately. The model doesn't see a level's number or its neighbours", and the note that high-cardinality choices run "a 2 stage-system of scoring independently then making an explicit choice." I labelled it Reasoned and attached a falsifier, because that is all it was.

A 0/100 on relational choice is the behavioural signature that reasoning predicts. It is not proof — a model could fail this way for other reasons — but it is the outcome the independent-scoring hypothesis requires, and it is a strange outcome under any architecture where the options share a context.

It also sharpens the parallelism claim. The launch post says questions are processed in parallel. This result suggests the options within a question are too, which is a stronger statement about the shape of the thing, and it comes with a real cost: a whole class of question is not merely answered badly but is unanswerable by construction.

The size estimate, and how far to trust it

The same benchmark reports Jev at 79.75% on MMLU-Pro against the 4-bit Qwen3-4B prototype's 45.00%, with median latency of 338.6 ms remote. From that, Chopra infers Jev is "in the 30Bn range" and concludes it is "mostly a standard modern model with specific fast-inference related tradeoffs."

The latency half of that is weak evidence: 338.6 ms is a network round trip to someone else's datacentre, and it bounds nothing about the model. The MMLU-Pro half is the real argument, and it is a decent one — a 35-point gap over a 4B is a large gap — but benchmark score maps onto parameter count only loosely and only within a family. Architecture, training data and post-training move that curve enough that reading a parameter count off one benchmark is an estimate with a wide interval, not a measurement. Worth saying because TypeSafe's only published claim about size is "[Jev is] neither small nor an LLM", and a credible outside estimate of large is genuinely new information, even a loose one.

Laya: the most complete open System One, and a timeline that does not hold

Laya is a 421M ModernBERT-large decision model, Apache 2.0, with the same three primitives Jev exposes — choice, score and noul — in a single non-autoregressive forward pass, plus multilingual routing. It is the most complete open thing in this space, and unlike CUA-S1 or Nimble it publishes the two numbers everybody else omits: a calibration error, 0.081 after temperature fitting, and an option-order stability figure.

It is also being described as having existed months before Jev. That is not what its artefacts say.

PyPI upload times are set server-side. Hugging Face commit dates come from the repo's own API. Both put Laya's first public release at 2026-09-18, three days after Jev launched on the 15th. The GitHub history agrees, though it is the weakest of the three because it has been rewritten — one of its own commit messages says so.

What is real is the prior work. The author points to two arXiv papers, and both exist and both predate Jev:

• arXiv 2503.23303, SalesRLAgent, 30 March 2025 — RL for real-time sales conversion prediction.
• arXiv 2510.01237, Confidence-Aware Routing for Large Language Model Reliability Enhancement, 23 September 2025 — assessing model uncertainty before generation and routing on it.

The second is a year ahead of Jev's launch and is genuinely in the neighbourhood: pre-generation confidence, used to route. That is the idea behind confidence-gated routing, published early, by someone who then shipped a model. Worth crediting properly.

It is also not what the write-up says it is. It is described as "the exact framework for schema-based decisions guided by reinforcement learning"; it is a hallucination-mitigation paper about routing on confidence signals. Adjacent, early, and not a typed-decision model. The honest version of the claim — this author was working on confidence-aware routing a year before Jev launched, and released Laya three days after it — is a better story than the overclaim, because it is checkable.

Credit where the benchmarks are honest

Two things in these releases are worth more than any number in them.

Laya's BENCHMARKS.md opens by disarming its own comparison: "Jev figures are third-party published, never measured here — no TypeSafe API access — so sample sizes and prompts differ; treat them as indicative." The README goes further in its own Honest limits section: the base checkpoints score near chance zero-shot — 0.362 and 0.352 against a 0.318 random baseline, and below the 0.461 majority-class baseline — so the 0.766 headline "comes from the checkpoint fine-tuned on that benchmark's own training split," and you should "treat Laya as a fast base to specialise, not as a zero-shot decision engine." That is a project publishing the least flattering framing of its own model, in its own README, above the fold.

expand

And Chopra published row eight. A benchmark author whose headline is "Jev is a standard model with tradeoffs" had every incentive to bury a row where Jev scores zero as an outlier or a bug. Publishing it is what makes the rest of the table worth reading.

One correction to the earlier piece

The RLCD article said there is no published calibration number for Jev "anywhere", and that the only number in existence was one outsider's MMLU probe. The first half stands — TypeSafe still publishes none. The second half is now out of date. Third parties have measured Jev's ECE at 0.246 and its option-order instability at 0.13, and there is now a fifteen-task benchmark. Nobody at TypeSafe published any of it, which was the point, but "nobody has one" is no longer true and the sentence should not have implied it would stay that way.

The pattern, three for three

Every open System One release so far reports beating Jev on its own evaluation:

In every case the open model is measured in-domain — on data from the generator it was trained against — and Jev is measured cold, having never seen the distribution. That is the correct way to demonstrate a specialist beats a generalist on its own turf, which is the real thesis of small specialist models and is worth demonstrating. It is not evidence that Jev is worse at what Jev does.

Nimble is the one that breaks the pattern, and Bespoke published it anyway.