AgentJev gets both halves

malevrigns/agent-jev is a 0.6B open-weights decision model with a claim list I have seen most of before. Beats Laya on Typed Decisions, 79.25% against 77.00%. Sixty-four candidates in one forward pass with KV prefix reuse, 92.4% of the computation removed, about 2x faster. Sixty-eight milliseconds per decision, zero decoded tokens, no JSON to parse. Twice Laya's context.

One item on that list is not like the others, and it is the reason this article exists:

Every other claim needs a benchmark, a denominator and an argument about whose hardware it ran on. That one is a claim about a type signature, and this site has spent four articles building the experiment that tests it. Reverse the option list. A per-option scalar scorer must flip exactly none of them by construction; a vocabulary readout cannot. openjev's letter readout flips 10 of 36 reversed questions and carries a +1.71 logit prior for slot A. The hosted model was measured this week flipping 12 of 100 under permutation.

The weights are Apache-2.0 and 1.2 GB, which is small enough to run on the four CPU cores I have. So I ran it.

The answer changed on 0 of 50 reversed questions, and on 0 of 2,520 permutations of those same option sets. The largest movement of any single logit anywhere is 9.5e-07 — a few units in the last place of a float32. Every candidate's backbone vector comes back bit-identical between the two orders. This is not a training result and it is not a number that got small: nothing in this model has anywhere to put a position prior, so the only thing left to measure is the width of the float.

That is a real result. Other models in this category have been measured at zero flips on a small set; this is the first one where the invariance has been measured at the level of the arithmetic rather than at the level of the answer. What makes it worth an article rather than a footnote is how it is arranged, because it is not the arrangement that was supposed to buy this — and because the same model's score for an option does depend on what the other options say: a median of 0.088 logits, where a per-option scorer moves by exactly zero.

The third arrangement

The fork this site has been describing since the cua-s1 piece is a genuine one, and I have stated it as a law more than once: the shape does not say how options reach the scorer, and the two answers fail in opposite directions. Isolate the options and order becomes inexpressible, but a question spanning two options has no answer to find — Jev scores 0 of 100 on exactly that. Share the context and relational questions become answerable, but position acquires a prior. Nobody gets both halves.

(The hosted model's 12 of 100 does not contradict that. Order-invariance is a property of the shape; what a hosted API does to it on the way through a batched fleet is a different question, and the determinism piece takes that apart. What follows is a model you can run yourself, where there is no fleet in the middle.)

AgentJev is arranged a third way, and it gets both halves. Here is the whole of it.

Read the serving file first, because its docstring is the clearest statement of the design anywhere in the repository:

# agent-jev · jev_service/prefix.py @ b66a8ce — the module docstring, in full
"""Inference-only shared-prefix KV branches, preserving independent causal paths.
 
Each branch receives a private cache. Sibling candidates never attend one
another. The trained candidate-set head runs after all branch vectors exist.
"""

Four sentences, and between them the whole design. The prefix is encoded once and cached — it is whatever the candidate paths agree on, which is the state, the question, the [CANDIDATE] marker and any leading words the candidates happen to share. Each candidate is then a private branch off that cache, so no candidate is in any other candidate's context. And every branch resumes at the same position index, because they all continue the same prefix, so candidate 7 and candidate 40 are encoded at identical rotary positions. There is no slot for a prior to attach to. That last property holds in the fallback route too, where each candidate gets its own full sequence: the candidate text still begins at token len(prefix) in every row.

# agent-jev · jev_service/prefix.py @ b66a8ce — the shared branch, reflowed from its one-line-per-statement original
prefix = torch.tensor([entries[0][0][:prefix_length]], device=device)
encoded = backbone(input_ids=prefix, attention_mask=torch.ones_like(prefix), use_cache=True)
root_cache = encoded.past_key_values
for start in range(0, len(entries), engine.path_batch):
    chunk = entries[start:start + engine.path_batch]
    suffixes = [seq[prefix_length:] for seq, _ in chunk]     # candidate text only
    cache = copy.deepcopy(root_cache)
    cache.batch_repeat_interleave(len(chunk))                # one private copy per candidate
    hidden = backbone(input_ids=ids, attention_mask=attention,
                      past_key_values=cache, use_cache=True).last_hidden_state

batch_repeat_interleave is the whole trick: the cache is copied across the batch axis, and the batch axis carries no attention. This is the same observation the cua-s1 piece made about score_pair_batch — stacking sequences into one tensor looks like sharing and is not — used deliberately rather than incidentally.

It is worth being precise about what the prefix sharing does and does not buy, because the obvious reading is wrong. Sharing the prefix is an efficiency device. It changes no property of the answer: the fallback route, which gives every candidate its own full sequence, computes the same thing more slowly, and the repository measured the two against each other and got a maximum probability difference of 0.000508. Order-invariance does not come from the sharing.

So far this is an efficient per-option scalar scorer. The options would meet at the softmax, four scalars long, and a relational question would be unanswerable. The departure is one module further on:

# agent-jev · agentjev/model.py @ b66a8ce
class CandidateSetEncoder(nn.Module):
    """Transformer over the candidate set of one question.
 
    No positional embeddings, so the mapping is permutation-equivariant:
    permuting the candidates permutes the outputs identically.
    """

The candidates do meet, in a two-layer transformer over the set of them, and they meet before the scalar is read rather than after. A transformer with no positional embeddings is a set function: permuting its inputs permutes its outputs and changes nothing else. That is the property the whole article turns on, and it is bought in the only place where cross-candidate information does not cost you a position.

The whole scoring path is seven lines:

# agent-jev · agentjev/model.py @ b66a8ce — AgentJevModel._score, annotations mine
def _score(self, cand_vecs: torch.Tensor, cand_mask: torch.Tensor) -> torch.Tensor:
    x = self.proj_in(cand_vecs)                  # [Bq, C, 1024] -> [Bq, C, 256]
    x = x * cand_mask.unsqueeze(-1).to(x.dtype)
    enc = self.set_encoder(x, cand_mask)         # candidates see each other here, and only here
    rep = cand_vecs + self.proj_out(enc)         # residual back onto the isolated vector
    logits = self.scorer(rep)                    # [Bq, C] — one scalar each
    return logits.masked_fill(~cand_mask, -1e4)

Note what the residual does. cand_vecs is the isolated per-candidate reading; proj_out(enc) is the correction that knowing the rest of the set buys you. Zero the set encoder's output and what is left is the shape of open-jev-deberta — a learned scalar head over an isolated encoding, which is the per-option family exactly. The whole departure from it is one additive term.

The channel is real, and I can put a number on it

A residual term is easy to write and easy to have contribute nothing. So the second half of the experiment below asks the complementary question: hold one candidate's text fixed, change which other candidates are on the list, and see whether that candidate's raw logit moves. Not its probability — that moves for free through the softmax when the set changes size — its logit, the number the softmax is taken over.

A per-option scalar scorer must give exactly the same logit. Its scalars exist before anything knows the set. AgentJev's move by a median of 0.088 logits and by as much as 0.71.

More than five decades separate the two perturbations — 5.3 comparing medians, 5.9 comparing worst cases. That is the article in one picture: the score depends on what else is on the list and not at all on where on the list it is.

One control, because dropping an option changes the size of the set as well as its contents, and a length effect would look the same. Take a candidate and two other candidates, and compare the two subsets that drop one of those others. Both subsets have the same cardinality and both still contain the candidate you are watching, so the only difference is which one of its neighbours is present. That is a pure content perturbation with the size held fixed: median 0.081 logits, worst 0.83, the same order of magnitude as before. It is the contents, not the count.

And the proportionate qualifier, because a tenth of a logit is not a lot: on this sample, removing a losing option changed the winner in zero of 183 drops. The channel is measurable and it is not, on these questions, decisive. That is roughly what you would expect from a 256-dimensional summary competing with a 1024-dimensional reading of the candidate itself — and it is why the open question is whether the channel can carry a reference, not whether it exists.

What the third arrangement costs

Two things, and neither is order.

The cross-candidate channel is narrow. A candidate arrives at the set encoder as one 256-dimensional vector, projected down from the 1024-dimensional hidden state at its last token. Option B does not get to read option D's text; it gets to read a 256-number summary of it. So the channel exists — the bar above is not zero — but whether it is wide enough to resolve "ship to the address in option D", the case spelled out in the jev-scores-zero piece, is untested by them and by me. It is the obvious next experiment and there is a falsifier for it at the bottom.

Equivariance is exact in the maths and approximate in the arithmetic. A permuted set changes the summation order inside the attention, and floating-point addition is not associative. The training code knows this: there is a permutation_invariance_kl term that re-runs the head on a shuffled order and penalises the divergence, with a comment saying it exists to "guard the property under bf16 numerics". Its weight is zero in the shipped protocol, which I think is the right call — this does not need training, it needs a wide enough float. What I measured on CPU is float32, and 9.5e-07 is what that costs. The repository's own check ran on GPU under bf16 autocast and reported exactly 0.0 — but that is a probability, on one three-option question, read off a head running in bf16, whose significand is eight bits against float32's twenty-four. A difference of the size I measured could not have shown up there. It is a smaller number on a coarser instrument.

The measurement, and how I got the numbers

Their check is jev_service/verify_http.py. It is one question, three options, reversed once:

# agent-jev · jev_service/verify_http.py @ b66a8ce — the entire permutation check
permuted['questions'][0]['options'] = dict(reversed(list(choice['questions'][0]['options'].items())))
shuffled = evaluate(permuted)
perm_error = max(abs(a[k] - b[k]) for k in a); assert perm_error < .02, perm_error

permutation_max_absolute_error: 0.0, on n = 1. True, and thin. So I ran it at scale, on the benchmark's own data, through the shipped code, on four CPU cores.

There are two runs, and the first one licenses the second.

Run one is strict and end-to-end. Take a question from the Typed Decisions test split, build the caller's payload twice — once with the option list as published, once reversed — and push each independently through jev_service.contract.prepare, encode_paths, the Qwen3 backbone, AgentJevModel._score, the per-primitive temperature and the softmax. Nothing is reused between the two: they are re-tokenised, re-batched and re-encoded, so the rows of the tensor that reaches the backbone are in a different order in the two runs. Realign the reversed answer and compare. Boolean questions are excluded, because contract.prepare hardwires their candidate order and leaves the caller nothing to reverse:

# agent-jev · jev_service/contract.py @ b66a8ce — a boolean has no order for the caller to reverse
keys = ['true', 'false']
candidates = [semantic(criteria[k], f'{k} criterion') if k in criteria else k.upper() for k in keys]

Two things came out, and the second is the one I did not expect to be able to state so bluntly.

The answer never changed. 0 of 50, Choice and Score questions with four or five candidates. On the same test the openjev readout changes its mind on 27.8% and the hosted model on 12%.

The backbone vectors are bit-identical. max|Δv| = 0.0 on every question, on every candidate, to the last bit. Reversing the list reverses the order of the rows in the batch that goes into the backbone and changes not one number that comes out. This is the load-bearing fact, and it is what makes the second run exact rather than an approximation: the vector for candidate j is a function of candidate j and the shared prefix, and of nothing else in the request.

Run two is the stronger perturbation, for the price of the first. Because the vectors do not depend on the arrangement, scoring a permuted set of the same vectors through _score is precisely what the service computes for a permuted request. So: one backbone pass per question, then every permutation of the candidate set — all 24 for a four-option question, all 120 for five — scored at the head and realigned. 57 questions, 2,520 permutations, zero answer changes, and the largest realigned movement of any single logit anywhere in the sweep is 9.5e-07.

Two to the minus twenty-three is 1.19e-07, so that worst case is a handful of units in the last place of a float32 mantissa, arriving in the set encoder's attention sum and nowhere else. To put it on the scale this corpus has been building: kev reports probabilities "within 4e-6" when the same questions are asked together or separately in fp32, and calls that four parts in a million.

The whole test is short enough to paste. This is the core of it, against the repository's own modules:

# reverse one question and compare, using the shipped serving code
from jev_service.contract import prepare, encode_paths
from agentjev.model import AgentJevModel
 
def logits(state, question):                      # question is the caller's dict
    paths, _, _ = encode_paths(prepare({"state": state, "questions": [question]}), tok, 2048)
    batch = pad_into_a_batch(paths)               # ids, attention_mask, cand_end_pos, …
    vecs = model.encode_candidates(batch)         # [1, C, 1024], one per candidate
    return vecs[0], model._score(vecs, batch["cand_mask"])[0].tolist()
 
forward = {**q, "options": dict(pairs)}
reversed_ = {**q, "options": dict(reversed(pairs))}
vf, zf = logits(state, forward)
vr, zr = logits(state, reversed_)
 
assert (vf - vr.flip(0)).abs().max() == 0          # the backbone never saw the order
print(max(abs(a - b) for a, b in zip(zf, reversed(zr))))

The assert is the line worth running on any model in this category. It passes here.

598,418,689 parameters, and 2.4 million of them are the model

"0.6B" is a claim, so I pulled the safetensors header for aimeigaoshou/agent-jev with range requests and summed the shapes. 343 tensors, all BF16, 598,418,689 parameters. The README's own latency table says 598M, so the claim and the artifact agree. That is the same check DiffusionGemma's 25.2B needed and it costs the same five minutes; it is worth doing every time, and it is worth saying when a project passes it.

The split is the interesting part:

Everything that makes this a decision model rather than a language model is 2,368,769 parameters — 0.4% of the checkpoint. The cross-candidate channel accounts for 1,579,520 of them, and the order-invariance accounts for none: it is a positional embedding table that was never created. For scale, the whole of cua-s1-forms is 706,048 parameters, so this head is about three and a third cua-s1s bolted to a Qwen3.

One detail the README's "language-model head removed" glosses. Qwen3-0.6B sets tie_word_embeddings: true, so the LM head is the embedding matrix transposed. Removing it frees exactly zero parameters. The saving is that no logit vector 151,936 wide is ever materialised, which is real and is a saving of compute and bandwidth, not of weights.

The option ceiling that is not there

This corpus has now found an option count welded into an export twice: logits[batch_size, 25] in the only open ONNX export, and K = 32 frozen into a Core ML bundle along with batch and sequence length. A third would make it a pattern, so I went looking.

It is not there — and part of the reason is that there is nothing here to freeze it into. Both earlier ceilings were properties of an export: a graph compiled to a fixed-shape target. AgentJev ships a PyTorch state dict and a Python server. No ONNX, no Core ML, no traced graph, so no shape got baked.

The scorer ends in scorer.fc2.weight [1, 256] — output dimension one, evaluated once per candidate, the same "N is a loop bound, not a weight" property the cua-s1 piece drew. And the set encoder, which is the one module that does see the whole set at once, carries no positional embedding table at all; that is the same absence that buys the order-invariance. A transformer with no positions has no maximum length.

So every limit AgentJev appears to have is a raise ValueError in jev_service/contract.py, plus one argparse default. The 64 candidates in the headline is a workload, not a ceiling; the contract takes 255, which is the number TypeSafe's own Choice takes. Even the 2,048-token context is --max-tokens, against a published config.json that declares max_position_embeddings: 32768. Doubling Laya's 1,024 is a real operational difference and it is a launch flag, not an architecture.

The one bound that is not in that file is the training distribution. The typed-decisions fine-tune saw questions with 2, 4 and 5 candidates and nothing wider. 255 is what the contract accepts, not what the checkpoint has been shown.

92.4% of the tokens, 51% of the clock

The efficiency claim is the one with the most committed evidence behind it and the most room to be misread, so I rebuilt it.

expand

jev_service/verify_prefix.py builds one fixed request — 64 Choice options and one Boolean over a repeated state — and times it under all three encoder modes, median of three after warmup. I re-ran its tokenisation and got the repository's numbers exactly: 66 candidate paths, 33,547 path tokens, a 492-token shared prefix on the Choice question and 493 on the Boolean. Sixty-four candidates paying 492 tokens each for the same state is the whole of the redundancy, and removing it is the whole of the 92.4%.

So: "92.4% computation reduced" is measured in backbone input tokens, and it is exact. What it buys on the clock is 2.04x. The gap between those two ratios is the interesting engineering, and the middle row of that table is the proof that it is real rather than a rounding: forcing every question to share its prefix feeds the backbone 19% fewer tokens than the automatic mode and takes 17% longer.

The reason is that the branched pass stops being compute-bound. auto sends 2,551 tokens through a 0.6B model — about 2.2 TFLOP, which is nothing — but it does so as four sequential microbatches of sixteen rows and roughly sixteen tokens each, after copying a 492-token KV cache sixteen times per microbatch. Twenty-eight layers of kernel launches over a batch that is 256 tokens wide does not fill a GPU. The auto heuristic is one line and it is the right line:

# agent-jev · jev_service/prefix.py @ b66a8ce — share the prefix only where it pays
use_shared = mode == 'shared' or (mode == 'auto' and len(entries) >= 8 and prefix_length >= 128)

Fewer than eight candidates, or a prefix under 128 tokens, and it falls back to independent paths.

While I am in that loop: "64 candidates in one forward pass" is the one phrase in the claim list that is not literally true. path_batch is 16, so a 64-candidate question is one prefill of the prefix plus four batched branch passes — five backbone invocations, not one. What happens once is the encoding of the state, which is the part that matters and the part the 92.4% measures. It is a small thing, and it is also exactly the four sequential microbatches that keep the speedup at 2.04x instead of something larger.

That eight-candidate threshold is why the repo's own 255-candidate smoke test reports shared_prefix_questions: 0: its state was short, so nothing was cached, and 10,090 tokens went through the backbone at 684 ms. The figure's "up to 255 candidates without linear slowdown" is the one line on it that no committed measurement supports.

Worth saying what the sharing does not cost: the maximum probability difference against the independent-path route is 0.000508 and the chosen option does not change. That is a checked claim in a category where people usually assert parity.

The 68 ms is in a GIF

The per-decision latency figure is the one number in the claim list with no file behind it. It comes from assets/agentjev_reflex_demo.gif.

expand

The README's own latency table is more careful than its GIF, and it is worth noticing which way the rounding runs:

Laya's figures are measured to two decimals. AgentJev's are a tilde and a range, and they are the slower of the two. That is a project rounding against itself, which is the opposite of the usual direction and part of why I trust the rest of the table. Note also the unit: 60–70 ms is per case, five questions over one state, so the per-decision figure is nearer 13 ms — while the GIF's 68 ms is for a single four-option Choice. Same magnitude, different workloads, and neither of them is in a committed artifact the way every other number in this repository is.

The benchmark, and the line its own card draws

Typed Decisions is LocalLLaMA/typed-decisions: Apache-2.0, public, 400 test cases and 2,000 questions across four workflows, five typed questions over one shared state. It is independent — its card says so outright, "not affiliated with TypeSafe and it does not reproduce their Jev model" — and it is the best-documented benchmark in this category, because it publishes its own ceilings.

Who made it, I cannot tell you beyond the handle. The dataset is owned by a Hugging Face account called LocalLLaMA, uploaded on 2026-09-16, with no paper, no author list and no repository behind it; it has 2,599 downloads and eleven likes. That is worth stating rather than glossing, because two open releases are now being ranked against each other on it. What it does have is a build procedure written down in full, four reference rows, two ceilings, and a Jev measurement with a date, a request count and a bill attached — which is more provenance than most benchmarks with an author list.

Two rows of AgentJev's table were measured in AgentJev's repository. The Laya row is a real re-measurement — convaiinnovations/laya-typed-decisions at a pinned revision, all 2,000 questions in 19.2 seconds — not a quoted figure, which is better practice than the three-for-three pattern the jev-scores-zero piece catalogued. The Jev row is quoted, but from a card that ran the model itself: all 400 cases through the TypeSafe API on 2026-09-18, p50 710 ms per case, $0.016 total. So the 72.7% is a measurement by a third party with no stake in either specialist.

Now the part the README leaves out.

The card publishes a saturation point as well as a floor. A fresh teacher sample, scored against gold built from the other samples, agrees 73.5% of the time. A model that recovers the latent factors each case was generated from exactly scores 70.4%. And the card states the reading in its own words:

AgentJev scores 79.25%. Laya scores 77.00%. Both are above the noise floor of the labelling process, and the gap between them — 2.25 points, bootstrap interval [+0.65, +3.90] — sits entirely in the region the benchmark's author says is teacher idiosyncrasy rather than task. Neither project mentions this. AgentJev's README is careful in every other respect about what its number is — "agreement with the public teacher argmax… not a measured coding-agent success rate", "beating a row here does not mean a pull request merged" — and the one sentence it does not carry is the one the card wrote for it.

The README's own by-workflow table sharpens it further, and I do not think anyone has done this arithmetic. The four workflows are 500 questions each, so the overall delta is their mean:

The whole margin is two workflows. On the other two, the 421M encoder wins. A result that is +5 here and −1 there is not a model that is 2.25 points better; it is two models that are better at different things, averaged.

This is not a reason to disbelieve the result. A specialist fitted on 960 cases from four workflows, evaluated on 400 cases from the same four workflows, should learn the teacher. The hygiene on both sides is good — better than the usual. The benchmark generates its two splits in separate runs at different seeds and refuses to package if a case id or a state hash appears in both. AgentJev splits by case id, reserves 120 development and 120 calibration cases out of the 1,200, picks the checkpoint on development soft cross-entropy before the test split is opened, and pins the seed and the dataset revision. It is a reason to read +2.25 points over Laya as what it is, which is two specialists agreeing with the same teacher to within its own sampling noise.

The training run, incidentally, is 600 steps and 841 seconds. Fourteen minutes on one GPU takes the phase-4 checkpoint from 38.70% to 79.25%, which says rather more about how much of this task is in the four workflows than about the architecture.

Calibration: published, unusual, and not what the number looks like

The cua-s1 article had to be corrected for saying nobody published a calibration number when two projects had, so I went looking carefully here before saying anything.

It publishes plenty. Expected calibration error, sum-of-candidates Brier, soft cross-entropy and score MAE, per primitive and per workflow, before and after temperature scaling, for its own checkpoint and for the Laya checkpoint it re-measured, with all 2,000 per-question probability vectors committed. The temperature is one positive scalar per primitive, fitted on 180 boolean, 180 choice and 240 score questions from cases reserved for exactly that and used for nothing else. Only Bespoke's thirteen-dataset public suite goes further, and nobody in this category has better calibration hygiene than a split that reserves 120 cases for temperature fitting before the test set is opened. That is the protocol the RLCD piece asked for, done without being asked.

The part nobody else in this category has

Before the numbers, the thing I did not expect to find in the training code. agentjev/synth.py is a generator of decision questions whose gold distribution is exactly computable: biased coins with a Beta prior, loaded dice, urn draws, card draws, small Markov chains, Bayesian updates. Not a teacher's opinion of the probability — the probability.

# agent-jev · agentjev/synth.py @ b66a8ce — gen_coin, with the state string and four
# of its six questions elided
a0, b0 = rng.randint(1, 4), rng.randint(1, 4)
p = rng.betavariate(a0, b0)          # the world's actual bias, drawn from the prior
n = rng.randint(0, 20)
k = sum(1 for _ in range(n) if rng.random() < p)   # heads observed
K = rng.randint(1, k + 6)            # how many heads the goal needs in total
R = rng.randint(0, 12)               # flips left
post_a, post_b = a0 + k, b0 + (n - k)
if k >= K:
    p_goal = 1.0
else:
    # future flips share one unknown p, so this is the Beta-binomial posterior predictive
    p_goal = beta_binom_tail_at_least(K - k, R, post_a, post_b)
 
qs.append(_q("stop", "boolean",
             f"If we keep flipping for the remaining {R} flip(s), will the goal be reached?",
             BOOL, {"distribution": _round_dist([p_goal, 1 - p_goal])},
             "known_distribution", 1.0))

And agentjev/losses.py gives that label a supervision type, with a weight attached to how much the label is worth believing: known_distribution and deterministic at 1.0, binomial_counts and multiclass_counts scored as a count NLL with the weight scaling in the number of trials, and empirical/heuristic/teacher at 0.1–0.3. A teacher's distribution is explicitly the cheapest kind of supervision in this loss.

That is the most serious answer to this category's standing complaint that I have read. Every project here returns a probability; the criticism, restated at least four times on this site and once by kev in its own model card, is that the number is a statistic about a distribution rather than a verified frequency. Training on a closed-form posterior is the obvious fix, and this is the first repository I have seen do it.

Two honest limits. The phase checkpoints and the training configs are gitignored, so I can read the generator and the loss but I cannot verify what mixture was actually used or at what weight. And the generator's world is arithmetic — coins, urns, Markov chains — so whether calibration learned on "will this reach five heads in three flips" survives the trip to "should this invoice be paid" is precisely the open question. The answer visible in this repository is: partly. The phase-4 checkpoint that came out of that pipeline is the one over-confident row in the whole comparison, and it is 38.70% accurate.

Two things about the numbers themselves.

The Brier column is comparable and the ECE column is not. The README says plainly that the Jev, ModernBERT and MiniLM rows are copied from the dataset card and "use the card's definitions". So I checked whether the definitions agree, by running AgentJev's own metrics() over a uniform predictor — 1/C on every option, nothing fitted, nothing to disagree about — on the same 2,000 questions. Brier comes out at 0.2382 against the card's 0.238: the same statistic, so AgentJev's 0.0448 really does sit on the same scale as Jev's 0.148. ECE comes out at 0.046 against the card's 0.169. No bin count between 10 and 20, and no hard/soft accuracy variant I tried, gets within a factor of three. The 0.1687 next to Jev's 0.144 is two different statistics in one column.

And the model is under-confident, not over-confident. Both releases committed their per-question probabilities and neither drew the curve, so here it is.

Every populated bin of both models sits above the diagonal. AgentJev's mean stated confidence is 0.624 and it is right 0.793 of the time; Laya's is 0.553 against 0.770. Neither is over-confident anywhere. That inverts the warning this category usually needs — kev's model card putting "a confidence: 0.92 from this model is a statistic about its own distribution" on the wall — because a threshold at 0.5 here escalates decisions the model gets right 78% of the time.

There is a clean reason, and it is in the benchmark rather than the model. The training target is a teacher distribution, the mean of three samples at temperature 0.7, and the mean top mass of that gold across the test split is 0.659. A model trained on soft cross-entropy against those targets reports 0.624. It is reproducing the teacher's spread almost exactly. The 0.169 is measuring the gap between the teacher's spread and the teacher's argmax, which is a property of the labels, not a miscalibration of the model.

Two consequences follow, and both are visible in the repository's own tables.

Temperature scaling made the ECE worse: 0.1573 uncalibrated, 0.1687 after fitting. The fitted temperatures are all above 1, so they soften a model that was already too soft. This is not an error — the temperature was fitted to minimise soft cross-entropy, which it does, from 0.85064 to 0.84943. It is the correct move if you consume the distribution and the wrong move if you threshold the top-1, and the two objectives genuinely disagree here.

And the best ECE in AgentJev's whole table belongs to the checkpoint it replaced. Phase 4, before the fine-tune, scores 0.1050 — against the calibrated model's 0.1687 — while being 38.70% accurate instead of 79.25%, and it is the only row that is over-confident. The dataset card made the same point about its own Prior baseline: "Prior also has the best ECE on the table, at 0.088, while knowing nothing… That is the clearest argument for reading KL and Brier here instead of ECE." AgentJev's own table contains the demonstration, one row below the headline.

One more caveat on the Brier, since it is the column that survives. The loss is soft cross-entropy plus 0.1 times sum-of-candidates Brier, so the model is optimised directly on the metric it is reported against, on a distribution drawn from the same generator. That is not a hidden fact — protocol.json states it — but 0.0448 against Jev's 0.148 is an in-domain model trained on the scoring rule against a cold model that was not.

What the repository does that I wish more of them did

A note on evidence, because this is the most thoroughly self-audited release in this category and the contrast with its own GIFs is instructive.

verification.json records the HTTP smoke tests with their failures spelled out: question isolation moves an answer by up to 0.00282 under bf16, and the file says so rather than claiming bit-exactness. serving_verification.json checks the HTTP path against the offline evaluation path on 4 cases and 20 questions and gets a maximum probability difference of 1.07e-07. selection.json records that the checkpoint was chosen before the test split was opened. transfer_verified.json is ten SHA-256 comparisons. The Chinese serving notes go further than any English README I have read this month — my translations, so read them as such: Phase 5 was abandoned because "policy value did not improve and the stall judgment collapsed", the weights loaded are "not a random head or a 27B pretending to be Jev", and the unimplemented TreeEncoder is explicitly "not falsely claimed as a completed training speedup" — which is true, and which I only noticed because they said it.

One thing those files do not say loudly enough. Both verification.json and prefix_verification.json were produced against runs/phase4/final.pt, not the typed-decisions checkpoint on Hugging Face — you can see it in the "model": "phase4" field of every response they record. The architecture and the tensor shapes are identical so the timings carry over, but the 2.04x and the 92.4% were measured on a different set of weights from the ones you download. serving_verification.json is the one that used agentjev_v1.

That last one is worth pulling out, because it is a real limitation stated plainly. The shared-prefix runtime exists in serving only. In training, every (state, question, candidate) triple is still an independent sequence that recomputes the prefix, and TreeEncoder.forward raises NotImplementedError. The 92.4% is an inference property.

The shipped integration is a Claude Code PreToolUse hook: agentjev_hook.py posts the tool payload to the local server for Bash, Write and Edit, asks one boolean and one four-level score, and blocks only when the score is level 3 and the boolean says unsafe. It fails open on any error, which is the right posture for a gate in front of your own tools and also means the gate is advisory. It is a demo, and it has the tell — the audit log path is hardcoded to somebody's C:\Users\… directory, inside a try/except pass.

So what is actually new here

Four things, in order of how much I think they matter.

A decision model can have both halves, and the place to buy them is the head. The fork I described as a law is a law about the backbone. Isolate the candidates there — which the shared prefix does more cheaply than anyone else in this category — and then let them meet in a function that has no positions. Position carries nothing, composition carries 0.088 logits at the median, and both properties come out of the same 2.4M parameters. What I still do not know is whether that channel is wide enough to carry a reference rather than a comparison, and that is now the most interesting open question in this category, because it is the one the 0/100 was supposed to have settled architecturally.

Order-invariance is now measured somewhere, not just claimed. Zero flips across 2,520 permutations, with bit-identical backbone vectors and residuals at float32 epsilon. The hosted model the category is named after was measured this week at 12 of 100. An open 0.6B that anyone can download has the property the closed one is described as having.

Somebody trained on probabilities that are actually true. synth.py builds decision questions out of coins, urns and Markov chains whose gold distribution is a closed-form posterior, and losses.py weights that supervision at 1.0 while weighting a teacher's distribution at 0.1–0.3. Every project in this category returns a probability and every one of them has been asked what it means. This is the first answer to that question that is not a temperature fitted after the fact. Whether it transfers out of the arithmetic is unsettled, and the phase-4 row says not completely.

Everything else on the claim list is a real number with a denominator you have to read. 92.4% is tokens and buys 2.04x. 79.25% is above the benchmark's own saturation line. 0.0448 Brier is optimised for and in-domain. 68 ms is a GIF, against an unnamed 27B. None of that is dishonest — almost all of it is disclosed somewhere in the repository, and this repository discloses more than most — but the claim list and the evidence live in different files, and the claim list is the one that travels.