Kev, and the price of not being able to see the other options

jaredpalmer/kev has been in these pages once already. Any model can be Jev used it as the counter-example at the end of an article arguing that the System One readout is a serving feature rather than a model feature: a LoRA on Qwen2.5-0.5B that published an ECE before and after temperature scaling, measured its own out-of-domain Brier against Jev's and lost, and ran the option-reversal experiment this site had asked somebody to run — 2.78% flips for kev, 0.00% for Jev, on 36 items.

Three days later it is a family. Qwen3.5 bases at 0.8B, 4B and 9B, a fine-tuning script for Modal, an Apache-2.0 licence, and — the part that makes this worth a second article — 954 tracked files including every trial's config, hashes and results, so the questions this site has been asking from the outside for two weeks can be answered from the inside of somebody's ledger. (The commit history says it was built with Devin, which is Cognition's; the author line says Jared Palmer.)

Two of them get answered here. The first is the one my brief asked: which of the two architecture families does this belong to? The answer is neither, and the reason is structural. The second is the one the ledger answers by accident, and it is better: option-order sensitivity is not a property you have or lack. It is a number, it shrinks with scale, and Jev's is not zero.

What was actually changed from stock Qwen3.5

Three things, and no more: a rank-16 LoRA over every projection, a head trained from scratch, and a fitted temperature stored beside the head. The vocabulary head is not loaded at all — kev/model.py takes .model off the causal LM and throws the rest away, because nothing is ever generated.

Every row above is measured, not quoted. The adapter counts come from range-requesting each adapter_model.safetensors header and multiplying out the shapes; the head counts come from head.pt's file size, which pins the hidden size at 1,024 / 2,560 / 4,096 and reproduces the model card's "45.4M trainable" exactly at 9B. The base counts come from the same treatment of Qwen's own shards.

Two things in that table are worth a second look. The 9B base checkpoint carries a separate lm_head.weight of 1,017,118,720 parameters that Kev never loads, so the thing serving your decisions is about 7.98B of weights and not 9B. And every Qwen3.5 base ships a vision tower (100.6M / 333.5M / 456.0M) and a multi-token-prediction module (20.5M / 120.6M / 243.3M) that never run here: encode tokenises text, and nothing in the decision path emits an image token or a speculative one. At 9B that is 700M of download with no path to the head.

The training objective is plain cross-entropy on the option distribution (kev/train.py: question_loss), with the pointer head trained from scratch and the base frozen. Bespoke's contrastive curation, the ordinal RPS term, label smoothing, a Brier term and focal loss are all in the function as flags, and the released models use none of them. No Jev outputs were used for training, which the repository states and the training data supports — ten public classification datasets plus generated policy cases.

The third family

The CUA-S1 piece split open decision models in two: a vocabulary readout, where options are lettered text in one prompt and you slice the rows for A, B and C out of a 151,936-wide logit vector, and a per-option scalar scorer, where each option is encoded alone and the softmax is over the scalars. It then gave a test for telling them apart from outside: reverse the option list. A per-option scorer is invariant by construction; a readout cannot be.

Kev is in neither. Every option is a span inside one sequence, closed by an </opt> delimiter, and a trained bilinear head scores each </opt> hidden state against the hidden state at <decide>:

# kev/model.py — the entire readout
class PointerHead(nn.Module):
    def __init__(self, d, dp=256):
        super().__init__()
        self.q, self.k = nn.Linear(d, dp), nn.Linear(d, dp)
        self.scale = 1 / math.sqrt(dp)
        self.temperature = 1.0
 
    def forward(self, h_decide, h_opts):  # [d], [K,d] -> logits [K]
        z = (self.k(h_opts) @ self.q(h_decide)) * self.scale
        return z if self.training or self.temperature == 1.0 else z / self.temperature

Linear(d, 256) twice, a dot product, one scalar per option. The output dimension is a rank, not a class count, so the option set really is data here — send two options or sixteen and no weight changes shape. That is the property CUA-S1's 706K-parameter head has and Jev-Omni's Linear(3840, 256) does not.

What Kev does not have is option isolation. The mask is block-causal by question: a token may read the state and its own question, and nothing of any other question. Inside a question, option 2's span is computed in a context that contains option 1's. <decide> comes last precisely so it can see the whole list. On Qwen3.5 the packed mask is impossible anyway — the Gated DeltaNet layers are recurrent and ignore attention masks — so each question runs as its own causal row continuing from the shared state, which makes question isolation exact by construction and leaves option isolation exactly where it was.

The README says so itself, in the limitations, without being asked: "Changing option order can change an answer. Question isolation doesn't prevent this."

The measurement nobody had made

Every frozen suite in the repository ships a permuted variant of each Choice question — the same evidence, the same options, the option keys shuffled under a per-record seed — and kev.benchmark joins each one back to its parent, realigns the probabilities by option key, and records two numbers: the argmax flip rate, and the mean per-question maximum change in probability.

The repository commits 259 of those blocks — one per split per trial, across every trial it has ever run. Pooling them by base model is fifteen minutes of arithmetic, and it produces the first scaling curve I have seen anywhere for this property.

Read the flip column down the Qwen3.5 rows: 6.0% at 0.8B, 3.8% at 4B, 1.8% at 9B. The Qwen3 generation does the same thing one generation earlier (6.8% at 0.6B, 6.1% at 4B, 2.6% at 8B), and the Qwen2.5-0.5B prototype — the one the earlier article measured at 2.78% on 36 items — sits at 15.3% once you pool its 504 committed item-evaluations instead of 36.

So the letter-prior story generalises to a position prior, and capacity buys it down. That is a genuinely new datapoint: it means a good chunk of order sensitivity in this family is a capacity artefact rather than a structural one, and it predicts that the gap between "readout" and "scorer" narrows with scale rather than staying fixed. It does not predict that the gap closes. 1.8% of 672 is twelve decisions that changed because somebody reordered a dropdown.

Two honest limits on that curve. The items repeat across trials of the same suite, so this is a population of checkpoints, not of questions — thirty-two 4B evaluations are not 1,536 independent draws. And the trend is not monotone everywhere: Qwen3.6-35B-A3B, the mixture-of-experts base tried on night two, comes back at 5.7%, worse than the 9B dense model, on only four evaluations.

Jev is not exactly order-invariant

Here is the part I did not expect, and it qualifies a claim three articles on this site have leaned on.

Jev flips nothing here. Seven committed runs, 324 permuted Choice questions across four different frozen suites, zero argmax changes. That is stronger evidence than the 36 items any model can be Jev quoted, and it points the same way.

The qualifier is load-bearing, because on somebody else's suite it does flip. Order is not noise, published the same day as this piece, recomputes a third party's raw API logs and finds the hosted model changing its answer on 12 of 100 permuted items — on Banking77, at 77 options. Kev's frozen suites are narrow by comparison: the widest option count any of them names is Emotion's six. So the two measurements are not in conflict, and what they differ on is cardinality, which nobody has swept. Hold that alongside everything below.

But even here the second column is not zero. Jev's mean maximum probability movement under permutation is 0.0161 to 0.0249, weighted mean 0.0202. And the repository contains a control for what zero actually looks like: the option_isolation arm, nine trials in which every option span is its own sub-branch at a shared position, comes back at 1.2e-07. That is fp32 noise, measured on a run where invariance is a property of the tensor layout.

A model that is order-invariant by construction moves by 1e-7. Jev moves by 0.02. That is five orders of magnitude, and it is about a third of the movement of an ordinary Kev-9B.

Kev's own PLAN.md reached this before I did and wrote it in one line under Evidence and corrections: "Jev's zero observed argmax flips do not prove architectural invariance. Its probabilities move under permutation." And in the open-questions list at the bottom: "Deferred: option-order architecture experiments. Do not infer Jev's architecture from zero argmax flips."

So what does 0.02 mean? Three readings, and I cannot separate them from here.

1. Jev's options share a context after all, and it is simply very good at ignoring the order — which would make it a large, well-trained version of exactly what Kev is, and would make the 0/100 on relational choice a training outcome rather than a structural impossibility.
2. Jev is a per-option scorer with something order-dependent downstream — batching, a shared prefill, a scheduler that packs requests differently.
3. The hosted endpoint is not deterministic, and 0.02 is what you would measure by sending the same request twice.

Reading three is the boring one, and it stopped being hypothetical while I was writing this. The control is one line of code — send the identical option order twice and compute the same statistic — and Order is not noise found it already run, by accident, inside a third party's benchmark that repeated every item three times to take a latency median. Over 1,300 items with byte-identical request bodies, 67.2% came back with a different probability vector and 3.3% changed their answer, mean movement 0.036.

That is not a number I can subtract from mine. It is a different suite, a different cardinality, and a maximum taken over 77 options rather than over a handful, so 0.036 and 0.0202 are not two measurements of one quantity. What it does do is move reading three from "nobody has looked" to "somebody looked somewhere else and found plenty." The version of the control that would settle this ledger is the same protocol on Kev's own frozen suites, which is a --repeat 3 flag on kev.benchmark and nobody's afternoon.

What invariance costs, from somebody who built it

The reason this ledger is worth more than another benchmark table is that Kev implemented both available fixes for order sensitivity and then declined to ship either, with numbers.

The architectural fix. encode(..., option_isolation=True) gives every option span its own sub-branch, puts all spans at the same position ids, and parks <decide> at one fixed position after the longest span. Permutation invariance is then exact by construction, and the eighteen evaluations that used it report exactly that: 0 flips in 864 items, mean max Δp between 7.2e-08 and 6.7e-07. It was run at 0.6B, 4B and 8B. At 0.6B it was free and bought nothing. At 4B, against its matched low-learning-rate control, PLAN.md records "option isolation at low lr 0.729 (−5.8 pp, significant) — isolation costs accuracy at 4B."

The loss fix. --perm_kl runs a second forward pass on a shuffled option list and adds a symmetric KL between the two distributions. It is in kev/train.py, it is in the config space the overnight search explored, and the trials that used it still flip — 1 to 3 items of 36 — while landing inside the noise band on accuracy. PLAN.md files it under what did not work.

And then the constraint that makes this more than a tradeoff. On Qwen3.5 the architectural fix is not available at all:

# kev/model.py — DecisionModel.__init__
if self.hybrid and option_isolation:
    raise ValueError("option_isolation needs the packed mask; not available on hybrid backbones")

The packed block-causal mask is what isolation is made of, and a Gated DeltaNet layer is recurrent — it does not consult an attention mask, so there is nothing to write the isolation into. Moving to the newest open base family therefore removed the ability to buy the property Jev is admired for. That is the same direction of travel the Apple silicon piece found for the option count: going closer to the metal, or closer to the frontier, costs you more of the structure, not less.

What the training buys that a serving flag does not

Any model can be Jev argued that the readout is free — SGLang has shipped /v1/score since June 2025 — and that calibration is the half you pay for. Kev is a test of that claim and it does not confirm it: it is a trained checkpoint with a trained head, not a wrapper, and its own comparison says the training is doing work. Its accuracy at 9B out of domain is 0.822 development and 0.852 on a locked test read once, against Jev's 0.857.

But the more interesting number is the one the release leads its limitations with, and it is not accuracy or ECE:

Coverage at an error budget is the share of decisions you can accept in confidence order before the accepted set exceeds 5% error. It is the metric that actually decides whether a decision model saves you money, and Kev's own cards publish the gap without softening it: 0.45–0.47 against Jev's 0.70, while accuracy is within 3.5 points and ECE after the built-in temperature is better than Jev's on some suites (0.042 against 0.049 out of domain).

PLAN.md states the diagnosis precisely: "After the built-in temperature, Kev's probabilities have the right scale … but not the right order." A single temperature is monotone within a question, so it cannot reorder which decisions look most confident. Sixteen of 26 high-confidence errors at 9B are PAWS adversarial paraphrases — items where a hard-label cross-entropy objective demanded certainty on genuinely ambiguous rows, which is the textbook route to an overconfident fine-tuned classifier.

So the refinement to the earlier article's thesis is this. The readout is free. A calibration number costs a few hundred labelled rows and an afternoon. What costs a research programme is confidence that ranks correctly across your whole traffic, and nobody in the open has bought it yet.

Which is also, for once, a product argument rather than a benchmark argument. The kev-finetune skill states it in one sentence: "Jev … is a fixed hosted model: on the user's data it is out of distribution and its probabilities cannot be recalibrated." You cannot fit a temperature on somebody else's weights. The skill's own reported datapoint for what that is worth: on 836 support-tool decisions, a fine-tune started from the base scored 0.33 on Kev's evaluation set against 0.84 for the released model, while the same data started from the released checkpoint with --init_from kept 0.83 there and reached 0.88 on the new domain.

Credit where the ledger is uncomfortable

expand

The experimental hygiene in this repository is better than in any release this series has read, and most of it cuts against the author.

• The locked test is read once per candidate, with the criteria written down before the run. The 35B mixture-of-experts trial met its screening gate, trained, landed at 0.823 transfer — and was not shipped, because the pre-registered bar was Kev-9B plus 2 points and it came in at plus 1.2 with worse calibration and eight times the memory. The checkpoint is on a volume; the row is in the leaderboard; nothing was quietly promoted.
• Failures are kept. The runs/ tree includes trials that crashed, arms that regressed, and a metric audit that found the previous coverage metric could return 0.94 or 0.00 for the same predictions under a row permutation — published as a correction, with Jev's own recomputed number (0.704 → 0.695) alongside.
• An earlier statement in the plan is marked wrong by its own author: "The previous statement that temperature cannot reorder confidence was incorrect."
• The external suites are other people's. SemIf's 144 authored decisions (Kev-9B 0.917, live Jev 0.965) and scienthoon's 900 support tickets (Kev-9B 0.952 on routing against Jev's 0.897, and behind Jev on tone) are run on the same items as their published Jev numbers. Kev wins one of those and loses the other, and both are in the README.

expand

That last one deserves its own sentence. The experiment the CUA-S1 piece proposed in September as "one afternoon and a reversed list" is now a button in somebody's playground and a route on their server — POST /v1/systemone/permute, six orders, returns argmax_stable and the per-option spread. The fastest way to find out which family a model belongs to has become a UI affordance, which is roughly the best outcome a falsifier can have.

What I would actually take from this

• Kev is a trained checkpoint, not a serving wrapper. 45.4M trained parameters at 9B, cross-entropy on the option distribution, a head from scratch. The "readout is a serving feature" thesis survives as a statement about the readout; this release is evidence that it stops being true one level up.
• Order sensitivity is a dial, not a switch, and the dial is mostly capacity: 15.3% → 6.0% → 3.8% → 1.8% across four backbone sizes on one recipe.
• Exact invariance has a measured price for the first time: −5.8 points at 4B, and on hybrid bases it is not for sale at any price.
• Jev's zero is a flip rate on narrow questions, not an invariance. Its probabilities move by 0.0202 under permutation here, and on a 77-option suite its answer moves too — 12 of 100. The repeat-request control that would tell us how much of either is the server has been run on that suite and not on this one.
• The remaining gap is confidence ordering, not accuracy. 0.45 against 0.70 coverage at a 5% error budget is the number to beat, and no open release has come near it.

Nothing here was executed against a model. There is no GPU in the machine this was written on, so every accuracy, latency and calibration figure is Reported — read out of jaredpalmer/kev at 1c35199, cloned rather than summarised. What is Measured is the arithmetic I did on those artifacts: parameter counts summed from range-requested safetensors headers for three Kev adapters and three Qwen3.5 bases, head sizes derived from head.pt's storage layout, and the permutation ledger pooled from the permutation block of every committed trial result.json, with the hosted model's rows coming from the seven standalone Jev report.json runs (dataset: option-order.json). Jev's figures are the repository's own runs through the Vercel AI Gateway, budget-capped, with usage accounting committed beside them. Companion pieces: A System One model in 706,048 parameters for the two families this one sits between, Any model can be Jev for the serving-feature argument it tests, Jev-Omni for the release that went the other way and welded the option count into a trained head, and Order is not noise for the same hosted model measured at 77 options, where it does flip.