# Kev, and the price of not being able to see the other options
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/kev
> date: 2026-09-22
> tags: explainer, llm, architecture, calibration, training
[`jaredpalmer/kev`](https://github.com/jaredpalmer/kev) has been in these pages
once already. [Any model can be Jev](/articles/any-model-can-be-jev) used it as
the counter-example at the end of an article arguing that the System One readout
is a serving feature rather than a model feature: a LoRA on Qwen2.5-0.5B that
published an ECE before and after temperature scaling, measured its own
out-of-domain Brier against Jev's and lost, and ran the option-reversal
experiment this site had asked somebody to run — 2.78% flips for kev, 0.00% for
Jev, on 36 items.
Three days later it is a family. Qwen3.5 bases at 0.8B, 4B and 9B, a
fine-tuning script for Modal, an Apache-2.0 licence, and — the part that makes
this worth a second article — **954 tracked files including every trial's
config, hashes and results**, so the questions this site has been asking from
the outside for two weeks can be answered from the inside of somebody's ledger.
(The commit history says it was built with Devin, which is Cognition's; the
author line says Jared Palmer.)
Two of them get answered here. The first is the one my brief asked: which of the
two architecture families does this belong to? The answer is neither, and the
reason is structural. The second is the one the ledger answers by accident, and
it is better: **option-order sensitivity is not a property you have or lack. It
is a number, it shrinks with scale, and Jev's is not zero.**
## What was actually changed from stock Qwen3.5
Three things, and no more: a rank-16 LoRA over every projection, a head trained
from scratch, and a fitted temperature stored beside the head. The vocabulary
head is not loaded at all — `kev/model.py` takes `.model` off the causal LM and
throws the rest away, because nothing is ever generated.
| | Kev-0.8B | Kev-4B | Kev-9B |
|---|---:|---:|---:|
| base | Qwen3.5-0.8B-Base | Qwen3.5-4B-Base | Qwen3.5-9B-Base |
| language-model parameters | 752,393,024 | 4,205,751,296 | 7,936,684,544 |
| LoRA, r=16 (372 / 496 / 496 tensors) | 10,822,656 | 32,464,896 | 43,278,336 |
| pointer head, 2 × Linear(d → 256) | 524,800 | 1,311,232 | 2,097,664 |
| **trained, total** | **11,347,456** | **33,776,128** | **45,376,000** |
| trained share | 1.49% | 0.80% | 0.57% |
Every row above is measured, not quoted. The adapter counts come from
range-requesting each `adapter_model.safetensors` header and multiplying out the
shapes; the head counts come from `head.pt`'s file size, which pins the hidden
size at 1,024 / 2,560 / 4,096 and reproduces the model card's *"45.4M
trainable"* exactly at 9B. The base counts come from the same treatment of
Qwen's own shards.
Two things in that table are worth a second look. The 9B base checkpoint carries
a separate `lm_head.weight` of **1,017,118,720** parameters that Kev never
loads, so the thing serving your decisions is about 7.98B of weights and not 9B.
And every Qwen3.5 base ships a vision tower (100.6M / 333.5M / 456.0M) and a
multi-token-prediction module (20.5M / 120.6M / 243.3M) that never run here:
`encode` tokenises text, and nothing in the decision path emits an image token or
a speculative one. At 9B that is 700M of download with no path to the head.
The training objective is plain cross-entropy on the option distribution
(`kev/train.py: question_loss`), with the pointer head trained from scratch and
the base frozen. Bespoke's contrastive curation, the ordinal RPS term,
label smoothing, a Brier term and focal loss are all in the function as flags,
and the released models use none of them. No Jev outputs were used for training,
which the repository states and the training data supports — ten public
classification datasets plus generated policy cases.
## The third family
[The CUA-S1 piece](/articles/cua-s1-forms#two-families-and-why-the-option-order-is-the-tell)
split open decision models in two: a **vocabulary readout**, where options are
lettered text in one prompt and you slice the rows for `A`, `B` and `C` out of a
151,936-wide logit vector, and a **per-option scalar scorer**, where each option
is encoded alone and the softmax is over the scalars. It then gave a test for
telling them apart from outside: reverse the option list. A per-option scorer is
invariant by construction; a readout cannot be.
Kev is in neither. Every option is a span inside one sequence, closed by an
`` delimiter, and a trained bilinear head scores each `` hidden
state against the hidden state at ``:
```python
# kev/model.py — the entire readout
class PointerHead(nn.Module):
def __init__(self, d, dp=256):
super().__init__()
self.q, self.k = nn.Linear(d, dp), nn.Linear(d, dp)
self.scale = 1 / math.sqrt(dp)
self.temperature = 1.0
def forward(self, h_decide, h_opts): # [d], [K,d] -> logits [K]
z = (self.k(h_opts) @ self.q(h_decide)) * self.scale
return z if self.training or self.temperature == 1.0 else z / self.temperature
```
`Linear(d, 256)` twice, a dot product, one scalar per option. The output
dimension is a rank, not a class count, so the option set really is data here —
send two options or sixteen and no weight changes shape. That is the property
[CUA-S1's 706K-parameter head](/articles/cua-s1-forms) has and
[Jev-Omni's](/articles/jev-omni) `Linear(3840, 256)` does not.
What Kev does *not* have is option isolation. The mask is block-causal by
**question**: a token may read the state and its own question, and nothing of
any other question. Inside a question, option 2's span is computed in a context
that contains option 1's. `` comes last precisely so it can see the
whole list. On Qwen3.5 the packed mask is impossible anyway — the Gated DeltaNet
layers are recurrent and ignore attention masks — so each question runs as its
own causal row continuing from the shared state, which makes question isolation
exact by construction and leaves option isolation exactly where it was.
The README says so itself, in the limitations, without being asked:
*"Changing option order can change an answer. Question isolation doesn't prevent
this."*
## The measurement nobody had made
Every frozen suite in the repository ships a `permuted` variant of each Choice
question — the same evidence, the same options, the option keys shuffled under a
per-record seed — and `kev.benchmark` joins each one back to its parent, realigns
the probabilities by option key, and records two numbers: the argmax flip rate,
and the mean per-question maximum change in probability.
The repository commits 259 of those blocks — one per split per trial, across
every trial it has ever run. Pooling them by base model is fifteen minutes of
arithmetic, and it produces the first scaling curve I have seen anywhere for
this property.
**Receipts.** Option-order sensitivity in the Kev family is continuous, not binary, and it shrinks with backbone size: from 15.3% of permuted Choice questions changing their answer on the Qwen2.5-0.5B prototype to 1.8% on Qwen3.5-9B. Two reference points bracket it. The option-isolation arm — the same code with every option span made its own sub-branch — flips nothing at all and moves its probabilities by 2e-7, which is fp32 noise. Jev also flips nothing, in 324 permuted questions across seven runs, but its probabilities move by 0.020, five orders of magnitude more than the arm that is invariant by construction.
| model / arm | evals | permuted items | answer flips | flip rate | mean max Δp |
| :--- | ---: | ---: | ---: | ---: | ---: |
| Kev, option_isolation arm (0.6B / 4B / 8B) | 18 | 864 | 0 | 0.0000 | 0.0000002 |
| Jev 1.13, hosted | 7 | 324 | 0 | 0.0000 | 0.0202 |
| Kev, Qwen3.5-9B | 14 | 672 | 12 | 0.0179 | 0.0326 |
| Kev, Qwen3-8B | 16 | 768 | 20 | 0.0260 | 0.0311 |
| Kev, Qwen3.5-4B | 32 | 1,536 | 59 | 0.0384 | 0.0369 |
| Kev, Qwen3.6-35B-A3B | 4 | 192 | 11 | 0.0573 | 0.0460 |
| Kev, Qwen3.5-0.8B | 10 | 480 | 29 | 0.0604 | 0.0544 |
| Kev, Qwen3-4B | 83 | 3,972 | 241 | 0.0607 | 0.0535 |
| Kev, Qwen3-0.6B | 47 | 2,244 | 152 | 0.0677 | 0.0646 |
| Kev, Qwen2.5-0.5B prototype | 10 | 504 | 77 | 0.1528 | 0.0895 |
Jev's zero is what the site has read three times as evidence of per-option scoring. Its non-zero probability movement is the part nobody has reported, and it has a dull alternative explanation nobody has ruled out either: a hosted endpoint can be non-deterministic on identical input. The control is one line — send the same option order twice and measure the same quantity.
> method: Every trial in jaredpalmer/kev at 1c35199 commits a `permutation` block, and so does every standalone benchmark run: each frozen suite ships a `permuted` variant of every Choice question (the option keys shuffled under a per-record seed), and kev.benchmark joins each one back to its parent, realigns the probabilities by option key, and records both the argmax flip rate and the mean per-question maximum change in probability. The Kev rows below pool those blocks by base model over every trial result.json that carries a provenance config, across both the in-distribution (decision-*) and out-of-domain (transfer-*) partitions, excluding six-item smoke runs and the two research arms (option_isolation, perm_kl) — the isolation arm is listed separately on its own row. The Jev row pools the seven standalone report.json runs against the hosted model through the Vercel AI Gateway. Items repeat across trials of the same suite, so 'items' counts question-evaluations, not distinct questions: this measures a population of checkpoints, not a population of questions.
> source: https://github.com/jaredpalmer/kev
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/kev/data/option-order.json (10 rows)
Read the flip column down the Qwen3.5 rows: **6.0% at 0.8B, 3.8% at 4B, 1.8% at
9B**. The Qwen3 generation does the same thing one generation earlier (6.8% at
0.6B, 6.1% at 4B, 2.6% at 8B), and the Qwen2.5-0.5B prototype — the one the
earlier article measured at 2.78% on 36 items — sits at **15.3%** once you pool
its 504 committed item-evaluations instead of 36.
So the letter-prior story generalises to a position prior, and **capacity buys
it down**. That is a genuinely new datapoint: it means a good chunk of order
sensitivity in this family is a capacity artefact rather than a structural one,
and it predicts that the gap between "readout" and "scorer" narrows with scale
rather than staying fixed. It does not predict that the gap closes. 1.8% of 672
is twelve decisions that changed because somebody reordered a dropdown.
Two honest limits on that curve. The items repeat across trials of the same
suite, so this is a population of *checkpoints*, not of questions — thirty-two
4B evaluations are not 1,536 independent draws. And the trend is not monotone
everywhere: Qwen3.6-35B-A3B, the mixture-of-experts base tried on night two,
comes back at 5.7%, worse than the 9B dense model, on only four evaluations.
## Jev is not exactly order-invariant
Here is the part I did not expect, and it qualifies a claim three articles on
this site have leaned on.
Jev flips nothing *here*. Seven committed runs, 324 permuted Choice questions
across four different frozen suites, zero argmax changes. That is stronger
evidence than the 36 items [any model can be Jev](/articles/any-model-can-be-jev)
quoted, and it points the same way.
The qualifier is load-bearing, because on somebody else's suite it does flip.
[Order is not noise](/articles/jev-is-not-deterministic), published the same day
as this piece, recomputes a third party's raw API logs and finds the hosted model
changing its answer on **12 of 100** permuted items — on Banking77, at 77
options. Kev's frozen suites are narrow by comparison: the widest option count
any of them names is Emotion's six. So the two measurements are not in conflict,
and what they differ on is cardinality, which nobody has swept. Hold that
alongside everything below.
But even here the second column is not zero. Jev's mean maximum probability movement under
permutation is **0.0161 to 0.0249**, weighted mean 0.0202. And the repository
contains a control for what zero actually looks like: the `option_isolation`
arm, nine trials in which every option span is its own sub-branch at a shared
position, comes back at **1.2e-07**. That is fp32 noise, measured on a run where
invariance is a property of the tensor layout.
A model that is order-invariant by construction moves by 1e-7. Jev moves by
0.02. That is five orders of magnitude, and it is about a third of the movement
of an ordinary Kev-9B.
Kev's own `PLAN.md` reached this before I did and wrote it in one line under
*Evidence and corrections*: *"Jev's zero observed argmax flips do not prove
architectural invariance. Its probabilities move under permutation."* And in the
open-questions list at the bottom: *"Deferred: option-order architecture
experiments. Do not infer Jev's architecture from zero argmax flips."*
So what does 0.02 mean? Three readings, and I cannot separate them from here.
1. **Jev's options share a context after all**, and it is simply very good at
ignoring the order — which would make it a large, well-trained version of
exactly what Kev is, and would make [the 0/100 on relational
choice](/articles/jev-scores-zero) a training outcome rather than a
structural impossibility.
2. **Jev is a per-option scorer with something order-dependent downstream** —
batching, a shared prefill, a scheduler that packs requests differently.
3. **The hosted endpoint is not deterministic**, and 0.02 is what you would
measure by sending the *same* request twice.
Reading three is the boring one, and it stopped being hypothetical while I was
writing this. The control is one line of code — send the identical option order
twice and compute the same statistic — and
[Order is not noise](/articles/jev-is-not-deterministic) found it already run, by
accident, inside a third party's benchmark that repeated every item three times
to take a latency median. Over 1,300 items with byte-identical request bodies,
**67.2% came back with a different probability vector and 3.3% changed their
answer**, mean movement 0.036.
That is not a number I can subtract from mine. It is a different suite, a
different cardinality, and a maximum taken over 77 options rather than over a
handful, so 0.036 and 0.0202 are not two measurements of one quantity.
What it does do is move reading three from "nobody has looked" to "somebody
looked somewhere else and found plenty." The version of the control that would
settle *this* ledger is the same protocol on Kev's own frozen suites, which is a
`--repeat 3` flag on `kev.benchmark` and nobody's afternoon.
*"Jev is exactly order-invariant"* is not a measurement anybody has. What is measured is that it does not flip, on 324 questions from suites that name nothing wider than six options, which is a weaker and still useful claim — and that on a 77-option suite it flips 12 of 100, which is the [companion piece's](/articles/jev-is-not-deterministic) finding and hard to reconcile with a strict per-option scorer at that width — with the caveat that piece states itself, that its order flips and its repeat flips sit on the same hundred items.
## What invariance costs, from somebody who built it
The reason this ledger is worth more than another benchmark table is that Kev
implemented both available fixes for order sensitivity and then declined to ship
either, with numbers.
**The architectural fix.** `encode(..., option_isolation=True)` gives every
option span its own sub-branch, puts all spans at the same position ids, and
parks `` at one fixed position after the longest span. Permutation
invariance is then exact by construction, and the eighteen evaluations that used
it report exactly that: 0 flips in 864 items, mean max Δp between 7.2e-08 and
6.7e-07. It was run at 0.6B, 4B and 8B. At 0.6B it was free and bought nothing.
At 4B, against its matched low-learning-rate control, `PLAN.md` records
*"option isolation at low lr 0.729 (−5.8 pp, significant) — isolation costs
accuracy at 4B."*
**The loss fix.** `--perm_kl` runs a second forward pass on a shuffled option
list and adds a symmetric KL between the two distributions. It is in
`kev/train.py`, it is in the config space the overnight search explored, and the
trials that used it still flip — 1 to 3 items of 36 — while landing inside the
noise band on accuracy. `PLAN.md` files it under what did not work.
And then the constraint that makes this more than a tradeoff. On Qwen3.5 the
architectural fix is not available at all:
```python
# kev/model.py — DecisionModel.__init__
if self.hybrid and option_isolation:
raise ValueError("option_isolation needs the packed mask; not available on hybrid backbones")
```
The packed block-causal mask is what isolation is made of, and a Gated DeltaNet
layer is recurrent — it does not consult an attention mask, so there is nothing
to write the isolation into. Moving to the newest open base family therefore
*removed* the ability to buy the property Jev is admired for. That is the same
direction of travel [the Apple silicon piece](/articles/laya-mlx) found for the
option count: going closer to the metal, or closer to the frontier, costs you
more of the structure, not less.
## What the training buys that a serving flag does not
[Any model can be Jev](/articles/any-model-can-be-jev) argued that the readout is
free — SGLang has shipped `/v1/score` since June 2025 — and that calibration is
the half you pay for. Kev is a test of that claim and it does not confirm it: it
is a trained checkpoint with a trained head, not a wrapper, and its own
comparison says the training is doing work. Its accuracy at 9B out of domain is
0.822 development and 0.852 on a locked test read once, against Jev's 0.857.
But the more interesting number is the one the release leads its limitations
with, and it is not accuracy or ECE:
Coverage at an error budget is the share of decisions you can accept in
confidence order before the accepted set exceeds 5% error. It is the metric that
actually decides whether a decision model saves you money, and Kev's own cards
publish the gap without softening it: 0.45–0.47 against Jev's 0.70, while
accuracy is within 3.5 points and ECE after the built-in temperature is *better*
than Jev's on some suites (0.042 against 0.049 out of domain).
`PLAN.md` states the diagnosis precisely: *"After the built-in temperature,
Kev's probabilities have the right scale … but not the right order."* A single
temperature is monotone within a question, so it cannot reorder which decisions
look most confident. Sixteen of 26 high-confidence errors at 9B are PAWS
adversarial paraphrases — items where a hard-label cross-entropy objective
demanded certainty on genuinely ambiguous rows, which is the textbook route to
an overconfident fine-tuned classifier.
So the refinement to the earlier article's thesis is this. The readout is free.
A *calibration number* costs a few hundred labelled rows and an afternoon. What
costs a research programme is **confidence that ranks correctly across your whole
traffic**, and nobody in the open has bought it yet.
Which is also, for once, a product argument rather than a benchmark argument. The
`kev-finetune` skill states it in one sentence: *"Jev … is a fixed hosted model:
on the user's data it is out of distribution and its probabilities cannot be
recalibrated."* You cannot fit a temperature on somebody else's weights. The
skill's own reported datapoint for what that is worth: on 836 support-tool
decisions, a fine-tune started from the base scored 0.33 on Kev's evaluation set
against 0.84 for the released model, while the same data started from the
released checkpoint with `--init_from` kept 0.83 there and reached 0.88 on the
new domain.
## Credit where the ledger is uncomfortable
The experimental hygiene in this repository is better than in any release this
series has read, and most of it cuts against the author.
- **The locked test is read once per candidate**, with the criteria written down
before the run. The 35B mixture-of-experts trial met its screening gate,
trained, landed at 0.823 transfer — and was **not shipped**, because the
pre-registered bar was Kev-9B plus 2 points and it came in at plus 1.2 with
worse calibration and eight times the memory. The checkpoint is on a volume;
the row is in the leaderboard; nothing was quietly promoted.
- **Failures are kept.** The `runs/` tree includes trials that crashed, arms
that regressed, and a metric audit that found the previous coverage metric
could return 0.94 or 0.00 for the same predictions under a row permutation —
published as a correction, with Jev's own recomputed number (0.704 → 0.695)
alongside.
- **An earlier statement in the plan is marked wrong by its own author**: *"The
previous statement that temperature cannot reorder confidence was incorrect."*
- **The external suites are other people's.** SemIf's 144 authored decisions
(Kev-9B 0.917, live Jev 0.965) and scienthoon's 900 support tickets (Kev-9B 0.952 on
routing against Jev's 0.897, and behind Jev on tone) are run on the same items
as their published Jev numbers. Kev wins one of those and loses the other, and
both are in the README.
That last one deserves its own sentence. The experiment [the CUA-S1
piece](/articles/cua-s1-forms) proposed in September as *"one afternoon and a
reversed list"* is now a button in somebody's playground and a route on their
server — `POST /v1/systemone/permute`, six orders, returns `argmax_stable` and
the per-option spread. The fastest way to find out which family a model belongs
to has become a UI affordance, which is roughly the best outcome a falsifier can
have.
## What I would actually take from this
- **Kev is a trained checkpoint, not a serving wrapper.** 45.4M trained
parameters at 9B, cross-entropy on the option distribution, a head from
scratch. The "readout is a serving feature" thesis survives as a statement
about the *readout*; this release is evidence that it stops being true one
level up.
- **Order sensitivity is a dial, not a switch**, and the dial is mostly
capacity: 15.3% → 6.0% → 3.8% → 1.8% across four backbone sizes on one recipe.
- **Exact invariance has a measured price for the first time**: −5.8 points at
4B, and on hybrid bases it is not for sale at any price.
- **Jev's zero is a flip rate on narrow questions, not an invariance.** Its
probabilities move by 0.0202 under permutation here, and on a 77-option suite
its answer moves too — 12 of 100. The repeat-request control that would tell
us how much of either is the server has been run on that suite and not on
this one.
- **The remaining gap is confidence ordering, not accuracy.** 0.45 against 0.70
coverage at a 5% error budget is the number to beat, and no open release has
come near it.
Send the same request twice — identical options, identical order — several hundred times through the same gateway, and compute the same statistic `kev.benchmark.summarize` computes. If the repeat-control lands near 0.02, every permutation number ever published for the hosted model has been measuring server noise and this section is wrong. If it lands at 0.000 while the permuted runs land at 0.02, the movement is the ordering and Jev's options share a context. On a different suite this has been partly answered and the answer is unfavourable to me: a third party's byte-identical repeats move the vector on 67.2% of 1,300 items, mean 0.036, which is larger than my 0.0202 — but on 77 options rather than on Kev's handful, so it bounds nothing here directly. It is why the three readings above stay unseparated rather than collapsing onto the ordering.
The curve above pools evaluations that reuse the same 36 or 60 permuted items, so it has far fewer independent draws than it looks like. Freeze one recipe, generate 500 fresh permuted Choice questions the checkpoints have never seen, and run 0.8B / 4B / 9B once each. If the three flip rates land inside one confidence interval of each other, the trend is trial-level noise and capacity is not the lever.
That figure is one comparison against one matched control in an overnight search, at Qwen3, on transfer-v4. Run three seeds of `option_isolation=1` against three seeds of the identical config with the flag off, at 4B, and report the paired bootstrap. If the interval crosses zero, isolation is free and the reason it is not shipped is the hybrid-backbone constraint alone.
`sum(p.numel() for p in model.lm.parameters())` after `Checkpoint.load`, plus the head. I get 7,936,684,544 for the language model from the safetensors headers and 45,376,000 trained on top, with the 1,017,118,720-parameter `lm_head` never loaded because `kev/model.py` takes `.model`. If a real load comes back at 9.65B, something is pulling in the vision tower or the vocabulary head and my count is wrong.
Coverage at a 5% error budget is an in-sample maximum over thresholds, which the repository flags in its own metric policy as "not a deployed error guarantee". Fit the threshold on the development partition and apply it unchanged to the locked test for both models. If Kev's coverage holds near 0.45 and Jev's near 0.70 out of sample, the ordering diagnosis stands; if both collapse, the metric was optimistic for both and the gap is smaller than it looks.
---
*Nothing here was executed against a model. There is no GPU in the machine this was written on, so every accuracy, latency and calibration figure is Reported — read out of `jaredpalmer/kev` at `1c35199`, cloned rather than summarised. What is Measured is the arithmetic I did on those artifacts: parameter counts summed from range-requested safetensors headers for three Kev adapters and three Qwen3.5 bases, head sizes derived from `head.pt`'s storage layout, and the permutation ledger pooled from the `permutation` block of every committed trial `result.json`, with the hosted
model's rows coming from the seven standalone Jev `report.json` runs (dataset: [option-order.json](/articles/kev/data/option-order.json)). Jev's figures are the repository's own runs through the Vercel AI Gateway, budget-capped, with usage accounting committed beside them. Companion pieces: [A System One model in 706,048 parameters](/articles/cua-s1-forms) for the two families this one sits between, [Any model can be Jev](/articles/any-model-can-be-jev) for the serving-feature argument it tests, [Jev-Omni](/articles/jev-omni) for the release that went the other way and welded the option count into a trained head, and [Order is not noise](/articles/jev-is-not-deterministic) for the same hosted model measured at 77 options, where it does flip.*