# A System One model in 706,048 parameters
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/cua-s1-forms
> date: 2026-09-18
> tags: explainer, llm, training, architecture, agents
The [previous piece](/articles/rlcd-calibrated-decisions) had to reconstruct Jev's
architecture from two context limits and a sentence about scoring levels
separately, because TypeSafe publishes fourteen words about it. A week later that
is no longer the only way to find out. Three projects have rebuilt the shape in
the open, with weights, data and training code, and between them they answer most
of what the closed version leaves out.
They also, all three, leave out most of the same thing — and two of them leave
out less of it than I first credited. More on that at the end.
## The smallest one is the most informative
[`cua-ai/cua-s1-forms`](https://huggingface.co/cua-ai/cua-s1-forms), from the
[trycua](https://github.com/trycua/cua) team, is **706,048 parameters**. Not
billions. Not millions. Seven hundred thousand, in a 2.8 MB checkpoint, trained
from scratch — no base model, no pretrained encoder, no LoRA on top of something
else.
**Receipts.** Three open System One releases in one week, three different answers to the same architectural question: where does the probability come from? Only one of them builds the scorer from scratch; the other two put a decision head on top of a general model. Every figure below is quoted from the release it belongs to.
| release | base | new params | what the probability spreads over | objective | calibration reported |
| :--- | :--- | ---: | :--- | :--- | :--- |
| cua-ai/cua-s1-forms | none — trained from scratch | 706,048 | one scalar per option; softmax over the live option count | cross-entropy over the live option count | ECE 0.000148 (22,054 val rows) — in cua-s1-forms.json, not the card |
| bespokelabs/Bespoke-Nimble-9B | Qwen3.5-9B (LoRA r=16) | adapter only, ~193 MB on disk | allowed answer tokens, scored directly | hard reference labels from rule application; no RL, no distillation | ECE + Brier on 13 public datasets, 3,880 rows, added 2026-09-19 |
| browser-use/jev-ultrafast | calls the hosted Jev API | no model of its own | an index into the runtime's validated action list | not a training project | none — not a training project |
| TypeSafe Jev 1.13 | not disclosed | not disclosed | up to 255 Choice options / 2-10 Score levels / 1 Noul | not disclosed | none first-party; third-party ECE + Brier since 2026-09-19 |
cua-s1-forms is the only one of the three whose parameter count and checkpoint size independently agree: 706,048 parameters at fp32 is 2,824,192 bytes, which is the 2.8 MB the card claims for state_dict plus config plus training history. That is a weak check, but it is the kind that catches a card written from memory rather than from the artifact — and reading that same sidecar is what corrected the calibration column. cua-s1-forms does publish an expected calibration error; it lives in metadata.best_validation, on a split where top-1 is 0.99941, which is where a near-zero ECE comes from. Bespoke's public suite, added the day after this article, is the first calibration measurement of Jev by anyone, and a competitor ran it.
> method: Read from each project's own model card, dataset card, README and config on 2026-09-18, and re-checked on 2026-09-19 against the published artifacts themselves rather than the prose about them. Parameter counts and checkpoint sizes are the ones each project states; where a size and a count can be cross-checked against each other, the note says so. The calibration column was wrong in the first version: it read "none" for every row.
> captured: 2026-09-19
> data: https://ai.thesatyajit.com/articles/cua-s1-forms/data/anatomy.json (4 rows)
Its own card describes the architecture plainly enough to redraw:
**"Byte-level embedding + 2-layer Transformer encoder (width 128, 4 heads)"** processing context and options separately, then an `AttentionHead` where **"each option becomes a query against the context tokens, producing an attended context vector, then a shared dot product turns each (option, attended-context) pair into one logit."**
That is the per-option scalar scorer the last article argued Jev must be, built
and shipped under MIT. Drawn:
The detail that matters is the head's output dimension. It is **one**. Not one
per class — one, full stop, evaluated once per option. Which is why the training
objective is *"cross-entropy over the live option count"*: the softmax is taken
over however many options the caller happened to send, and nothing in the weights
depends on that number. Send two, send sixteen; the parameter count does not
move.
This is also why the comparison to a classifier is the wrong one. A classifier
has `Linear[K, hidden]` with K welded into the weights and cannot accept an
option it was not trained on. This head has `Linear[…, 1]`, so the option set is
data. The idea is old — `BertForMultipleChoice` has worked this way since 2019 —
and the interesting part is not the mechanism but that 706K parameters is enough
of it.
### How it was trained
The [dataset](https://huggingface.co/datasets/cua-ai/cua-s1-forms) is
**234,067 rows** — roughly 150k train, 18k validation, 20k test, plus 196 real
decisions — and it is entirely synthetic. Each episode is assembled from three
generators: a random form of 2–16 fields drawn from a 55-concept catalogue, a
random synthetic person, and a random document built from that person's data.
The design decision worth stealing is in the third generator. Documents are
padded with *"distractor entities and forced look-alike confuser pairs"* — email
against street address, phone against emergency-contact phone — so that, in the
dataset's own words, *"the model has to read the whole label rather than
pattern-match on a keyword."* Window titles get random app suffixes, omitted
about 20% of the time. The negative space is constructed, not sampled.
The options for each decision are document pointers (`fill : `)
plus three fixed actions — check, click, skip — and the label is a zero-based
index into that list. Training is unremarkable on purpose: AdamW, cosine schedule
with warmup, 6 epochs, batch size 128.
The split discipline is better than the training recipe. Splits are *"disjoint by
the exact sorted set of element descriptions in an episode"*, so a test form
cannot appear in training under a different person's data. That is a real leak
closed, and it is the reason the headline number is worth anything at all.
### The number, and its denominator
99.95% on the form-disjoint synthetic test. 100% on 196 real decisions across
three forms and three PDFs. And against the hosted Jev API on that same real
eval: **99.7% versus 83.6%**.
Read that last one carefully, because it is doing less work than it looks like.
CUA-S1 was trained on 150,000 rows from the generator that also produced the
evaluation; Jev has never seen any of it. That is a specialist measured at home
against a generalist measured cold. The honest claim is *a 706K-parameter
specialist beats a general hosted model on the narrow task it was built for*,
which is genuinely the thesis of small specialist models and is worth
demonstrating — but it is not evidence that Jev is worse at what Jev does.
To the team's credit, the [source release](https://github.com/trycua/cua/tree/main/libs/cua-s1)
says as much itself: *"No checkpoint performance claim is established by this
source-only release."*
## Two families, and why the option order is the tell
Everything above glosses over a split that matters more than any of it. There
are **two** ways to make a model answer from a caller-supplied option list, they
are not variations on a theme, and you can tell them apart from the outside by
one experiment.
They are also, stripped of everything else, two short functions. This site has
leaned on the distinction for three articles without once putting them next to
each other, so here they are, as they are actually written.
**Vocabulary readout.** The options go in the prompt as lettered text, and the
model only ever picks a letter. This is what `openjev` does, and its
`src/semif_phase1/direct.py` is the clearest statement of it anywhere:
```python
# openjev · src/semif_phase1/direct.py @ b9cb325 — device/timing setup and the `inputs` dict elided
def score(model, tokenizer, row: dict, metadata: dict, max_tokens: int = 4096) -> dict:
ids, slots, prompt_hash = encode_prompt(tokenizer, row, max_tokens)
with torch.inference_mode():
vocabulary = _forward(model, inputs)[0].float() # [151936] — the last position
selected = vocabulary[slots].cpu().tolist() # slots: the ids for "A", "B", "C"
return {
"probabilities": softmax(selected),
"option_logits": selected,
"readout": "native full-vocabulary last-position logits restricted to declared answer slots",
"probability_status": "conditional option score; uncalibrated as decision confidence",
}
```
One forward pass over one prompt, then a fancy-index into a 151,936-long vector.
The option *count* never appears in that function; `slots` does, and `slots` is a
list of token ids.
The prompt is JSON — `{"evidence": …, "criterion": …, "options": [{"letter": "A", "description": …}]}` —
and the system message says *"Respond with only its uppercase letter."* The
model's dynamic option set is just text it reads; the thing it actually chooses
between is sixteen fixed vocabulary tokens. `LETTERS = "ABCDEFGHIJKLMNOP"`, which
is why the validator caps options at 16. There is even a guard, `_slot_ids`,
that refuses to run unless each letter is *"one exact round-trip token"* — because
if `A` tokenised as anything else you could not slice its row out.
**Per-option scalar scorer.** Each option's own text is encoded and scored alone,
and the softmax is over the scalars. This is CUA-S1 above, and `open-jev-deberta`,
and — per the last article's argument — Jev. The entire family is this `forward`:
```python
# cua-s1 · libs/cua-s1/python/src/cua_s1/model.py @ 9bbfa7d — AttentionHead.forward
def forward(self, context, context_mask, options, option_mask, shuffle_context=False):
context = self.context_norm(context.float()) # [B, L, width] — the context tokens
options = self.option_norm(options.float()) # [B, N, width] — one vector per option
query = self.query(options)
key = self.key(context)
value = self.value(context)
scores = torch.einsum("bnr,blr->bnl", query, key) / math.sqrt(self.rank)
scores = scores.masked_fill(~context_mask[:, None, :], torch.finfo(scores.dtype).min)
attended = torch.einsum("bnl,blr->bnr", scores.softmax(-1), value)
logits = (query * attended).sum(-1) / math.sqrt(self.rank) # [B, N] — one scalar each
return logits.masked_fill(~option_mask, torch.finfo(logits.dtype).min)
```
Read the two `einsum` subscripts. Every contraction is `n` against `l` — option
against *context* token. There is no `n`-against-`n` term anywhere in the
function, so option 3 cannot see option 1, and `N` enters only as a tensor
dimension that arrives with the batch. Permute the options and you permute the
rows of `logits`; you do not change any of them. That is not a training result.
It is the type signature.
One more piece of evidence that these are one product with two wirings: a
serving stack that ships both behind a single route.
[SGLang](https://github.com/sgl-project/sglang)'s `/v1/score` takes
a `query`, a list of `items` and `label_token_ids`, and its docstring names the
fork outright — *"1. Single-Item scoring (default): Process each query+item pair
independently. 2. Multi-Item scoring: When `--enable-mis` is set, combine query
and multiple items into a single sequence using delimiter for efficient
processing."* One `if`, one `else`:
```python
# sglang · python/sglang/srt/managers/tokenizer_manager_score_mixin.py @ 3a64faa
if use_multi_item_scoring:
# Format: queryitem1item2item3
combined_input_ids, delimiter_indices = self._build_multi_item_token_sequence(
query_ids, items_ids, delimiter_token_id
)
input_ids = [combined_input_ids]
else:
# Single-item scoring: create separate prompts for each item
if item_first:
text_prompts = [f"{item}{query}" for item in items_list]
else:
text_prompts = [f"{query}{item}" for item in items_list]
```
Same endpoint, same `label_token_ids`, same weights. The `else` branch builds N
prompts and scores each option in a context that contains no other option —
option isolation, whatever the head on top happens to be, and `item_first` is
there because the only positional choice left is whether the option goes before
or after the query. The `if` branch concatenates every option into one sequence
and reads a logprob at each delimiter position — a shared context, in which an
option suddenly has neighbours and an index. Nothing about the model changed.
The options moved.
Both give you a probability over an option set the caller chose at request time.
They are not equivalent, and here is the difference that shows:
**Receipts.** Reversing the order of a question's options, changing nothing else, changes openjev's answer on 10 of 36 cases — 27.8%. This is the cost of the vocabulary-readout design, where options live in the prompt as lettered text and the model only ever picks a letter. It is also exactly the failure the per-option scalar scorer removes by construction, and the reason TypeSafe can say Jev 'doesn't see a level's number or its neighbours'.
| original chose | p | reversed chose | p | gold | |
| :--- | ---: | :--- | ---: | :--- | :--- |
| insufficient | 0.871 | contradicted | 0.987 | contradicted | reversal fixed it |
| insufficient | 0.506 | prohibited | 0.657 | prohibited | reversal fixed it |
| A | 0.904 | B | 0.500 | A | reversal broke it |
| A | 0.832 | B | 0.898 | B | reversal fixed it |
| A | 0.817 | B | 0.970 | B | reversal fixed it |
| prohibited | 0.877 | permitted | 0.621 | prohibited | reversal broke it |
36 pairs from 36 source cases, so these are not independent of the authored set they came from, and 10/36 carries a wide interval. The direction is not in doubt though: the flips are confident in both directions, not coin-flips near 0.5. Accuracy happens to be similar either way (80.6% reversed against 80.6% on irrelevant context) — order sensitivity shows up as churn between the two runs, not as a drop in the aggregate, which is exactly why an accuracy number alone would hide it.
> method: openjev commits both its benchmark fixtures and its raw predictions. benchmarks/data/perturbations108.jsonl carries provenance.variant and provenance.base_id, so every option_reversal row can be joined back to the authored144 row it was derived from. For each pair I realigned both predictions from their own option_ids into the row's option order, took the argmax, and compared the chosen option id. Model: a frozen Qwen3.5-4B read through direct option logits, no training.
> source: https://github.com/TheoLeeCJ/openjev
> captured: 2026-09-18
> data: https://ai.thesatyajit.com/articles/cua-s1-forms/data/order-sensitivity.json (6 rows)
openjev ships an `option_reversal` perturbation: the same evidence, the same
question, the same options, listed in the opposite order. Because it commits both
its fixtures and its raw predictions, and because every perturbed row carries the
`base_id` it came from, you can join each reversed case back to its original and
ask whether the model changed its mind.
**It changed its mind on 10 of 36 — 27.8%.** Not marginally: it went from
`insufficient` at p=0.871 to `contradicted` at p=0.987 on evidence that did not
move.
Note what this does *not* show up as. Accuracy on the reversed set is 80.6%,
identical to the irrelevant-context set and only 2.8 points off the unperturbed
144. Some flips land on the gold answer and some fall off it, so the aggregate
barely twitches while more than a quarter of the individual decisions are
unstable. An accuracy number would have hidden this completely; it is visible
only because openjev published per-row predictions and the provenance to pair
them up.
That instability is a property of the *design*, not of the model being small. In
a vocabulary readout, `A` and `B` are different tokens carrying different learned
priors, so moving an option's content from one letter to another moves it to a
different prior. There is no amount of scale that makes the letters
interchangeable, which is why position bias in multiple-choice evaluation has
been a known problem for years.
An independent benchmark has since put a number on the other side of this trade: Jev scores [0 of 100 on relational choice](/articles/jev-scores-zero), where one option's content decides another. Order-insensitivity and option-isolation are the same property, and that is what it costs.
The scalar scorer does not fix this. It never has the problem: there is no letter,
no position, nothing for a prior to attach to. Each option is encoded on its own
and scored on its own. Which is precisely what TypeSafe claims for Jev, in the
sentence the last article leaned on hardest — *"Every level is evaluated
separately. The model doesn't see a level's number or its neighbours"* — and it
is why the ordering has to be imposed by the caller afterwards, with `score`
computed as the expectation over **your** array indices, client-side.
So: *does the order of options matter?* In one family it is a 27.8% liability
that you have to train out with shuffling augmentation. In the other it is not
expressible. That is the single most useful thing to know about which family a
given model belongs to, and it takes one afternoon and a reversed list to find
out.
## Bespoke's Nimble: the data is the contribution
[`bespokelabs/Bespoke-Nimble-9B`](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B)
takes the other route — a LoRA adapter (rank 16, lr 5e-5, effective batch 8, one
epoch, BF16, seed 17) on Qwen3.5-9B, scoring the allowed answer tokens directly
rather than growing a head. Trained on an L40S. Apache 2.0.
The architecture is the boring part. The data recipe is the paper.
**Contrastive data curation** builds examples in pairs. Both halves share the
question, the policy and every sentence of evidence but one. That one sentence is
edited so that it flips the correct answer — and the pipeline verifies that with
the sentence removed, the fact it settles is *unknowable* from the rest of the
document.
So the model cannot get the pair right by learning what a question of this shape
usually resolves to. It has to read the sentence that moved. Bespoke's framing:
*"We push the model to be calibrated to be a better decision maker by creating
negative examples, which forces the model to become a better discriminator."*
Two consequences they call out, both real:
- **The training data needs no probabilities.** Labels come from applying the
decision rule, not from a teacher's confidence. Which means —
- **No distillation from Jev.** *"We did not distill from Jev"* — Jev is used to
evaluate, never to supervise. Saved Jev probabilities sit unused in the repo for
anyone who wants soft targets later. There is also no RL.
3,000 examples total across ten categories — commerce, education, home, media,
public services, science, software, supply chain, travel, workplace — split 2,676
train and 324 held out.
### The result that actually matters
A 9B model with a rank-16 adapter, trained for one epoch on 2,676 examples,
clears an untuned 27B by 17 examples. Against its own base it goes 215/324 to
292/324 — 77 examples, 23.8 points, from three thousand rows of well-constructed
data. That is the whole argument for contrastive curation and it is a strong one.
And then Jev, which has never seen this dataset, scores **302/324**, ten examples
ahead of the model tuned on its distribution.
That is the number a lab with an interest in the opposite result chose to publish
on its own benchmark, and it is the most credible thing in the release. Bespoke
flag it themselves: *"there is no standard benchmark to measure performance, and
it's possible Nimble is much worse on other benchmarks compared to Jev."*
### Two things in the latency table to read twice
Nimble is reported at **106.0 ms** median on an H100 against Jev's **246.7 ms**.
Both figures are honest and they are not comparable: one is local GPU compute,
the other is an HTTPS round trip to someone else's datacentre. Strip the network
and the comparison disappears.
More interesting, and unremarked in the release: the base Qwen3.5-9B is measured
at **58.1 ms** on the same hardware and the same 324 rows. Nimble is that model
plus an adapter, at nearly double the latency. That gap is the adapter being
applied at runtime rather than merged into the base weights — `merge_and_unload`
would give most of it back for free. Also worth noting the Nimble H100 row is
measured over **120** examples where every other row uses 324.
## jev-ultrafast: the boundary is the product
[`browser-use/jev-ultrafast`](https://github.com/browser-use/jev-ultrafast) is not
a model. It is the harness that makes a decision model useful, and it is the
clearest demonstration of why the Jev shape suits agents.
The loop is `page → indexed elements → operation + target → execution`. The
runtime reads the DOM, enumerates the controls that actually exist, and hands the
model a numbered list. The model returns an index. The runtime executes it.
No screenshots — *"the model does not consume them"*. No coordinates. And,
critically, no generated code: *"Never let the model emit selectors or executable
code"*, and *"Targets must map to observed elements and supported operations."*
That is a containment property, not an optimisation. A model that emits a CSS
selector can emit one that matches something you did not intend. A model that
returns `3` can only ever pick one of the things the runtime already validated and
is prepared to do. The action space is rebuilt every step from what is on the
page, so it is dynamic without ever being open-ended.
The operational rules around it are the sort written after something went wrong:
*"Never retry a browser mutation. Log execution before observing its result."*
A click that appears to fail may have gone through.
The demo figure going around — Zürich to London in 7.1 seconds for \$0.0039 — is a
single run of one task and should be read as a demo, not a benchmark. The
architecture is the interesting claim. There is already a macOS port,
[`max1874/jev-computer-use`](https://github.com/max1874/jev-computer-use), doing
the same thing to native apps, and a
[roundup](https://github.com/fatwang2/awesome-jev) of the rest.
## What none of them published, and what two of them did
Look back at the receipts table, last column.
The first version of this section said *"not one of these projects reports a
calibration number."* That was wrong when I published it, and it got more wrong
the next morning. Both corrections are worth more than the claim was.
### cua-s1-forms ships an ECE, in the checkpoint
Not in the README and not in the model card — in the artifact. `cua-s1-forms.json`,
the JSON sidecar that sits next to the safetensors on Hugging Face, carries a
`metadata.best_validation` block — the validation pass with the lowest NLL — and
inside it:
```json
{"ece": 0.00014795374386267213, "nll": 0.0019850827802381913,
"top1": 0.9994105100631714, "examples": 22054}
```
The code that produced it is in the repo. `libs/cua-s1/training/train.py` bins
the top-1 confidence into ten equal-width buckets and sums the size-weighted gap
between mean confidence and mean accuracy, which is textbook ECE, and it runs on
every validation pass. I missed it because I read the card and not the sidecar,
which is exactly the mistake the receipts table exists to prevent.
One loose thread while we are in there: that block counts **22,054** examples,
and the published `validation.jsonl` is *"~18k"* rows. Twenty-two percent apart
is not rounding, so either the shipped checkpoint was trained against a
differently-sized split than the one released, or `examples` counts something
other than dataset rows. Neither is a problem; it is just the second thing the
sidecar says that the cards do not.
Read it carefully, though, and it is thin for the same reason the headline
accuracy is thin. `top1` on that split is 0.99941. A model that is right 9,994
times in 10,000 and says so confidently cannot have a large ECE; near-zero
calibration error is what near-perfect in-domain accuracy *implies*, not an
independent finding about the probabilities. The band where a threshold actually
earns its keep — somewhere between 0.6 and 0.9, on data the generator did not
make — has almost no rows in it. The number is real, it is published, and it
does not settle anything.
### Bespoke published the comparison, one day after this article
On 2026-09-19 [the `nimble` repo](https://github.com/bespokelabsai/nimble) added
`docs/PUBLIC_BENCHMARKS.md`: thirteen human-labelled public datasets, 3,880
records, Nimble and Jev 1.13.0 both run on every one, with per-subset ECE,
Brier, NLL, expected-score MAE and divergence
against the human label distribution. Committed subset manifests and checksums,
so each subset rebuilds byte-for-byte. It is the first calibration comparison in
this category that is not a vendor measuring itself, and the commit message
describes the suite, accurately and without ceremony, as *"vibe-coded, offered
as-is"*.
The finding is not the one a lab shipping a competitor would have picked:
*"Jev has the lower expected calibration error on 11 of the 13 subsets, with
Nimble lower on `massive-en-US` and `summeval-relevance`, and the lower Brier
score on 10 of 13."* No temperature fitted for either model, so those describe
both as shipped. Nimble's ECE runs from 0.061 to 0.343 across the suite, Jev's
from 0.038 to 0.261. (Their own table says 11 of 13 on Brier, not 10 — Jev is
lower everywhere except `multinli` and `summeval-relevance`. The prose undercounts
its own numbers by one, in the direction that flatters neither model.) They also
note that Jev's NLL is not comparable at all, because the API rounds every
probability to two decimals and a true label arriving as `0.00` blows up a log —
which is a good catch and a real limit on what you can measure through that API.
That is the second time in this piece that Bespoke have published a number
favouring the model they are competing with. It is the main reason I believe the
rest of the release.
### What is actually still missing
Narrower than I first wrote, and more specific:
- **jev-ultrafast** reports no calibration and should not. It is a harness, not
a training project.
- **TypeSafe still publishes nothing.** No ECE, no Brier, no reliability curve
for Jev, anywhere. The only calibration figures that exist for the model whose
category is named after calibration were computed by a competitor, on datasets
the vendor did not choose, through an API that rounds the probabilities to two
decimals.
- **Nobody has tested Bespoke's own hypothesis.** *"The calibration is
implicit"* — the claim that contrastive pairs induce calibration without
optimising for it — is still, in their words, *"Didn't do ablations but I
think this is a critical piece!"*, and the training objective still uses hard
labels with no calibration term anywhere. The public suite measures Nimble
against Jev. The ablation that would settle the hypothesis measures Nimble
against its own untuned base, and that row does not exist in either release.
- **No reliability diagram anywhere.** ECE is one scalar summarising a binned
curve; the curve is what tells you whether a model is over- or under-confident
and where. Every one of these repos emits the probabilities needed to draw
one — cua-s1-forms emits a probability per option, Nimble scores the allowed
answer tokens directly — and the
[last article](/articles/rlcd-calibrated-decisions#the-calibration-reality-check)
drew exactly that for a fourth repo in an afternoon, from committed logits its
authors had never joined to their own labels.
This matters more here than it would elsewhere, because calibration is not a
nice-to-have in this category — it is the category's name. RLCD is
*reinforcement learning for calibrated decisions*. The entire argument for a
System One model over a prompted LLM is that the probability it returns means
something, so you can put a threshold on it and automate what sits above.
The open reproductions closed the gap on architecture and training inside a
week. On calibration they were a few days behind, and then one of them went past
the vendor entirely. The silence that is left is TypeSafe's.
Run CUA-S1 on a form-and-document set generated by someone else's pipeline — Bespoke's contrastive holdout would do — and report both models. If CUA-S1 holds near 99% off its own generator, the in-domain reading is wrong and it generalises far better than I am giving it credit for.
Run `merge_and_unload` on the LoRA, re-measure on the same 324 rows and the same H100. If the median stays near 106 ms rather than falling towards the base model's 58.1 ms, the overhead is somewhere else and my explanation is wrong.
This is Bespoke's hypothesis and their public suite does not test it: it measures Nimble against Jev, not Nimble against the model the adapter was trained on. Compute ECE and Brier for Nimble *and for the untuned Qwen3.5-9B* on the same rows — the 324 held-out contrastive examples, or any subset of the thirteen. If the adapter improves accuracy and leaves calibration flat or worse, "the calibration is implicit" is false as stated. If it improves both, it is the most interesting result in the release and deserves the ablation they skipped.
Load the checkpoint and print the final layer's shape. Anything other than a trailing 1 — a fixed K — means it is a classifier with a fixed answer set and I have described it wrongly.