A System One model in 706,048 parameters

The previous piece had to reconstruct Jev's architecture from two context limits and a sentence about scoring levels separately, because TypeSafe publishes fourteen words about it. A week later that is no longer the only way to find out. Three projects have rebuilt the shape in the open, with weights, data and training code, and between them they answer most of what the closed version leaves out.

They also, all three, leave out most of the same thing — and two of them leave out less of it than I first credited. More on that at the end.

The smallest one is the most informative

cua-ai/cua-s1-forms, from the trycua team, is 706,048 parameters. Not billions. Not millions. Seven hundred thousand, in a 2.8 MB checkpoint, trained from scratch — no base model, no pretrained encoder, no LoRA on top of something else.

Its own card describes the architecture plainly enough to redraw:

That is the per-option scalar scorer the last article argued Jev must be, built and shipped under MIT. Drawn:

The detail that matters is the head's output dimension. It is one. Not one per class — one, full stop, evaluated once per option. Which is why the training objective is "cross-entropy over the live option count": the softmax is taken over however many options the caller happened to send, and nothing in the weights depends on that number. Send two, send sixteen; the parameter count does not move.

This is also why the comparison to a classifier is the wrong one. A classifier has Linear[K, hidden] with K welded into the weights and cannot accept an option it was not trained on. This head has Linear[…, 1], so the option set is data. The idea is old — BertForMultipleChoice has worked this way since 2019 — and the interesting part is not the mechanism but that 706K parameters is enough of it.

How it was trained

The dataset is 234,067 rows — roughly 150k train, 18k validation, 20k test, plus 196 real decisions — and it is entirely synthetic. Each episode is assembled from three generators: a random form of 2–16 fields drawn from a 55-concept catalogue, a random synthetic person, and a random document built from that person's data.

The design decision worth stealing is in the third generator. Documents are padded with "distractor entities and forced look-alike confuser pairs" — email against street address, phone against emergency-contact phone — so that, in the dataset's own words, "the model has to read the whole label rather than pattern-match on a keyword." Window titles get random app suffixes, omitted about 20% of the time. The negative space is constructed, not sampled.

The options for each decision are document pointers (fill <entity>: <value>) plus three fixed actions — check, click, skip — and the label is a zero-based index into that list. Training is unremarkable on purpose: AdamW, cosine schedule with warmup, 6 epochs, batch size 128.

The split discipline is better than the training recipe. Splits are "disjoint by the exact sorted set of element descriptions in an episode", so a test form cannot appear in training under a different person's data. That is a real leak closed, and it is the reason the headline number is worth anything at all.

The number, and its denominator

99.95% on the form-disjoint synthetic test. 100% on 196 real decisions across three forms and three PDFs. And against the hosted Jev API on that same real eval: 99.7% versus 83.6%.

Read that last one carefully, because it is doing less work than it looks like. CUA-S1 was trained on 150,000 rows from the generator that also produced the evaluation; Jev has never seen any of it. That is a specialist measured at home against a generalist measured cold. The honest claim is a 706K-parameter specialist beats a general hosted model on the narrow task it was built for, which is genuinely the thesis of small specialist models and is worth demonstrating — but it is not evidence that Jev is worse at what Jev does.

To the team's credit, the source release says as much itself: "No checkpoint performance claim is established by this source-only release."

Two families, and why the option order is the tell

Everything above glosses over a split that matters more than any of it. There are two ways to make a model answer from a caller-supplied option list, they are not variations on a theme, and you can tell them apart from the outside by one experiment.

They are also, stripped of everything else, two short functions. This site has leaned on the distinction for three articles without once putting them next to each other, so here they are, as they are actually written.

Vocabulary readout. The options go in the prompt as lettered text, and the model only ever picks a letter. This is what openjev does, and its src/semif_phase1/direct.py is the clearest statement of it anywhere:

# openjev · src/semif_phase1/direct.py @ b9cb325 — device/timing setup and the `inputs` dict elided
def score(model, tokenizer, row: dict, metadata: dict, max_tokens: int = 4096) -> dict:
    ids, slots, prompt_hash = encode_prompt(tokenizer, row, max_tokens)
    with torch.inference_mode():
        vocabulary = _forward(model, inputs)[0].float()   # [151936] — the last position
    selected = vocabulary[slots].cpu().tolist()           # slots: the ids for "A", "B", "C"
    return {
        "probabilities": softmax(selected),
        "option_logits": selected,
        "readout": "native full-vocabulary last-position logits restricted to declared answer slots",
        "probability_status": "conditional option score; uncalibrated as decision confidence",
    }

One forward pass over one prompt, then a fancy-index into a 151,936-long vector. The option count never appears in that function; slots does, and slots is a list of token ids.

The prompt is JSON — {"evidence": …, "criterion": …, "options": [{"letter": "A", "description": …}]} — and the system message says "Respond with only its uppercase letter." The model's dynamic option set is just text it reads; the thing it actually chooses between is sixteen fixed vocabulary tokens. LETTERS = "ABCDEFGHIJKLMNOP", which is why the validator caps options at 16. There is even a guard, _slot_ids, that refuses to run unless each letter is "one exact round-trip token" — because if A tokenised as anything else you could not slice its row out.

Per-option scalar scorer. Each option's own text is encoded and scored alone, and the softmax is over the scalars. This is CUA-S1 above, and open-jev-deberta, and — per the last article's argument — Jev. The entire family is this forward:

# cua-s1 · libs/cua-s1/python/src/cua_s1/model.py @ 9bbfa7d — AttentionHead.forward
def forward(self, context, context_mask, options, option_mask, shuffle_context=False):
    context = self.context_norm(context.float())      # [B, L, width] — the context tokens
    options = self.option_norm(options.float())       # [B, N, width] — one vector per option
    query = self.query(options)
    key = self.key(context)
    value = self.value(context)
    scores = torch.einsum("bnr,blr->bnl", query, key) / math.sqrt(self.rank)
    scores = scores.masked_fill(~context_mask[:, None, :], torch.finfo(scores.dtype).min)
    attended = torch.einsum("bnl,blr->bnr", scores.softmax(-1), value)
    logits = (query * attended).sum(-1) / math.sqrt(self.rank)   # [B, N] — one scalar each
    return logits.masked_fill(~option_mask, torch.finfo(logits.dtype).min)

Read the two einsum subscripts. Every contraction is n against l — option against context token. There is no n-against-n term anywhere in the function, so option 3 cannot see option 1, and N enters only as a tensor dimension that arrives with the batch. Permute the options and you permute the rows of logits; you do not change any of them. That is not a training result. It is the type signature.

One more piece of evidence that these are one product with two wirings: a serving stack that ships both behind a single route. SGLang's /v1/score takes a query, a list of items and label_token_ids, and its docstring names the fork outright — "1. Single-Item scoring (default): Process each query+item pair independently. 2. Multi-Item scoring: When --enable-mis is set, combine query and multiple items into a single sequence using delimiter for efficient processing." One if, one else:

# sglang · python/sglang/srt/managers/tokenizer_manager_score_mixin.py @ 3a64faa
if use_multi_item_scoring:
    # Format: query<delimiter>item1<delimiter>item2<delimiter>item3<delimiter>
    combined_input_ids, delimiter_indices = self._build_multi_item_token_sequence(
        query_ids, items_ids, delimiter_token_id
    )
    input_ids = [combined_input_ids]
else:
    # Single-item scoring: create separate prompts for each item
    if item_first:
        text_prompts = [f"{item}{query}" for item in items_list]
    else:
        text_prompts = [f"{query}{item}" for item in items_list]

Same endpoint, same label_token_ids, same weights. The else branch builds N prompts and scores each option in a context that contains no other option — option isolation, whatever the head on top happens to be, and item_first is there because the only positional choice left is whether the option goes before or after the query. The if branch concatenates every option into one sequence and reads a logprob at each delimiter position — a shared context, in which an option suddenly has neighbours and an index. Nothing about the model changed. The options moved.

Both give you a probability over an option set the caller chose at request time. They are not equivalent, and here is the difference that shows:

openjev ships an option_reversal perturbation: the same evidence, the same question, the same options, listed in the opposite order. Because it commits both its fixtures and its raw predictions, and because every perturbed row carries the base_id it came from, you can join each reversed case back to its original and ask whether the model changed its mind.

It changed its mind on 10 of 36 — 27.8%. Not marginally: it went from insufficient at p=0.871 to contradicted at p=0.987 on evidence that did not move.

Note what this does not show up as. Accuracy on the reversed set is 80.6%, identical to the irrelevant-context set and only 2.8 points off the unperturbed 144. Some flips land on the gold answer and some fall off it, so the aggregate barely twitches while more than a quarter of the individual decisions are unstable. An accuracy number would have hidden this completely; it is visible only because openjev published per-row predictions and the provenance to pair them up.

That instability is a property of the design, not of the model being small. In a vocabulary readout, A and B are different tokens carrying different learned priors, so moving an option's content from one letter to another moves it to a different prior. There is no amount of scale that makes the letters interchangeable, which is why position bias in multiple-choice evaluation has been a known problem for years.

An independent benchmark has since put a number on the other side of this trade: Jev scores 0 of 100 on relational choice, where one option's content decides another. Order-insensitivity and option-isolation are the same property, and that is what it costs.

The scalar scorer does not fix this. It never has the problem: there is no letter, no position, nothing for a prior to attach to. Each option is encoded on its own and scored on its own. Which is precisely what TypeSafe claims for Jev, in the sentence the last article leaned on hardest — "Every level is evaluated separately. The model doesn't see a level's number or its neighbours" — and it is why the ordering has to be imposed by the caller afterwards, with score computed as the expectation over your array indices, client-side.

So: does the order of options matter? In one family it is a 27.8% liability that you have to train out with shuffling augmentation. In the other it is not expressible. That is the single most useful thing to know about which family a given model belongs to, and it takes one afternoon and a reversed list to find out.

Bespoke's Nimble: the data is the contribution

bespokelabs/Bespoke-Nimble-9B takes the other route — a LoRA adapter (rank 16, lr 5e-5, effective batch 8, one epoch, BF16, seed 17) on Qwen3.5-9B, scoring the allowed answer tokens directly rather than growing a head. Trained on an L40S. Apache 2.0.

The architecture is the boring part. The data recipe is the paper.

expand

Contrastive data curation builds examples in pairs. Both halves share the question, the policy and every sentence of evidence but one. That one sentence is edited so that it flips the correct answer — and the pipeline verifies that with the sentence removed, the fact it settles is unknowable from the rest of the document.

So the model cannot get the pair right by learning what a question of this shape usually resolves to. It has to read the sentence that moved. Bespoke's framing: "We push the model to be calibrated to be a better decision maker by creating negative examples, which forces the model to become a better discriminator."

Two consequences they call out, both real:

• The training data needs no probabilities. Labels come from applying the decision rule, not from a teacher's confidence. Which means —
• No distillation from Jev. "We did not distill from Jev" — Jev is used to evaluate, never to supervise. Saved Jev probabilities sit unused in the repo for anyone who wants soft targets later. There is also no RL.

3,000 examples total across ten categories — commerce, education, home, media, public services, science, software, supply chain, travel, workplace — split 2,676 train and 324 held out.

The result that actually matters

A 9B model with a rank-16 adapter, trained for one epoch on 2,676 examples, clears an untuned 27B by 17 examples. Against its own base it goes 215/324 to 292/324 — 77 examples, 23.8 points, from three thousand rows of well-constructed data. That is the whole argument for contrastive curation and it is a strong one.

And then Jev, which has never seen this dataset, scores 302/324, ten examples ahead of the model tuned on its distribution.

expand

That is the number a lab with an interest in the opposite result chose to publish on its own benchmark, and it is the most credible thing in the release. Bespoke flag it themselves: "there is no standard benchmark to measure performance, and it's possible Nimble is much worse on other benchmarks compared to Jev."

Two things in the latency table to read twice

Nimble is reported at 106.0 ms median on an H100 against Jev's 246.7 ms. Both figures are honest and they are not comparable: one is local GPU compute, the other is an HTTPS round trip to someone else's datacentre. Strip the network and the comparison disappears.

More interesting, and unremarked in the release: the base Qwen3.5-9B is measured at 58.1 ms on the same hardware and the same 324 rows. Nimble is that model plus an adapter, at nearly double the latency. That gap is the adapter being applied at runtime rather than merged into the base weights — merge_and_unload would give most of it back for free. Also worth noting the Nimble H100 row is measured over 120 examples where every other row uses 324.

jev-ultrafast: the boundary is the product

browser-use/jev-ultrafast is not a model. It is the harness that makes a decision model useful, and it is the clearest demonstration of why the Jev shape suits agents.

The loop is page → indexed elements → operation + target → execution. The runtime reads the DOM, enumerates the controls that actually exist, and hands the model a numbered list. The model returns an index. The runtime executes it.

No screenshots — "the model does not consume them". No coordinates. And, critically, no generated code: "Never let the model emit selectors or executable code", and "Targets must map to observed elements and supported operations."

expand

That is a containment property, not an optimisation. A model that emits a CSS selector can emit one that matches something you did not intend. A model that returns 3 can only ever pick one of the things the runtime already validated and is prepared to do. The action space is rebuilt every step from what is on the page, so it is dynamic without ever being open-ended.

The operational rules around it are the sort written after something went wrong: "Never retry a browser mutation. Log execution before observing its result." A click that appears to fail may have gone through.

The demo figure going around — Zürich to London in 7.1 seconds for $0.0039 — is a single run of one task and should be read as a demo, not a benchmark. The architecture is the interesting claim. There is already a macOS port, max1874/jev-computer-use, doing the same thing to native apps, and a roundup of the rest.

What none of them published, and what two of them did

Look back at the receipts table, last column.

The first version of this section said "not one of these projects reports a calibration number." That was wrong when I published it, and it got more wrong the next morning. Both corrections are worth more than the claim was.

cua-s1-forms ships an ECE, in the checkpoint

Not in the README and not in the model card — in the artifact. cua-s1-forms.json, the JSON sidecar that sits next to the safetensors on Hugging Face, carries a metadata.best_validation block — the validation pass with the lowest NLL — and inside it:

{"ece": 0.00014795374386267213, "nll": 0.0019850827802381913,
 "top1": 0.9994105100631714, "examples": 22054}

The code that produced it is in the repo. libs/cua-s1/training/train.py bins the top-1 confidence into ten equal-width buckets and sums the size-weighted gap between mean confidence and mean accuracy, which is textbook ECE, and it runs on every validation pass. I missed it because I read the card and not the sidecar, which is exactly the mistake the receipts table exists to prevent.

One loose thread while we are in there: that block counts 22,054 examples, and the published validation.jsonl is "~18k" rows. Twenty-two percent apart is not rounding, so either the shipped checkpoint was trained against a differently-sized split than the one released, or examples counts something other than dataset rows. Neither is a problem; it is just the second thing the sidecar says that the cards do not.

Read it carefully, though, and it is thin for the same reason the headline accuracy is thin. top1 on that split is 0.99941. A model that is right 9,994 times in 10,000 and says so confidently cannot have a large ECE; near-zero calibration error is what near-perfect in-domain accuracy implies, not an independent finding about the probabilities. The band where a threshold actually earns its keep — somewhere between 0.6 and 0.9, on data the generator did not make — has almost no rows in it. The number is real, it is published, and it does not settle anything.

Bespoke published the comparison, one day after this article

On 2026-09-19 the nimble repo added docs/PUBLIC_BENCHMARKS.md: thirteen human-labelled public datasets, 3,880 records, Nimble and Jev 1.13.0 both run on every one, with per-subset ECE, Brier, NLL, expected-score MAE and divergence against the human label distribution. Committed subset manifests and checksums, so each subset rebuilds byte-for-byte. It is the first calibration comparison in this category that is not a vendor measuring itself, and the commit message describes the suite, accurately and without ceremony, as "vibe-coded, offered as-is".

The finding is not the one a lab shipping a competitor would have picked: "Jev has the lower expected calibration error on 11 of the 13 subsets, with Nimble lower on massive-en-US and summeval-relevance, and the lower Brier score on 10 of 13." No temperature fitted for either model, so those describe both as shipped. Nimble's ECE runs from 0.061 to 0.343 across the suite, Jev's from 0.038 to 0.261. (Their own table says 11 of 13 on Brier, not 10 — Jev is lower everywhere except multinli and summeval-relevance. The prose undercounts its own numbers by one, in the direction that flatters neither model.) They also note that Jev's NLL is not comparable at all, because the API rounds every probability to two decimals and a true label arriving as 0.00 blows up a log — which is a good catch and a real limit on what you can measure through that API.

That is the second time in this piece that Bespoke have published a number favouring the model they are competing with. It is the main reason I believe the rest of the release.

What is actually still missing

Narrower than I first wrote, and more specific:

• jev-ultrafast reports no calibration and should not. It is a harness, not a training project.
• TypeSafe still publishes nothing. No ECE, no Brier, no reliability curve for Jev, anywhere. The only calibration figures that exist for the model whose category is named after calibration were computed by a competitor, on datasets the vendor did not choose, through an API that rounds the probabilities to two decimals.
• Nobody has tested Bespoke's own hypothesis. "The calibration is implicit" — the claim that contrastive pairs induce calibration without optimising for it — is still, in their words, "Didn't do ablations but I think this is a critical piece!", and the training objective still uses hard labels with no calibration term anywhere. The public suite measures Nimble against Jev. The ablation that would settle the hypothesis measures Nimble against its own untuned base, and that row does not exist in either release.
• No reliability diagram anywhere. ECE is one scalar summarising a binned curve; the curve is what tells you whether a model is over- or under-confident and where. Every one of these repos emits the probabilities needed to draw one — cua-s1-forms emits a probability per option, Nimble scores the allowed answer tokens directly — and the last article drew exactly that for a fourth repo in an afternoon, from committed logits its authors had never joined to their own labels.

This matters more here than it would elsewhere, because calibration is not a nice-to-have in this category — it is the category's name. RLCD is reinforcement learning for calibrated decisions. The entire argument for a System One model over a prompted LLM is that the probability it returns means something, so you can put a threshold on it and automate what sits above.

The open reproductions closed the gap on architecture and training inside a week. On calibration they were a few days behind, and then one of them went past the vendor entirely. The silence that is left is TypeSafe's.