Any model can be Jev, for the price of a serving flag

The claim going around this week is short enough to quote whole: SGLang exposes a /v1/score endpoint alongside /generate; you hand it an input and a set of candidate answers and it hands back a probability per candidate. The worked example is "what is most common letter in abcccde?" against a, b, c, returning (c, 0.9) (b, 0.05) (a, 0.05). Somebody built deepseek-v4.1-flash-jev that way, with one extra trick, and their own verdict on it is "dsv4.1 flash is not as good as jev".

Everything written here so far has been an argument about what Jev is. RLCD is not constrained decoding reasoned, from two sentences in TypeSafe's docs, that Jev must be a per-option scalar scorer. A System One model in 706,048 parameters split the open reproductions into two families — vocabulary readout and per-option scorer — and showed you can tell them apart by reversing an option list. Jev scores zero found the behavioural signature the scorer reading predicts.

This one points the other way. The readout is a serving feature, not a model feature. There is nothing in a checkpoint that makes it a decision model. There is a way of calling it.

Before the handler, the shape — because the question I keep being asked about these models is not what are they for but what actually goes in, and what actually comes out.

What /v1/score actually computes

This is the crux, so I read the handler rather than the docs. SGLang at 3a64faa, python/sglang/srt/managers/tokenizer_manager_score_mixin.py. The request it builds for a generative model has four arguments that do all the work:

# tokenizer_manager_score_mixin.py — score_request(), the generation branch
batch_request = GenerateReqInput(
    text=text_prompts,
    input_ids=input_ids,
    token_ids_logprob=label_token_ids,
    return_logprob=True,
    logprob_start_len=0 if use_multi_item_scoring else -1,
    stream=False,
    sampling_params={"max_new_tokens": 0},
    ...
)

max_new_tokens: 0. Nothing is generated. The model does one prefill and stops, and the reply carries the logprobs of the token ids the caller named. Then:

# the same file — what "score" means, in full
def _convert_logprobs_to_scores(self, logprobs, label_token_ids, apply_softmax):
    score_list = [logprobs.get(token_id, float("-inf")) for token_id in label_token_ids]
    if apply_softmax:
        score_list = torch.softmax(torch.tensor(score_list), dim=0).tolist()
    else:
        score_list = [math.exp(x) if x != float("-inf") else 0.0 for x in score_list]
    return score_list

So, to answer the question directly, because it decides everything downstream: this is not a full-sequence logprob and it is not length-normalised. It is a single-position vocabulary readout. The code's own comment on the extraction says "Extract logprobs for the first (and only) position." The candidate text, when you pass candidates as items, is conditioned on — it goes into the prompt — and is never itself scored. Only the next-token distribution at the end of the prompt is read, and only at the rows you asked for.

That matters more than it sounds. A naive "score each candidate by the log-probability of its text" is length-biased: longer answers score lower for being longer. This design does not have that problem, because it never scores the candidate's tokens at all. What it has instead is a constraint I will come back to: a candidate has to be a token id.

Two smaller things in those seven lines, both of which will bite somebody.

apply_softmax defaults to False, and the false branch returns exp(logprob) — the model's actual probability mass on that token, out of the whole vocabulary. Those do not sum to one, and they should not. Set it to True and you get a softmax over the slice, which renormalises the candidates against each other. That is the usual thing to want, and it quietly throws away the one signal a decision model most needs: the mass sitting on every token that is not one of your options. That mass is the model declining. Renormalisation deletes the abstention and hands you back a confident-looking distribution over a set the model may think is wrong.

And logprobs.get(token_id, float("-inf")) falling through to 0.0 means a candidate the sampler never reported scores exactly zero rather than raising. In the normal path token_ids_logprob guarantees those rows come back, so it is a defensive branch; it is also a silent one.

It is fifteen months older than the category

Worth saying plainly. Score serving landed in SGLang on 2025-06-21, in PR #7399, titled "[OAI refactor] Add rerank and score serving". It sits next to /v1/rerank in the docs, under the heading "decoder-only scoring", and the example in SGLang's own docstring is a relevance judgement:

query = "<|user|>Is the following city the capital of France? "
items = ["Paris <|assistant|>", "London <|assistant|>"]
label_token_ids = [2332, 1223]  # "Yes" / "No"
# -> [[0.9, 0.1], [0.2, 0.8]]

Nobody built this for decision models. It was built for rerankers and reward models, over a year before Jev launched, and it has been sitting in every SGLang install since. Reasoned: that is most of why the trick is "shockingly easy" — not because anyone made it easy, but because the capability was already in the box under a name nobody reads as decision model.

The same endpoint is two different model families

Here is the part I did not expect. /v1/score does not commit you to a family. It will serve either one, depending only on how you map your problem onto query, items and label_token_ids, and the wire format looks the same both ways.

Put the options in the prompt as lettered text and make label_token_ids the letters, and you have a vocabulary readout: one forward pass, all options sharing one context, three rows sliced out of one logit vector. That is precisely openjev's direct.py, and the cua-s1 piece measured what it costs — reversing the option order changed openjev's answer on 10 of 36 cases, 27.8%, because A and B are different tokens carrying different learned priors.

Put the options in items and make label_token_ids a fixed Yes/No pair, and single-item mode builds one prompt per item — text_prompts = [f"{query}{item}" for item in items_list] — and runs a separate forward pass for each. Now no option can see any other option. That is the isolation property the whole scorer argument turned on, and you get it from a serving choice rather than from an architecture.

It is not the same thing as Jev, and the difference is worth being precise about. The scalar in a per-option scorer is a learned number: open-jev-deberta's head is a 3072 → 1024 → 1 MLP trained on the decision. The scalar here is p(Yes | query + option) from an untouched language model. Same shape, same isolation, nothing trained. Reasoned: mode C gets you Jev's structural property and none of its training, which is a decent first guess at why deepseek-v4.1-flash-jev's author says it is not as good.

The constraint that shows up everywhere

label_token_ids are token ids. Two candidates that share a first token are the same row. There is no way to ask this endpoint about "positive" versus "positively".

Everyone who builds this hits it and everyone solves it the same way. openjev ships _slot_ids, a guard that refuses to run unless each letter is "one exact round-trip token" — which is why its options are LETTERS = "ABCDEFGHIJKLMNOP" and cap at sixteen. The vLLM DiffusionGemma PR, a completely different architecture, states it as a rule: "Each label must be a single token in the answer template, which the server checks with the tokenizer." A discrete diffusion model denoising a canvas and an autoregressive decoder reading one position have almost nothing in common, and they arrive at the identical restriction, because it is a property of the readout and not of the model.

The closing think tag, and what it costs

The reported extra trick: for DeepSeek you must append a closing think tag before the response to force a direct answer instead of a reasoning trace. This is true, it is necessary, and it is not a hack — it is the model's own documented switch.

SGLang's python/sglang/srt/entrypoints/openai/encoding_dsv4.py builds the DeepSeek-V4 prompt, and after the user turn it appends the assistant marker and then one of two tokens:

# encoding_dsv4.py — thinking_start_token is "<think>", thinking_end_token is "</think>"
prompt += ASSISTANT_SP_TOKEN
if not drop_thinking and thinking_mode == "thinking":
    prompt += thinking_start_token
elif drop_thinking and thinking_mode == "thinking" and index >= last_user_idx:
    prompt += thinking_start_token
else:
    prompt += thinking_end_token

And the system prompt the same file assembles says it outright: "If thinking_mode is enabled (triggered by <think>), you MUST output your complete reasoning inside <think>...</think> BEFORE any tool calls or final response. Otherwise, output directly after </think> with tool calls or final response."

So the reason the trick is needed is structural. /v1/score takes a raw query string, not a message list, so it does not run the chat template — you are assembling the prompt yourself, and if you stop at the assistant marker, the position you are scoring is the first token of a reasoning trace. Appending </think> moves the scored position to the first token of the answer. Going through /v1/chat/completions with thinking off would do this for you; scoring does not, because scoring is a lower-level door.

What it costs is the interesting part, and I have not seen it stated. You are reading a reasoning model's answer distribution at the position where its reasoning has been switched off. For DeepSeek-V4.1-Flash — 40 layers, 384 routed experts, 6 active, 129,280 vocabulary — a large amount of post-training went into making the model good after a trace, not instead of one. Scoring the post-</think> position asks a System Two model for its System One answer.

That is exactly what you want from a decision model, and it is also the distribution the model was trained to route around. "dsv4.1 flash is not as good as jev" is, I think, mostly this. Reasoned, and there is a falsifier at the bottom.

Is this "any model"? Only on SGLang

vLLM has an endpoint at the same path with the same name, and it will not do this. enable_scoring_api in vllm/entrypoints/pooling/utils.py returns True only for embed and token_embed models, or for classify models with num_labels == 1. Point it at a generative checkpoint and serving_scores is None and the route raises "The model does not support Rerank (Score) API". There is no logprob path in vllm/entrypoints/pooling/scoring/ at all — I grepped the directory for logprob and label_token and got nothing. You can rebuild the readout client-side, because /v1/completions takes allowed_token_ids and prompt_logprobs, but that is you writing it.

llama.cpp does not have the endpoint. Its /reranking needs a reranker model, and the nearest primitive on /completion is n_probs, which returns "the probabilities of top N tokens". Top-N is the wrong shape: a decision model needs the probability of the candidates you asked about, and a candidate that is not in the top N simply is not in the reply.

So "any open model becomes a decision model with a serving flag" is a true statement about SGLang and a false statement about the other two big servers. That is a narrower claim than the one being made, and it is still a striking one.

The purest case: nothing trained, nothing changed

The author's summary is "No training (for now) just parallel decisions", and the repository is built to let you check it. BASE_MODEL_MANIFEST.json carries the upstream revision and a SHA-256 per file; RELEASE_MANIFEST.json records "weights_modified": false and "training_performed": false and digests everything else too. The RLCD piece noted that this repo is tagged inference-only — that the name says RLCD while the repo trains nothing. The author has now said so in their own words, which is the better version of that observation.

The engine is 122 lines. The scoring is six of them, and the comment is the part to read:

# rlcd/engine.py — constrained(), after one prefill shared across all branches
out = self.model(ids, past_key_values=fork_cache(cache, len(branches)),
                 attention_mask=mask, use_cache=True)
# Full likelihood, not first-token proxy. No length normalization.
for row, (_, _, start, value) in enumerate(metadata):
    logp = out.logits[row, start - 1:start + len(value) - 1].float().log_softmax(-1)
    score = logp.gather(1, self.tensor(value)[:, None]).sum()

Note that this is the other readout. SGLang reads one position and never scores the candidate's own text; this sums log-probabilities across every token of the candidate. It buys the thing SGLang cannot do — candidates that share a prefix are distinguishable — and it buys back the length bias, which the README flags: "Scores depend on candidate wording, tokenization, length, and a newline terminator. They are not calibrated confidence."

I checked that caveat against the repo's own committed scores, because it is checkable. Across 237 unique (run, case, field) groups in results/*.json, within each group, the correlation between a candidate's length in characters and its log-likelihood is r = −0.004 over 489 candidates, a slope of −0.013 log-prob per character. There is a trap here worth flagging: pool the candidates across groups without centring and you get r = −0.572, which looks like a strong length effect and is entirely an artefact of different fields having different score scales. Measured: within a decision, on this model and this corpus, length bias is not detectable. The author's caution is theoretically right and empirically inert here, and I would rather report the check than repeat the warning.

Where the 63x comes from

The headline is "63x faster on an L40S. 8x on MPS." The committed number is 62.91x, and both figures are real. Neither is a property of the method.

The baseline is model.generate() emitting a whole JSON object one token at a time. The scored path does two forward calls: one prefill, then one batched pass over every candidate branch at once. So the ratio is how many tokens the baseline had to type, and the cases differ in almost nothing else. Fit the five L40S rows and the autoregressive line is 95.4 ms + 15.21 ms × generated_tokens, R² = 0.983. The 62.91x case generated 226 tokens. The 1.09x case generated 13.

The hardware collapse tells you the same thing from the other side, and not in the way I expected. A decode step costs 15.21 ms on an L40S, 11.40 ms on an H100 and 13.74 ms on an M2 Max — essentially the same everywhere. A 350M model at fp16 in eager PyTorch, one token at a time, is bound by per-step overhead, not by the machine; the release says as much ("Optimized causal-conv1d, torch.compile, FlashAttention and MLX are not enabled"). What changes across machines is the batched cost: 0.0392 ms per branch token on the L40S against 0.5732 on the M2 Max, 14.6x worse. The GPU's advantage appears only in the path that has parallel work to give it. So "63x on an L40S, 8x on a Mac" is not the baseline being faster on the Mac. It is the scored path being slower there.

To the author's credit, essentially all of this is already in their README: they publish all five cases, they report the 12-case suite at 6.25x–9.68x, and they volunteer that with 255 candidates on a Mac the constrained path is 3.23x slower. They also report that neither method got the 28-field object fully right, and that field accuracy is 64.3% against the baseline's 53.6%. A release that hands you the row that undercuts its own headline is doing the thing the rest of this batch mostly does not.

DiffusionGemma, and a PR that has not landed

The other System One claim this week is vLLM PR #57250, "[Core] structured generation mode for DiffusionGemma model (Jev-like)". Say the status first, because nobody else does: it is open and unmerged. Fourteen commits, a ready label added on 19 September, and a review comment noting that three prerequisite PRs are stacked inside the branch and open standalone, which "guarantees a mess". The model support — vllm/model_executor/models/diffusion_gemma.py — is on main. The decision server, examples/features/diffusion_reads/, is not.

The parameter claim checks out exactly. I pulled the safetensors headers for google/diffusiongemma-26B-A4B-it with range requests across all eleven shards and summed the shapes:

25.25B and 3.84B, so "25.2B total / 3.8B active" is right and "26B-A4B" is the usual rounding. The routed experts are 22.84B of the 25.25B, which is why a 25B-parameter model runs on a DGX Spark at all.

The throughput numbers need their denominators read out, and the PR gives them: "1-way 8.7 req/s at 0.12 s, 32-way 54.0 req/s at 0.58 s, about 162 decisions/s" on one DGX Spark, three decisions per request. The arithmetic is exact — 54.0 × 3 = 162 — and these are the author's own measurements, not anyone's independent benchmark. Three things the number does not say on its own:

• The model served is nvidia/diffusiongemma-26B-A4B-it-NVFP4, a 4-bit quantised checkpoint, not the bf16 one whose parameters I just counted.
• It runs at --canvas 32. The model's own config.json has canvas_length: 256. The separate claim of roughly 85 questions per canvas is 256 ÷ 3, and it needs the compressed answer template that the PR says kicks in "past ten questions". The throughput run is at a canvas eight times smaller and three questions.
• It is read-only single reads, reads=1. The accuracy demos in the same PR run at reads=4 — four noise draws averaged to get the ± figures — and take 420–530 ms. The 120 ms and the error bars are different configurations.

None of that makes 162 decisions/s wrong. It makes it a number about one quantisation, one canvas size and one sampling policy, reported by the person proposing the change.

Worth noting what is not in the PR, because it is the strongest thing going for the approach: on JevBench's public hard tier, djev — Maisa's diffusion-gemma build, the same model family — scores 67.6% against Jev's 72.1%, the closest any open entrant gets. The bet that a canvas denoiser makes a good decision model looks like a reasonable one. The PR's numbers are about how fast you can serve it, and those are the two separate questions this article keeps having to pull apart.

jeff, and "only a mild hit in accuracy"

logan-markewich/jeff is a self-hosted System One server on a frozen GliFormer encoder, MIT, speaking TypeSafe's SDK. The claim made for it is that Jev "can basically be boiled down to a classifier or an encoder, just like GliFormer. Latency is the same, cost is cheaper, with only a mild hit in accuracy."

Two of those three hold. jeff's own measurements put it at 266 ms p50 on an L4 against Jev's 233 ms, and $0.0173 per 1,000 decisions against $0.0384 — same order on latency, less than half the cost. The third does not.

37.8% against 72.1% is not a mild hit. jeff scores 0 of 7 on ambiguous items and 1 of 19 on long policies with 3–4k-token states. On the standard tier it is 76.4% against 98.6%. The encoder reading is right about the mechanism — a frozen DeBERTa-family encoder with a scoring head really does reproduce the shape — and wrong about what the mechanism is worth on anything that needs inference rather than lexical cues.

Which is, again, the article's point from a different angle: the shape is cheap. The competence is not.

Credit where it is uncomfortable. Every number above is jeff's own, measured by jeff's author, against jeff's author's interest, using JevBench's own runner and scoring code. The repository also reports jeff's temperature-3.2 calibration, notes it is "score renormalization, not a calibrated posterior" in the source comment, and runs a probe confirming that Jev is "unchanged on every metric" when three unrelated questions are packed ahead of the labelled one — an independent confirmation of the question-isolation claim the earlier pieces argued for from the docs.

Compatible on the wire, incompatible in the field that matters

Every project here advertises SDK compatibility. Point TYPESAFE_BASE_URL at it and the official client works. That is true, and it is a real achievement, and it conceals something.

Three implementations, three different definitions of confidence, same field name, same contract. On a near-uniform (0.40, 0.35, 0.25), one of them reads 0.016 and another reads 0.575. A confidence-gated routing threshold at 0.5 ships that decision unattended under one and escalates it to a person under the other two. TypeSafe declines to publish the formula — "a specialized topic that we'll keep to a separate cookbook" — so nobody is wrong, exactly. Everybody guessed, and they guessed differently, and the wire format cannot tell you which one you are talking to.

What none of this reproduces

The serving half is done. Six projects, six ways to get a typed request in and a probability per option out in one pass, and between them they cover every family: single-position readout, full-sequence likelihood, canvas slots, independent sigmoids, learned scalar heads. If a /v1/score endpoint turns any model into a decision model, then "System One model" is not a kind of checkpoint. It is a serving discipline plus a training objective, and the serving discipline is now thoroughly, publicly, redundantly reproduced.

The training objective is the half the category is named after — RLCD is reinforcement learning for calibrated decisions — and it is where the projects separate. Look at the last column. The three that are pure serving tricks publish no calibration number, and two of the three say so in their own words. The three that fit or train something publish one.

That is not a coincidence and it is not really a criticism. Calibration is the one property you cannot get from the serving layer, because measuring it costs something the serving layer never touches: labelled outcomes. exp(logprob) from an untouched checkpoint is whatever pretraining left there, and Kadavath et al. showed that is surprisingly good on multiple choice and that preference optimisation then wrecks it. So a serving trick inherits its base model's calibration, whatever that happens to be, and not one of these projects went and found out.

Which is the smaller, more annoying version of the finding. Nobody needed a training run to publish a reliability number. A few hundred labelled decisions and an afternoon would have done it, for any of them, and it is the one number that would tell a reader whether the probability on the wire means anything.

The exception is the one that is the most useful thing in the batch:

expand

kev is a LoRA adapter plus a pointer head on Qwen backbones from 0.5B to 8B, Apache-2.0, trainable on a laptop — kev-0.5b is 9.3M trained parameters on a frozen Qwen2.5-0.5B, about an hour and three quarters on an Apple M5. Its model card carries ECE 0.065, and 0.031 after fitting a temperature of 1.47 on even-indexed records and testing on odd. It also carries this, which I would put on the wall:

And it measured the thing the cua-s1 piece's falsifier asked somebody to measure:

expand

0.00% on 36 permutations. The cua-s1 article measured openjev's vocabulary readout flipping on 27.8% of reversals and argued that a per-option scorer cannot express the failure. Here is the other half of that experiment, on the hosted model, from a third party. It is 36 items and it is one run, so it is a weak number in the statistical sense and a strong one in the structural sense: a vocabulary readout does not produce 0.00% by luck.

One correction to my own earlier piece while I am here. The cua-s1 article ended on "not one of these projects reports a calibration number." That was true of the three projects in front of me a day ago. It is not true of this week's: jeff publishes ECE and Brier on two tiers and measures Jev's alongside, and kev publishes ECE before and after temperature scaling plus an out-of-domain Brier. Both are worse than the hosted model they measured against, and both shipped the comparison. The pattern I described was real and it has already partly broken, which is the better outcome.

So the through-line, stated as narrowly as I can make it. The readout is a serving feature; SGLang has had it since June 2025; it works on essentially any generative checkpoint, one endpoint, one flag, and it gives you either model family depending on where you put the options. What it does not give you is the thing the name RLCD points at. Everyone has reproduced the serving half. Whether a project publishes a calibration number is now the fastest way to tell whether it did anything else.