Order is not noise

A complaint about Jev has been going around, and it is worth quoting whole because the shape of it matters more than the grievance:

"JEV actually doesn't even run deterministically. Exact same prompts give you different probabilities when you run it multiple times. The ORDER of the choices DRASTICALLY changes the output probs. I am more and more confused by their 'no-hallucination' claim."

That is three claims, not one. They are not equally supported, they are not supported by the same evidence, and the experiment that settles one of them tells you nothing about the others. Bundling them is what turns a set of findings into a verdict.

Two of the three turn out to be true. One is a misreading — of a claim that is narrower and duller than the word "hallucination" suggests. And the separation matters more than either result, because a benchmark that reorders options while also re-sending requests reports one number containing both effects, which is exactly what the single published figure for this is.

Everything below is recomputed from published artifacts. I have no TypeSafe API key. I did not need one, which is the part I did not expect when I started.

Claim 1 · the order changes the output

This one this site has already measured, on an open model, and the number is not small.

openjev ships an option_reversal perturbation — the same evidence, the same question, the same options in the opposite order — and commits both its fixtures and its raw per-row predictions with the base_id each perturbed row came from. Joining those back gives 36 pairs. I re-ran the join at commit ca3ba65:

• The answer changes on 10 of 36 — 27.8%.
• The probability vector moves by a median of 0.156 and a maximum of 0.870, in openjev's own metric (the largest movement of any single option).
• The mechanism is a position prior. All 36 questions have three options, so reversal swaps slots A and C and leaves B exactly where it was; using B as a drift control, slot A is worth +1.71 logits on average, +1.56 at the median, positive on 32 of the 36 questions. Moving an option's text from first to last costs 1.54 logits; moving it from last to first gains 1.88. That is about five and a half times the odds, for nothing written in the option.

One methodological detail I checked this time and had not before, because it is the thing that would invalidate the whole measurement: for all 36 pairs, both sides were cache misses sharing the same state-prefix hash. The base row and its reversed twin took the same execution path through the same fresh prefill. Nothing about the machine differed. Only the order did.

The derivation is short and it is architectural, not a training artifact. In a vocabulary readout the options go into the prompt as lettered text and the model picks a letter, so A and B are different tokens carrying different learned priors. There is no amount of scale that makes them interchangeable.

And now the hosted model

That is an open 4B model with a letter readout. The interesting question is whether the complaint is right about Jev, which this site has argued for four articles is a per-option scalar scorer whose options never share a context — a shape that cannot express an order effect, because permuting the caller's array permutes the rows of a logits tensor and changes none of them.

Somebody ran it. nibzard/decision-model-benchmark is an independent five-suite benchmark of decision models that calls the TypeSafe API directly with its own key, publishes every raw per-attempt log, and reports its own spend. Its S4 suite is 100 Banking77 items rerun under three option permutations, and its README states the finding in one line:

The raw archive carries the provider's full response body for every attempt, including the complete probability vector over all 77 options. So the claim can be checked at the level the complaint actually makes it — probabilities, not just the argmax. Holding the permutation fixed and taking the median over its three repeats, across the 100 base items:

• The answer changes on 12 of 100.
• The probability vector moves by a median of 0.04 and a maximum of 0.69.
• The confidence field moves by a mean of 0.073 and up to 0.37.

So the complaint is right, on the hosted model, in the specific terms it uses. The order of the choices does change the output probabilities, and on the worst item it changes one by 0.69.

Claim 2 · the same request, twice

This is a different claim and a stronger one, and the mundane explanations for it have nothing to do with the model being unsound. A hosted API batches your request with whatever else arrives in the same window, and a different batch can take a different kernel path. GPU reductions are not associative and do not promise a fixed order. A fleet that is being scaled is not homogeneous. And a model alias can move under the caller without the caller changing anything.

The honest thing to do here is to say which of those are plausible for this API before reaching for any of them, and then to notice that the same benchmark ran the experiment without meaning to.

Its protocol pins repeats: 3 — three shots per item, recorded so the report can take a latency median. Three shots per item is also, exactly, the determinism arm. Across all five suites that is 1,300 items sent three times each with identical request bodies, confirmed identical by the input_tokens the provider bills back: the same number on all three repeats, for every one of the 1,300.

67.2% of items came back with a different probability vector on at least one repeat, and 3.3% changed their answer. On the suite built out of genuinely uncertain items it is 98% and 12%.

What that rules out

Three of the usual explanations are dead on arrival here, and it is worth walking them because the surviving ones are what the experiment has to be designed around.

It is not sampling. There is nothing to sample. The whole point of the readout is that the answer is read off one forward pass rather than drawn token by token, and the API surface reflects that: I went through the request schema and there is no seed, no temperature, no top_p, no idempotency key, no request id. For a generative model your first move would be to set the temperature to zero. Here there is no knob, because nothing is being drawn. Any variation is numerical or infrastructural by elimination.

It is not the version pointer. jev-latest and jev-preview are aliases that both currently resolve to jev-1.13.0, and the models page is straightforward about what that costs you:

The benchmark calls the alias, which I would not do. But because it logs the whole response, the check is available after the fact: all 3,900 responses that carried an answer reported jev-1.13.0. The pointer did not move under this run.

It is not the client. Identical input_tokens on every repeat of every item means the bytes that reached the tokenizer were the same bytes. Nothing was being re-rendered between calls.

The trap in reading "different probabilities"

There is one more thing to dispatch before treating the number as real, and it is the trap the complaint could easily have fallen into.

Every probability the API returns is quantised to two decimals. That is not a formatting choice in the docs — I checked the values: 69,300 of 69,300 returned probabilities in the permutation suite sit exactly on the 0.01 grid, and the vectors sum to 0.99 or 1.00. Bespoke ran into the same thing from the other side and noted that it makes Jev's NLL incomparable, because a true label arriving as 0.00 blows up a log.

A quantiser with a step of 0.01 turns arbitrarily small noise into a visible change whenever the underlying value happens to sit near a rounding boundary. So "I ran it twice and the numbers were different" is, on its own, compatible with a wobble of 0.0001. The existence of a difference proves nothing. The size of it does.

Sixty of the hundred decisions move by two grid steps or more under pure repetition, and 36 of them move by more than four. That is not a value flickering across a boundary; that is the underlying number moving by 0.05 or worse, between two requests that were byte-identical.

For contrast, here is what the same measurement looks like when someone controls the serving path. openjev ran its 777-decision workload three ways on one machine — fresh prefill, a shared state-prefix cache, and a batched pass over every option at once — with the prompt hashes verified identical across all three:

Three changes per 777 decisions on each of the first two rows, and every one of those six was an exact 0.5/0.5 tie in one of the two execution shapes. The repo's own summary of that table is the sentence to take away: "This is measured speed with small numerical differences, not bit-identical reuse." Its reproduction guide says the same thing up front — "BF16/kernel differences can change borderline probabilities or choices", so treat model outputs "as measurements to compare with the committed row-level evidence, not byte-identical golden outputs."

expand

So run-to-run variation in this class of model is real, expected, and documented by the people building the open versions. What is different about the hosted number is the size: a mean movement of 0.036 against 0.0073, and an answer-change rate of 3.3% across the hosted model's five suites against 0.4% here. Five times the movement and eight times the flips.

kev shows what buying the other end of that tradeoff looks like. Its README states the two design decisions plainly: "The server handles one request at a time. It caches repeated state text, but doesn't batch requests from different callers." And it scores its benchmarks in fp32 even though it serves bf16, with the result that "asking questions together or separately produces probabilities within 4e-6 in the fp32 tests." Four parts in a million. That is what a serving stack that refuses to batch across callers and refuses to round the arithmetic can promise, and it is also why nobody runs a hosted API that way.

What I cannot settle

Which of the remaining mechanisms is responsible. From outside the API there is no way to tell batch composition from reduction order from a heterogeneous fleet, and the rate limits page volunteers that the fleet is in motion — limits "can change without notice while we do, as upcoming large GPU deals land", with 529 Overloaded in the error table.

There is one hypothesis I would put money on and cannot test, so it is labelled. Reasoned: if Jev is a mixture-of-experts model — and the only outside size estimate puts it "in the 30Bn range" — then batch composition is not incidental to the arithmetic, it is in it. Capacity-limited expert routing makes a token's expert assignment depend on what else is in the batch, which is a per-request-neighbour effect rather than a rounding effect, and that is the right order of magnitude for what is measured here. The Rollout Routing Replay piece covers the training-side version of the same problem, where the paper reports that running the same sequence through Megatron twice gives a KL of 0.84×10⁻³ because "the MoE forward pass is not deterministic." A dense model served with fixed batching should not behave like this. An MoE served with dynamic batching should behave exactly like this.

The 13% is two numbers wearing one coat

Here is where separating the claims stops being pedantry and starts changing a published figure.

The benchmark's S4 flip rate is computed by pooling every observation of a base item — three permutations times three repeats, nine calls — and asking whether the choice was constant. That is an honest metric and it is also the conflation this article is about: it counts an item as order-unstable when it was only ever repeat-unstable.

Splitting the two, on the same 100 items:

My pipeline reproduces the published 13.0% exactly, and the published mean confidence range of 0.092 to within a thousandth, which is the check that the decomposition is reading the same rows the report did.

So the real reading of that number is: 12% order instability, 5% run-to-run instability, and one item that is counted as a position-bias failure while being nothing of the kind. Nobody has published the second figure, because nobody was looking for it — the repeats were there to take a latency median.

A correction to an earlier piece

What a decision model cannot do said of this number: "Laya's BENCHMARKS.md reports 'Jev measured at 0.13' for option-order instability, with no citation… I could not find the primary source." It then called it "the single most load-bearing unverified figure in this whole category."

The source exists, and I should have found it. Laya's own comparison chart names it in the subtitle — "Jev figures are third-party published (AbdelStark/jev-benchmarks, nibzard/decision-model-benchmark) — Jev was never run in this project" — and the second of those is the benchmark above. The 0.13 is real, it was measured through the API by somebody who paid for it, and the raw logs are committed.

Two things about it were still wrong in the received version, though, and they are the more interesting half:

• It is not 13% order instability. It is the union of order and repetition, and the split is 12 and 5.
• It is measured only at 77 options. Which matters a great deal, as the next section argues.

Credit where it belongs: that repository publishes a CORRECTIONS.md mapping every number it revised to the defect that produced it, prices its own run to the cent ($0.2079 for 4,125 jev decisions; $28.34 across all contenders), and reports that jev has the worst calibration error in the field it assembled. It is the most careful outside measurement this category has.

Claim 3 · "no hallucination"

Work out what the claim is before judging it, because the version in most people's heads is not the version that was shipped.

TypeSafe's own framing, from the launch post: "No type errors: this would be an easy thing to falsify with just a single counter-example, but it is mathematically impossible." And in the same breath, about the bar on the chart: "Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots."

That is a closure property, not a performance claim. A Choice answer can only be one of the options you supplied; there is no decode step, so there is no path by which a misspelled option or an invented one can appear. The WindTunnel harness makes the same invariant explicit in code — the returned id is looked up in the menu that was sent, and an id that was not offered throws rather than doing something approximate.

The vendor is also explicit about what that does not buy, in the one sentence on the whole documentation site that scopes every other claim: "Calibration is measured across groups of predictions; it does not guarantee that an individual answer is correct."

So: does order sensitivity contradict any of it?

No — and the precise reason is worth having, because it is where the confusion in the complaint actually lives. Permuting the option list permutes the set. The answer is still an element of it. Nothing about a closure property can be disturbed by relabelling which element is first. P1 survives untouched, and it survives whether the model is order-stable, order-sensitive, well calibrated or garbage.

What order sensitivity does break is a third proposition that nobody published: that the answer is a property of the question alone. That is the reading the word "decision" invites, and it is the one that makes "cannot hallucinate" sound like "cannot be wrong." It is measured false: 12 of 100.

It also puts a floor under P2 that is worth stating plainly. Calibration is a property of a sequence of forecasts against outcomes. If the same question, with the same evidence, returns two probabilities 0.69 apart for the same option depending on where that option sits in an array, then the arrangement is either part of the event being forecast — which nobody believes — or at most one of the two numbers can be the model's belief about that event. Order sensitivity does not refute the calibration claim. It bounds how good it can be, and it means a threshold tuned on one arrangement is not tuned for another.

What the complaint accidentally proves

Here is the reframe, and it is the reason this is an article rather than a reply.

Reversing an option list is not a bug report. It is the experiment that tells the two architecture families apart, and this site established that before anyone complained. A per-option scalar scorer — CUA-S1, open-jev-deberta, and by the reading carried across four articles here, Jev — encodes and scores each option alone. In CUA-S1's attention head every contraction is option-against-context; there is no option-against-option term anywhere in the function. Permute the options and you permute the rows of the output tensor; you do not change any of them. Order instability is not small for that family. It is exactly zero, by the type signature. A vocabulary readout cannot reach zero for the same reason: the letters are different tokens.

So a user reporting that "the ORDER of the choices DRASTICALLY changes the output probs" has, without meaning to, run the family-identification experiment and published the result. And the result is: at 77 options, Jev is not a strict per-option scorer.

That is a direct hit on four articles of argument, and it needs saying rather than hedging — with one caveat I cannot remove from outside, which is that the 12 flips and the 5 repeat flips live on the same 100 items and are the same order of magnitude. Taking the majority of three repeats per permutation is the best separation the data allows, and it is not proof.

But the hit is survivable in a specific, testable way, and TypeSafe named the escape hatch in the launch post: high-cardinality Choices run "a 2 stage-system of scoring independently then making an explicit choice." Stage one can be exactly as isolated as the docs describe. Stage two — an explicit choice among survivors — is the one place an ordering could re-enter. Every measurement of order instability in this category has been taken at 77 options, or at 20 on Laya's own checkpoints. Nobody has run one on Jev at two.

The other hypothesis is simpler and the data leans towards it. Look at which items are unstable, under either treatment:

• Under repetition, the suite with a planted correct answer among distractors — where every contender scores 100% — is 74.5% bit-identical with zero answer changes, at a median of 96 options. The suite built to withhold a good option is 2% bit-identical with 12% answer changes, at six options.
• Under reversal on openjev, the ten questions that flipped had a median winning margin of 1.81 logits; the twenty-six that held had 4.50.
• Under execution-shape changes on openjev, every single flip was an exact 0.5/0.5 tie.

Every one of those points at the margin rather than the option count. Which means the high-cardinality suites may be unstable because 77-way intent classification is hard, not because 77 crosses a threshold. Cardinality and difficulty are confounded in every measurement that exists.

expand

And one number pulls hard in the opposite direction from the 13%. kev's published comparison chart reports a permutation flip rate over 36 items of 2.78% for kev and 0.00% for Jev — measured through the API, by somebody with an interest in the other result, and covered here before. Pooling every permutation block that repository commits takes the hosted model to 324 items across seven runs and still zero flips, which the Kev read has the ledger for. Zero and thirteen percent are not the same model behaving the same way. The option counts behind those items are the first thing I would want to know — the widest any of kev's suites names is a six-way emotion label, against 77 here — and the margin distribution is the second.

The experiment that is still missing

So claims 1 and 2 are both true and both measured. What is left is the localisation, and it is one afternoon and an API key. The design, because the existing runs each answer half of it by accident:

Pin the version, not the alias. Send jev-1.13.0. Then assert that response.model equals it on every row and discard any row that disagrees — the models page says that field reports the version that answered, so this is the one mundane explanation you can fully close.

Three arms on the same items, not two.

• Arm C, the floor. Each item sent K times, byte-identical, option order fixed. This is the noise, and without it nothing else is readable.
• Arm B, the order. The same items under P permutations, each permutation itself repeated K times so its own noise can be averaged out before comparing across permutations.
• Arm A, the composition control. The same items sent one question per request and again batched several questions into one request. kev measured this on its own server at 4×10⁻⁶ and nobody has measured it on the hosted one; if it is large, the mechanism is batching and you have localised it from outside.

Report the size and not the existence, in grid steps of 0.01, because that is the only resolution the API has. Arm B minus Arm C is the order effect. Arm C alone is the determinism answer.

Sweep the cardinality at fixed margin. This is the part nobody has done and it is the whole architectural question. Take one item set, hold the difficulty constant by construction — a planted correct answer with fixed distractor similarity, which is exactly what that benchmark's S3 suite already builds — and run Arm B at its own ladder of 2, 8, 32, 128 and 255 options. A strict per-option scorer must score exactly 0 at every cardinality. If instability is 0 at two options and appears above some N, stage two is where the order lives and the isolation reading survives for stage one. If it is non-zero at two options, the reading is wrong everywhere and four articles on this site need revising.

Reversing a two-option list is the cheapest experiment in this entire category and nobody has run it.

The cost is not the obstacle. That benchmark's complete jev run — 4,125 decisions across five suites, three repeats each — cost $0.2079. The arms above are maybe four times that.

So, is Jev deterministic?

No. Identical requests return different probabilities two thirds of the time and a different answer 3.3% of the time, and on genuinely ambiguous items the answer changes 12% of the time. That is measured, on the pinned version, from a third party's published logs, and it is the first time I have seen it written down.

Does the order of the choices change the output? Yes, and by more — about two and a half times the movement and two and a half times the flip rate, on the same items in the same run. Both are real. They are not the same thing, they do not have the same cause, and no published number has ever separated them.

Does any of that undermine "no hallucination"? Not the claim that was made. The answer is an element of the set you sent, always, and no arrangement of that set changes it. What it undermines is the thing people hear instead — that the answer is a property of the question — and the useful version of the complaint is that one.

The practical consequence is short enough to act on. If you have a confidence threshold in production, it is tuned to an arrangement and to a run. Average over a few permutations before you threshold, and measure your own floor by sending the same thing twice, because the vendor publishes neither number and its failure-modes page — which is unusually honest about eight other things — does not mention either.