# Order is not noise
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/jev-is-not-deterministic
> date: 2026-09-22
> tags: explainer, llm, architecture, calibration
A complaint about Jev has been going around, and it is worth quoting whole because
the shape of it matters more than the grievance:
> "JEV actually doesn't even run deterministically. Exact same prompts give you
> different probabilities when you run it multiple times. The ORDER of the choices
> DRASTICALLY changes the output probs. I am more and more confused by their
> 'no-hallucination' claim."
That is three claims, not one. They are not equally supported, they are not
supported by the same evidence, and the experiment that settles one of them tells
you nothing about the others. Bundling them is what turns a set of findings into a
verdict.
Two of the three turn out to be true. One is a misreading — of a claim that is
narrower and duller than the word "hallucination" suggests. And the separation
matters more than either result, because a benchmark that reorders options while
also re-sending requests reports one number containing both effects, which is
exactly what the single published figure for this is.
Everything below is recomputed from published artifacts. I have no TypeSafe API
key. I did not need one, which is the part I did not expect when I started.
## Claim 1 · the order changes the output
This one this site has already measured, on an open model, and the number is not
small.
[openjev](https://github.com/TheoLeeCJ/openjev) ships an `option_reversal`
perturbation — the same evidence, the same question, the same options in the
opposite order — and commits both its fixtures and its raw per-row predictions
with the `base_id` each perturbed row came from. Joining those back gives 36 pairs.
I re-ran the join at commit `ca3ba65`:
- **The answer changes on 10 of 36 — 27.8%.**
- The probability vector moves by a median of **0.156** and a maximum of **0.870**,
in openjev's own metric (the largest movement of any single option).
- The mechanism is a position prior. All 36 questions have three options, so
reversal swaps slots A and C and leaves B exactly where it was; using B as a drift
control, **slot A is worth +1.71 logits on average, +1.56 at the median, positive
on 32 of the 36 questions**. Moving an option's text from first to last costs 1.54
logits; moving it from last to first gains 1.88. That is about five and a half
times the odds, for nothing written in the option.
One methodological detail I checked this time and had not before, because it is the
thing that would invalidate the whole measurement: for all 36 pairs, **both sides
were cache misses sharing the same state-prefix hash**. The base row and its
reversed twin took the same execution path through the same fresh prefill. Nothing
about the machine differed. Only the order did.
[The derivation](/articles/what-decision-models-cannot-do) is short and it is
architectural, not a training artifact. In a vocabulary readout the options go into
the prompt as lettered text and the model picks a letter, so `A` and `B` are
different tokens carrying different learned priors. There is no amount of scale that
makes them interchangeable.
### And now the hosted model
That is an open 4B model with a letter readout. The interesting question is whether
the complaint is right about **Jev**, which this site has argued for four articles
is a per-option scalar scorer whose options never share a context — a shape that
*cannot* express an order effect, because permuting the caller's array permutes the
rows of a logits tensor and changes none of them.
Somebody ran it. [`nibzard/decision-model-benchmark`](https://github.com/nibzard/decision-model-benchmark)
is an independent five-suite benchmark of decision models that calls the TypeSafe
API directly with its own key, publishes every raw per-attempt log, and reports
its own spend. Its S4 suite is 100 Banking77 items rerun under three option
permutations, and its README states the finding in one line:
**"Position bias is a measurable axis. Rerun the same items with permuted option order: jev changes 13% of its choices, the worst LLM 37%. Vendor evaluations do not measure this."**
The raw archive carries the provider's full response body for every attempt,
including the complete probability vector over all 77 options. So the claim can be
checked at the level the complaint actually makes it — probabilities, not just the
argmax. Holding the permutation fixed and taking the median over its three repeats,
across the 100 base items:
- **The answer changes on 12 of 100.**
- The probability vector moves by a median of **0.04** and a maximum of **0.69**.
- The `confidence` field moves by a mean of **0.073** and up to **0.37**.
So the complaint is right, on the hosted model, in the specific terms it uses. The
order of the choices does change the output probabilities, and on the worst item it
changes one by 0.69.
## Claim 2 · the same request, twice
This is a different claim and a stronger one, and the mundane explanations for it
have nothing to do with the model being unsound. A hosted API batches your request
with whatever else arrives in the same window, and a different batch can take a
different kernel path. GPU reductions are not associative and do not promise a fixed
order. A fleet that is being scaled is not homogeneous. And a model alias can move
under the caller without the caller changing anything.
The honest thing to do here is to say which of those are plausible for this API
before reaching for any of them, and then to notice that the same benchmark ran the
experiment without meaning to.
Its protocol pins `repeats: 3` — three shots per item, recorded so the report can
take a latency median. Three shots per item is also, exactly, the determinism arm.
Across all five suites that is 1,300 items sent three times each with identical
request bodies, confirmed identical by the `input_tokens` the provider bills back:
the same number on all three repeats, for every one of the 1,300.
**Receipts.** Jev does not return the same probabilities twice for the same request. Across 1,300 items sent three times each with byte-identical bodies, 873 of them (67.2%) came back with a different probability vector on at least one repeat, and the chosen answer itself changed on 43 (3.3%). Every response echoed the pinned version jev-1.13.0, so no model change is involved, and the API exposes no seed, temperature or sampling parameter that could be blamed.
| suite | options | items | identical all 3× | answer changed | max |Δp| within an item (median / max) |
| :--- | ---: | ---: | ---: | ---: | ---: |
| S1 · Banking77 intent | 77 | 300 | 113 (37.7%) | 10 (3.3%) | 0.01 / 0.19 |
| S2 · SMS spam | 2 | 300 | 53 (17.7%) | 2 (0.7%) | 0.02 / 0.13 |
| S3 · planted answer, N sweep | 2–255 | 200 | 149 (74.5%) | 0 (0.0%) | 0.00 / 0.10 |
| S4 · S1 with permuted options | 77 | 300 | 108 (36.0%) | 7 (2.3%) | 0.02 / 0.16 |
| S5 · forced uncertainty | 6 | 200 | 4 (2.0%) | 24 (12.0%) | 0.04 / 0.15 |
| all five | 2–255 | 1,300 | 427 (32.8%) | 43 (3.3%) | 0.02 / 0.19 |
Read the last column, not the first. Instability is not spread evenly: S3 plants the correct answer among distractors and every contender scores 100% on it, and it is also the most reproducible suite — three quarters of its items are bit-identical and not one answer moved. S5 withholds a good option on purpose, and only 2% of its items are bit-identical while 12% change their answer with nothing changed. The readout is reproducible exactly where the decision is easy. That is the same pattern option reversal shows, and it is why an aggregate accuracy number hides both.
> method: Recomputed from the raw per-attempt logs published in nibzard/decision-model-benchmark, run v1.1 (2026-09-18T09:56:45Z), which records the provider's full response body for every attempt. The runner's protocol is repeats=3 at temperature 0 with concurrency 4; identical input_tokens across all three repeats of every item confirms the request bodies did not differ. An item counts as identical only if every option's probability is equal across all three repeats; S3's 275 items drop to 200 because the API answers `400 Too many choices` above 255 options, and a rejected request has no probabilities to compare. Probabilities arrive quantised to two decimals — all 69,300 values in the S4 suite lie exactly on the 0.01 grid — so this measures movement of at least half a grid step, not floating-point noise below it.
> source: https://github.com/nibzard/decision-model-benchmark
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/jev-is-not-deterministic/data/repeat-determinism.json (6 rows)
**67.2% of items came back with a different probability vector on at least one
repeat, and 3.3% changed their answer.** On the suite built out of genuinely
uncertain items it is 98% and 12%.
### What that rules out
Three of the usual explanations are dead on arrival here, and it is worth walking
them because the surviving ones are what the experiment has to be designed around.
**It is not sampling.** There is nothing to sample. The whole point of the readout
is that the answer is read off one forward pass rather than drawn token by token,
and the API surface reflects that: I went through the request schema and there is no
`seed`, no `temperature`, no `top_p`, no idempotency key, no request id. For a
generative model your first move would be to set the temperature to zero. Here there
is no knob, because nothing is being drawn. Any variation is numerical or
infrastructural by elimination.
**It is not the version pointer.** `jev-latest` and `jev-preview` are aliases that
both currently resolve to `jev-1.13.0`, and the models page is straightforward about
what that costs you:
**"An alias moves when a new release ships, so the answers behind it can change without a change on your side."** And then the fix, in the same paragraph: *"The response's `model` field reports the versioned ID that answered, so you can log which model produced each result."*
The benchmark calls the alias, which I would not do. But because it logs the whole
response, the check is available after the fact: **all 3,900 responses that carried
an answer reported `jev-1.13.0`**. The pointer did not move under this run.
**It is not the client.** Identical `input_tokens` on every repeat of every item
means the bytes that reached the tokenizer were the same bytes. Nothing was being
re-rendered between calls.
### The trap in reading "different probabilities"
There is one more thing to dispatch before treating the number as real, and it is
the trap the complaint could easily have fallen into.
Every probability the API returns is quantised to two decimals. That is not a
formatting choice in the docs — I checked the values: **69,300 of 69,300 returned
probabilities in the permutation suite sit exactly on the 0.01 grid**, and the
vectors sum to 0.99 or 1.00. Bespoke ran into the same thing from the other side and
noted that it makes Jev's NLL incomparable, because a true label arriving as `0.00`
blows up a log.
A quantiser with a step of 0.01 turns arbitrarily small noise into a visible change
whenever the underlying value happens to sit near a rounding boundary. So "I ran it
twice and the numbers were different" is, on its own, compatible with a wobble of
0.0001. The existence of a difference proves nothing. **The size of it does.**
Sixty of the hundred decisions move by two grid steps or more under pure
repetition, and 36 of them move by more than four. That is not a value flickering
across a boundary; that is the underlying number moving by 0.05 or worse, between
two requests that were byte-identical.
For contrast, here is what the same measurement looks like when someone controls the
serving path. openjev ran its 777-decision workload three ways on one machine —
fresh prefill, a shared state-prefix cache, and a batched pass over every option at
once — with the prompt hashes verified identical across all three:
| what changed | decisions | answer changed | mean movement | worst |
|---|---|---|---|---|
| batched pass instead of fresh | 777 | 3 | 0.0073 | 0.060 |
| shared state-prefix cache | 777 | 3 | 0.0074 | 0.067 |
| Metal instead of CUDA, same weights | 144 | 0 | 0.0086 | 0.105 |
Three changes per 777 decisions on each of the first two rows, and **every one of
those six was an exact 0.5/0.5 tie in one of the two execution shapes**. The repo's own summary of that
table is the sentence to take away: *"This is measured speed with small numerical
differences, not bit-identical reuse."* Its reproduction guide says the same thing
up front — *"BF16/kernel differences can change borderline probabilities or
choices"*, so treat model outputs *"as measurements to compare with the committed
row-level evidence, not byte-identical golden outputs."*
So run-to-run variation in this class of model is real, expected, and documented by
the people building the open versions. What is different about the hosted number is
the **size**: a mean movement of 0.036 against 0.0073, and an answer-change rate of
3.3% across the hosted model's five suites against 0.4% here. Five times the movement
and eight times the flips.
[`kev`](https://github.com/jaredpalmer/kev) shows what buying the other end of that
tradeoff looks like. Its README states the two design decisions plainly: *"The server
handles one request at a time. It caches repeated state text, but doesn't batch
requests from different callers."* And it scores its benchmarks in fp32 even though
it serves bf16, with the result that *"asking questions together or separately
produces probabilities within 4e-6 in the fp32 tests."* Four parts in a million. That
is what a serving stack that refuses to batch across callers and refuses to round the
arithmetic can promise, and it is also why nobody runs a hosted API that way.
### What I cannot settle
Which of the remaining mechanisms is responsible. From outside the API there is no
way to tell batch composition from reduction order from a heterogeneous fleet, and
the rate limits page volunteers that the fleet is in motion — limits *"can change
without notice while we do, as upcoming large GPU deals land"*, with `529 Overloaded`
in the error table.
There is one hypothesis I would put money on and cannot test, so it is labelled.
**Reasoned:** if Jev is a mixture-of-experts model — and the only outside size
estimate puts it *"in the 30Bn range"* — then batch composition is not incidental to
the arithmetic, it is *in* it. Capacity-limited expert routing makes a token's expert
assignment depend on what else is in the batch, which is a per-request-neighbour
effect rather than a rounding effect, and that is the right order of magnitude for
what is measured here. The [Rollout Routing Replay
piece](/articles/rollout-routing-replay) covers the training-side version of the same
problem, where the paper reports that running the same sequence through Megatron
twice gives a KL of 0.84×10⁻³ because *"the MoE forward pass is not deterministic."*
A dense model served with fixed batching should not behave like this. An MoE served
with dynamic batching should behave exactly like this.
## The 13% is two numbers wearing one coat
Here is where separating the claims stops being pedantry and starts changing a
published figure.
The benchmark's S4 flip rate is computed by pooling every observation of a base item
— three permutations times three repeats, nine calls — and asking whether the choice
was constant. That is an honest metric and it is also the conflation this article is
about: it counts an item as order-unstable when it was only ever repeat-unstable.
Splitting the two, on the same 100 items:
| what moved | items whose answer changed |
|---|---|
| the option order (majority of 3 repeats, per permutation) | 12 |
| nothing (a permutation not constant across its own 3 repeats) | 5 |
| both | 4 |
| **the union — the published figure** | **13** |
My pipeline reproduces the published 13.0% exactly, and the published mean
confidence range of 0.092 to within a thousandth, which is the check that the
decomposition is reading the same rows the report did.
So the real reading of that number is: **12% order instability, 5% run-to-run
instability, and one item that is counted as a position-bias failure while being
nothing of the kind.** Nobody has published the second figure, because nobody was
looking for it — the repeats were there to take a latency median.
### A correction to an earlier piece
[What a decision model cannot do](/articles/what-decision-models-cannot-do) said of
this number: *"Laya's `BENCHMARKS.md` reports 'Jev measured at 0.13' for
option-order instability, with no citation… I could not find the primary source."*
It then called it *"the single most load-bearing unverified figure in this whole
category."*
The source exists, and I should have found it. Laya's own comparison chart names it
in the subtitle — *"Jev figures are third-party published (AbdelStark/jev-benchmarks,
nibzard/decision-model-benchmark) — Jev was never run in this project"* — and the
second of those is the benchmark above. The 0.13 is real, it was measured through
the API by somebody who paid for it, and the raw logs are committed.
Two things about it were still wrong in the received version, though, and they are
the more interesting half:
- It is **not 13% order instability**. It is the union of order and repetition, and
the split is 12 and 5.
- It is measured **only at 77 options**. Which matters a great deal, as the next
section argues.
Credit where it belongs: that repository publishes a `CORRECTIONS.md` mapping every
number it revised to the defect that produced it, prices its own run to the cent
(\$0.2079 for 4,125 jev decisions; \$28.34 across all contenders), and reports that
jev has the worst calibration error in the field it assembled. It is the most
careful outside measurement this category has.
## Claim 3 · "no hallucination"
Work out what the claim is before judging it, because the version in most people's
heads is not the version that was shipped.
TypeSafe's own framing, from the launch post: **"No type errors: this would be an
easy thing to falsify with just a single counter-example, but it is mathematically
impossible."** And in the same breath, about the bar on the chart: *"Our number is
not empirical. Schema matching is guaranteed, thus we can confidently add 0% into
the plots."*
That is a closure property, not a performance claim. A Choice answer can only be one
of the options you supplied; there is no decode step, so there is no path by which a
misspelled option or an invented one can appear. The [WindTunnel
harness](/articles/what-decision-models-cannot-do) makes the same invariant explicit
in code — the returned id is looked up in the menu that was sent, and an id that was
not offered throws rather than doing something approximate.
The vendor is also explicit about what that does not buy, in the one sentence on the
whole documentation site that scopes every other claim: **"Calibration is measured
across groups of predictions; it does not guarantee that an individual answer is
correct."**
So: does order sensitivity contradict any of it?
No — and the precise reason is worth having, because it is where the confusion in
the complaint actually lives. Permuting the option list permutes the *set*. The
answer is still an element of it. Nothing about a closure property can be disturbed
by relabelling which element is first. P1 survives untouched, and it survives whether
the model is order-stable, order-sensitive, well calibrated or garbage.
What order sensitivity *does* break is a third proposition that nobody published:
**that the answer is a property of the question alone**. That is the reading the word
"decision" invites, and it is the one that makes "cannot hallucinate" sound like
"cannot be wrong." It is measured false: 12 of 100.
It also puts a floor under P2 that is worth stating plainly. Calibration is a
property of a sequence of forecasts against outcomes. If the same question, with the
same evidence, returns two probabilities 0.69 apart for the same option depending on
where that option sits in an array, then the arrangement is either part of the event
being forecast — which nobody believes — or at most one of the two numbers can be
the model's belief about that event. Order sensitivity does not refute the
calibration claim. It bounds how good it can be, and it means a threshold tuned on
one arrangement is not tuned for another.
## What the complaint accidentally proves
Here is the reframe, and it is the reason this is an article rather than a reply.
**Reversing an option list is not a bug report. It is the experiment that tells the
two architecture families apart**, and [this site established
that](/articles/cua-s1-forms) before anyone complained. A per-option scalar scorer —
CUA-S1, `open-jev-deberta`, and by the reading carried across four articles here,
Jev — encodes and scores each option alone. In CUA-S1's attention head every
contraction is option-against-*context*; there is no option-against-option term
anywhere in the function. Permute the options and you permute the rows of the output
tensor; you do not change any of them. Order instability is not small for that
family. It is **exactly zero, by the type signature**. A vocabulary readout cannot
reach zero for the same reason: the letters are different tokens.
So a user reporting that "the ORDER of the choices DRASTICALLY changes the output
probs" has, without meaning to, run the family-identification experiment and
published the result. And the result is: **at 77 options, Jev is not a strict
per-option scorer.**
That is a direct hit on four articles of argument, and it needs saying rather than
hedging — with one caveat I cannot remove from outside, which is that the 12 flips
and the 5 repeat flips live on the same 100 items and are the same order of
magnitude. Taking the majority of three repeats per permutation is the best
separation the data allows, and it is not proof.
But the hit is survivable in a specific, testable way, and TypeSafe named the
escape hatch in the launch post: high-cardinality Choices run *"a 2 stage-system of
scoring independently then making an explicit choice."* Stage one can be exactly as
isolated as the docs describe. Stage two — an explicit choice among survivors — is
the one place an ordering could re-enter. Every measurement of order instability in
this category has been taken at 77 options, or at 20 on Laya's own checkpoints.
Nobody has run one on Jev at two.
The other hypothesis is simpler and the data leans towards it. Look at which items
are unstable, under either treatment:
- Under repetition, the suite with a planted correct answer among distractors — where
every contender scores 100% — is **74.5% bit-identical with zero answer changes**,
at a median of 96 options. The suite built to withhold a good option is 2%
bit-identical with 12% answer changes, at six options.
- Under reversal on openjev, the ten questions that flipped had a median winning
margin of **1.81 logits**; the twenty-six that held had **4.50**.
- Under execution-shape changes on openjev, **every single flip was an exact 0.5/0.5
tie**.
Every one of those points at the **margin** rather than the option count. Which means
the high-cardinality suites may be unstable because 77-way intent classification is
hard, not because 77 crosses a threshold. Cardinality and difficulty are confounded in every
measurement that exists.
And one number pulls hard in the opposite direction from the 13%. `kev`'s published
comparison chart reports a permutation flip rate over 36 items of **2.78% for kev
and 0.00% for Jev** — measured through the API, by somebody with an interest in the
other result, and [covered here
before](/articles/any-model-can-be-jev). Pooling every `permutation` block that
repository commits takes the hosted model to **324 items across seven runs and
still zero flips**, which [the Kev read](/articles/kev) has the ledger for.
Zero and thirteen percent are not the same model behaving the same way. The
option counts behind those items are the first thing I would want to know — the
widest any of kev's suites names is a six-way emotion label, against 77 here —
and the margin distribution is the second.
## The experiment that is still missing
So claims 1 and 2 are both true and both measured. What is left is the localisation,
and it is one afternoon and an API key. The design, because the existing runs each
answer half of it by accident:
**Pin the version, not the alias.** Send `jev-1.13.0`. Then assert that
`response.model` equals it on every row and discard any row that disagrees — the
models page says that field reports the version that answered, so this is the one
mundane explanation you can fully close.
**Three arms on the same items, not two.**
- *Arm C, the floor.* Each item sent K times, byte-identical, option order fixed.
This is the noise, and without it nothing else is readable.
- *Arm B, the order.* The same items under P permutations, each permutation itself
repeated K times so its own noise can be averaged out before comparing across
permutations.
- *Arm A, the composition control.* The same items sent one question per request and
again batched several questions into one request. kev measured this on its own
server at 4×10⁻⁶ and nobody has measured it on the hosted one; if it is large, the
mechanism is batching and you have localised it from outside.
Report the *size* and not the existence, in grid steps of 0.01, because that is the
only resolution the API has. Arm B minus Arm C is the order effect. Arm C alone is
the determinism answer.
**Sweep the cardinality at fixed margin.** This is the part nobody has done and it
is the whole architectural question. Take one item set, hold the difficulty constant
by construction — a planted correct answer with fixed distractor similarity, which
is exactly what that benchmark's S3 suite already builds — and run Arm B at its own
ladder of 2, 8, 32, 128 and 255 options. A strict per-option scorer must score exactly 0 at every
cardinality. If instability is 0 at two options and appears above some N, stage two
is where the order lives and the isolation reading survives for stage one. If it is
non-zero at **two** options, the reading is wrong everywhere and four articles on
this site need revising.
Reversing a two-option list is the cheapest experiment in this entire category and
nobody has run it.
The cost is not the obstacle. That benchmark's complete jev run — 4,125 decisions
across five suites, three repeats each — cost **\$0.2079**. The arms above are maybe
four times that.
## So, is Jev deterministic?
No. Identical requests return different probabilities two thirds of the time and a
different answer 3.3% of the time, and on genuinely ambiguous items the answer
changes 12% of the time. That is measured, on the pinned version, from a third
party's published logs, and it is the first time I have seen it written down.
Does the order of the choices change the output? Yes, and by more — about two and a
half times the movement and two and a half times the flip rate, on the same items in
the same run. Both are real. They are not the same thing, they do not have the same
cause, and no published number has ever separated them.
Does any of that undermine "no hallucination"? Not the claim that was made. The
answer is an element of the set you sent, always, and no arrangement of that set
changes it. What it undermines is the thing people hear instead — that the answer is
a property of the question — and the useful version of the complaint is that one.
The practical consequence is short enough to act on. If you have a confidence
threshold in production, it is tuned to an arrangement and to a run. Average over a
few permutations before you threshold, and measure your own floor by sending the same
thing twice, because the vendor publishes neither number and its failure-modes page —
which is unusually honest about eight other things — does not mention either.
**Receipts.** One complaint about Jev contains three separate assertions. They are not equally supported, they need different experiments, and two of them turn out to be true while the third is a misreading of what was claimed. Filing them as one grievance is what makes the whole thing look like a verdict rather than three findings.
| the assertion | what kind of claim it is | strongest evidence | verdict |
| :--- | :--- | :--- | :--- |
| The order of the choices drastically changes the output probabilities. | Empirical, about one model. Settled by reordering one array. | Hosted Jev, 100 Banking77 items × 3 permutations × 3 repeats: the answer changes on 12, and the probability vector moves by a median of 0.04 and up to 0.69. Local vocabulary readout (openjev, 36 reversal pairs): 27.8% flip, slot A worth +1.71 logits. | True, and measured on both a hosted and an open model. |
| Exact same prompts give you different probabilities when you run it multiple times. | Empirical, about a serving stack. Needs a repeat arm, not a reorder arm. | Hosted Jev, 1,300 items sent three times each with identical bodies and the pinned version echoed on every response: 67.2% returned a different probability vector, 3.3% changed their answer. | True. The cause is not identified and cannot be from outside. |
| …which is why I am confused by the no-hallucination claim. | A claim about a claim. Settled by reading what was asserted. | TypeSafe asserts schema closure — "Schema matching is guaranteed" — and separately documents that "Calibration… does not guarantee that an individual answer is correct." Neither sentence says the answer is a function of the question alone. | The guarantee holds; the reading that order sensitivity would break was never made. |
The reason to separate them is that they have different fixes. Order sensitivity is trained out with option-order shuffling and worked around by averaging over permutations. Run-to-run variation is a serving property and is fixed, if at all, by the vendor's inference stack. And the type-safety guarantee needs no fix, because it is not what anybody was actually complaining about.
> method: Each row records what the assertion actually says, the strongest evidence that bears on it, and the verdict that evidence supports. Rows 1 and 2 are recomputed in this article from published raw logs: the hosted-model numbers come from nibzard/decision-model-benchmark run v1.1, the local-model numbers from openjev's committed predictions at ca3ba65. Row 3 is a reading of TypeSafe's own wording against its own documentation, not a measurement.
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/jev-is-not-deterministic/data/three-claims.json (3 rows)
Measured from nibzard/decision-model-benchmark run v1.1: 1,300 items sent three times each, identical `input_tokens` on every repeat, `jev-1.13.0` echoed on all 3,900 answered responses, 67.2% of items differing and 3.3% changing answer. This is one run on one day by one author, fired at concurrency 4 so the repeats were seconds apart and in each other's company — which is where batch effects would be *strongest*, and is therefore the least favourable sampling for the vendor. Re-run it with the repeats spread over hours and across load conditions. If the rate collapses when the repeats are separated in time, the effect is batch composition and not a general property of the service; if it stays, it is deeper in the stack.
The split comes from taking the majority choice across each permutation's three repeats before comparing across permutations. That assumes a majority of three is a good estimate of the permutation's "true" answer, which is exactly the assumption the noise measurement undermines. Run the same decomposition with K = 9 or 15 repeats per permutation. If the order-only count climbs towards 13 as K grows, my split is an artifact of a thin majority and the order effect was being masked by the noise I was trying to remove.
This follows only if the 12 measured flips are attributable to the ordering rather than to the run-to-run variation that shares the same items — and the two are the same order of magnitude, which is the uncomfortable part. The decisive version is the cardinality sweep at fixed margin described above, run at two options. A strict per-option scorer permuting a two-element array must return a bit-identical vector; there is no tie-break, no second stage, and nothing left to blame. A non-zero rate there falsifies option isolation outright. A zero rate at two options with a non-zero rate at 77 localises the effect to the documented second stage and leaves the isolation reading standing for stage one.
The evidence is three coincidences: the easy planted-answer suite is the most reproducible at a median of 96 options, the hard forced-uncertainty suite is the least at six, openjev's reversal flips had a median margin of 1.81 logits against 4.50 for the ones that held, and every execution-shape flip was an exact tie. Every one of those confounds difficulty with something else. Bin any of these suites by the winning margin and check whether the flip rate falls monotonically within a cardinality. If it does not — if 77-option items with a wide margin are as unstable as narrow ones — the margin story is wrong and cardinality is doing the work after all.
All 69,300 returned values sit on the 0.01 grid, and 60 of 100 decisions move by two or more grid steps under pure repetition, which a boundary crossing cannot produce. The weak point is that each value is a maximum over 77 options, so the statistic is biased upward by the option count. Redo it per-option rather than per-decision, or on the two-option spam suite where there is no maximum to take. If the two-option suite's movement collapses to one step, the effect is partly the maximum and my magnitudes are inflated — though note its answer still changed on 2 of 300 items, which no amount of rounding explains.
This is a hypothesis with no direct evidence, offered because the magnitude fits and the alternatives fit worse. It predicts that variance should correlate with concurrent load and should be much smaller for a request that lands alone. Run the repeat arm twice, once against an idle key and once while saturating your own rate limit with unrelated traffic. If the variance is indifferent to load, batching is not the mechanism and I would want to hear what is — reduction order alone should be far below a grid step.