What a decision model cannot do

Everything published about System One models this week has been about what they can do. The architecture, the open rebuilds, the one benchmark row that says zero. This piece is the negative space, and I think it is the more useful half, because none of these limits is a bug someone will fix in the next checkpoint. They fall out of the shape.

I started with a thesis I liked: one architectural choice produces every limitation observed this week. Each option is scored without seeing the others, the output is bounded to the option set, and everything else follows. It is a tidy claim, it makes a good diagram, and it is about 60% true.

Here is where it survived and where it broke, before the details:

• It cannot write a string that is not already an option. Forced. True of every family, no exceptions.
• It cannot keep an intermediate result between calls. Forced, and the reason the multi-step weakness is structural rather than a training gap — though some of the reported weakness is ordinary.
• It cannot answer a question that spans two options. Not forced. This follows from options being isolated, which is one of two available implementations.
• The option order changes the answer. Also not forced, and it is the other implementation's price. No model pays both. These two are not the same limit; they are the two sides of one coin the shape leaves face-down.
• There is a ceiling on how many options you can send. This one dissolves entirely. It is five numbers in five repositories with five unrelated causes, and one of them is the absence of a ceiling.

And a sixth thing, which is the part I think matters most for anyone reading the vendor's own failure-mode page: half the entries on it have nothing to do with the architecture. Literal reading, arithmetic, date ordering, context rot, prompt injection. A prompted 400B model has all of them. Filing them with the structural limits is what makes the shape look worse than it is.

The one choice, stated precisely

A generative model's output space is its vocabulary, sampled repeatedly. A decision model's output space is a list you sent in the request. That is the whole difference, and two families implement it:

• Vocabulary readout. The options go into the prompt as lettered text and the model picks a letter — openjev slices sixteen fixed rows out of a 151,936-wide logit vector. The options share a context.
• Per-option scalar scorer. Each option's text is encoded and scored on its own, and the softmax is taken over the resulting scalars — CUA-S1, open-jev-deberta, and, by the argument in the RLCD piece, Jev. The options never share a context.

There is a third arrangement, which I had not appreciated until I read Laya's code, and it matters later: all options in one sequence with marker tokens, a bidirectional encoder over the lot, and a scalar read from each marker position. Options share a context and each one gets its own scalar. Hold that thought.

The size of the output is worth putting a number on, because it is the source of two of the five limits. A 255-option Choice carries at most $\log_2 255 \approx 7.99$ bits. One call, eight bits, and nothing else comes back — no tokens, no text, no state. A generative model asked the same question emits a hundred tokens from a six-figure vocabulary and can carry a couple of thousand bits, most of it useless, some of it the scratchpad it needs to get to the answer.

Eight bits per call is not a criticism. It is the product: a bounded, calibrated, parseable answer. But you cannot spend eight bits on both the answer and the working.

1 · It cannot write the string

This is the limit TypeSafe states most plainly, on its own jaggedness page, which is a genuinely unusual document — a vendor enumerating eight ways its flagship model fails, with a "do this instead" for each:

What makes this more than a footnote is what it does to a real agent. WindTunnel is nekuda's benchmark of the four ways a browser agent can operate a website, and its board v1.2 added a Jev configuration. The leaderboard is covered separately and published at webmcp.com/benchmark, so I will not relitigate it here. The relevant fact is structural and it is in the repo's own docs/SPEC.md: "Jev (jev-1.13.0) selects actions; Mercury (mercury-2.5) writes arguments or field values and the final answer."

Two models, because one model can pick search_products out of the tool list and cannot produce the query string to put in it.

WindTunnel publishes the redacted per-attempt traces, so the split is measurable rather than rhetorical. I summed the 147 scored WebMCP attempt records:

385 of 413 selected actions could not be executed until a second model wrote their arguments, and 379 of those 385 requests carried at least one string property with no enum and no const — a field with no list to pick from. The 28 decisions that needed no second call were the ones the harness could execute with an empty argument object.

The decision model is genuinely cheap: it read half again as many input tokens as Mercury for under a third of the cost. It is also doing less than half the job. Every argument, every field value, and every one of the 143 final answers was written by the other model.

The version without a second model

The same repository contains the more interesting artifact: the argument filler they built to avoid the second model, and did not use for the measured runs. experiments/jev/frozen/arms/decision-spans.mjs turns generation into a chain of choices, exactly as the jaggedness page says you can and should not.

Read the fourth step again. A final answer is eight Choice questions over the same candidate list, deduplicated and joined with "; ". That is not a sentence. It is a concatenation of spans that already existed somewhere in the prompt or the observations. The whole of it is one function:

// WindTunnel · experiments/jev/frozen/arms/decision-spans.mjs @ 5ca8644 — one audit() call elided
export async function answerFromSpans(state, choose, audit = () => {}) {
  const { candidates, omitted } = collectCandidates(state.task, state.history.map(h => h.result ?? h));
  const values = candidates.filter(c => typeof c.value === "string" || typeof c.value === "number");
  const remaining = values.slice(0, 254); // Reserve one choice for stop.
  const questions = Object.fromEntries(Array.from({ length: 8 }, (_, i) => [`slot${i}`, {
    question: `Select answer evidence for slot ${i + 1} of 8, in task order. Choose a distinct relevant value for each slot, or stop when no more evidence is needed.`,
    options: [{ value: null, label: "Stop selecting answer evidence" }, ...remaining],
  }]));
  const answers = await choose(state, questions), selected = [];
  for (const id of Object.keys(questions)) {
    const value = answers[id].value;
    if (value === null) break;
    if (!selected.includes(value)) selected.push(value);
  }
  if (!selected.length) throw new Error("no answer evidence selected");
  return selected.join("; ").slice(0, 3900);
}

collectCandidates scrapes the task text and every observation for strings, numbers, quoted phrases, dates and contiguous spans of up to six words. The model picks eight of them in order, or says stop. join("; ") is the composition step, and it is a string method. The file's own comment marks where this runs out: "Longer newly composed strings require the hybrid filler."

The ladder is not a hack around the limit. It is the limit, drawn at full size.

Which is also the recipe

The useful half of this is that TypeSafe documents the same pattern as the correct architecture, not as a workaround. Its pre-parsed value extraction cookbook is three steps long:

expand

And the line that turns the limit into a feature:

Those three boxes are also three blocks of a real agent loop. Here is WindTunnel's, with everything but the boundary stripped out:

// WindTunnel · experiments/jev/frozen/arms/decision-agent.mjs @ 5ca8644 — abridged
snapshot = await bounded(() => adapter.observe(), signal);          // 1. enumerate
const menu = [...snapshot.actions, FINISH, ABSTAIN];
if (menu.length > 255) throw new Error('action menu exceeds 255 choices');
if (new Set(menu.map(a => a.id)).size !== menu.length) throw new Error('duplicate or reserved action ID');
const state = decisionState(task, menu, history, snapshot.observation, audit);
 
const selection = await select(state, 'Choose the next action to advance the user task. …',
  menu.map(a => ({ value: a.id, label: `${a.id}: ${a.description ?? ''}` })));   // 2. decide
const id = selection.value;
const action = menu.find(a => a.id === id);
if (!action) throw new Error('selector returned an unoffered action');
 
if (hybrid) args = await request('luna', () => providers.fillLuna(state, publicAction(action)));
else args = await fillFromSpans(action.inputSchema, state, choose, audit);       // 3. compose
assertArgs(action.inputSchema, args);

adapter.observe() lists the tools the page is exposing right now, so the answer set is finite before the model is called at all. select returns an id. menu.find turns that id back into the real object, and the throw on the next line is the invariant: an id that was not offered ends the attempt rather than doing something approximate.

The composition step is a provider, and it is swapped in from outside the loop. The published WebMCP runner is three lines:

// WindTunnel · experiments/jev/webmcp/arm.mjs @ 5ca8644 — reflowed, Jev fetch wrapper elided
const providers = createDecisionProviders({ ...opts, fetchCall: /* Jev, 60s deadline */ });
providers.fillLuna = mercuryProvider({ ...opts, fetchCall, requestTimeoutMs: 60000 });
return providers;

That is why the file says fillLuna while the leaderboard says Mercury: the decision half never learns which generative model is writing its strings. Replacing the composer is an assignment. That property — not the latency, not the price — is what makes this a design rule rather than one integration.

A model that cannot write cannot misspell your customer's account number. Whether that trade is worth it depends entirely on whether the candidates can be enumerated, which is where the difficulty went and where the rest of this article ends up.

2 · It cannot read two options at once

Paras Chopra's fifteen-task benchmark put Jev at 0 of 100 on relational choice — questions that "use information in one option to select another" — on a suite where the same model scores 100% on rule/evidence judgment and 98% on ARC-Challenge. Zero, not low. You cannot fail that reliably by accident.

The derivation is short. If each option is encoded and scored on its own, an option that refers to another option is referring to something that is not in its context. Not down-weighted — absent. That reading is Reasoned, and the 0/100 is the behavioural signature it predicts.

Here is where my one-limit thesis got its first real test, and lost. If option isolation is what causes this, then a model whose options do share a context should be able to answer relational questions. Two such models were measured in the same table:

The prompted Qwen readout, which sees every option in one context window, gets 53%. So sharing a context is not sufficient, but it is the only arrangement in which the question is answerable at all.

Laya is the awkward one, and it is awkward in a way that is worth being precise about. Its DecisionModel in laya/common.py builds one sequence containing every option, runs a bidirectional ModernBERT encoder over it — so every option attends to every other option, in both directions — and then gathers the hidden state at each option's marker position through a shared Linear(d, 1) scorer. The options could not see each other more thoroughly, and it scores 8 of 100.

So: option isolation is sufficient to make relational choice impossible, and shared context is necessary but nowhere near sufficient. A 421M encoder that sees all the options still cannot follow the reference. The honest version of limit 2 is therefore narrower than the version I wanted: a per-option scorer cannot answer a relational question, and nothing else in this family reliably can either.

3 · The option order — the same choice, the other price

Now run the argument backwards. If a model's options share a context, each option sits at a position, and a position is something a prior can attach to.

openjev ships an option_reversal perturbation — the same evidence, the same question, the same options in the opposite order — and commits both its fixtures and its raw per-row predictions with the base_id each perturbed row came from. The CUA-S1 piece joined those and found it changed its mind on 10 of 36 cases, 27.8%, while aggregate accuracy barely twitched.

That is the symptom. The mechanism is measurable from the same files, and as far as I can tell nobody has measured it, so I did.

All 36 pairs are three-option questions, which means reversal swaps slots A and C and leaves B exactly where it was. B is therefore a control. Take the change in B's logit as the global drift of the last-position logits, subtract it, and what remains is how much an option's score moved purely because its text changed slot. Average the two directions and you get one number per question: what slot A is worth, independent of what is written in it.

Slot A is worth +1.71 logits on average, +1.56 at the median, and it is positive on 32 of the 36 questions. Moving an option's text from first to last costs it 1.54 logits; moving it from last to first gains 1.88. In odds terms the first slot multiplies an option's odds by about five and a half, for no reason connected to the option.

The prior does not flip answers on its own — look at where the solid bars are. It flips the questions whose content margin happens to be the same size as the prior. The ten questions that flipped had a median winning margin of 1.81 logits; the twenty-six that held had 4.50. Five of the nine questions with a margin under 2 logits flipped; five of the twenty-seven above it did.

This is a property of the readout, not of the model being small. A and B are different tokens carrying different learned priors, and no amount of scale makes them interchangeable. Two honest caveats: openjev here is an unadapted Qwen3.5-4B, so this is the prior a base model brings rather than one a trained decision model is stuck with, and option-order shuffling during training is the known remedy — Laya says it needs more of it, reporting 0.150 and 0.230 instability at 20 options and calling that "worth fixing".

The number I cannot verify, and it is the important one

If Jev is a per-option scalar scorer, its order instability should be exactly zero. Not small — zero, up to tie-breaking. Reversing the caller's array changes nothing about any individual (state, question, option) triple, so every scalar is bit-identical and the softmax is over the same multiset.

Laya's BENCHMARKS.md reports "Jev measured at 0.13" for option-order instability, with no citation. Laya did not measure it — the file opens by saying it has no TypeSafe API access and that all Jev figures are third-party. I could not find the primary source; it is not in Chopra's gist, which reports no stability figure at all.

If that 0.13 is real, strict per-option isolation is wrong, and the likeliest explanation is the two-stage path: TypeSafe's launch post says high-cardinality Choices run "a 2 stage-system of scoring independently then making an explicit choice", and an explicit choice among survivors is exactly where an order could re-enter. If it is not real, it is a number that has been repeated into the record. Either way it is the single most load-bearing unverified figure in this whole category, and it is one afternoon and an API key away from being settled. Falsifier at the end.

4 · The ceiling that is five ceilings

This is the limit I expected to fold into the thesis and instead had to throw out.

Five numbers, five mechanisms, none of which is "the architecture". A product limit, a recommendation, a training-time weld, a string constant — and, in the family that is supposed to be the most constrained, no ceiling at all. open-jev-deberta's head ends in 2.weight [1, 1024]: one scalar per option, softmax afterwards, nothing in the weights that knows how many options there are.

The openjev row deserves a correction, because the explanation I have seen repeated — and half-believed myself — is that 16 is how many uppercase letters survive as single tokens. It is not. LETTERS = "ABCDEFGHIJKLMNOP" is a 16-character string constant on line 11 of src/semif_phase1/core.py, and the validator twenty lines later reads its length; that is the entire mechanism. Here it is in full:

# openjev · src/semif_phase1/core.py @ b9cb325 — lines 11 and 30-32
LETTERS = "ABCDEFGHIJKLMNOP"
 
def validate_row(row: dict) -> None:
    ...
    options = row["options"]
    if not isinstance(options, list) or not 2 <= len(options) <= len(LETTERS):
        raise ValueError("options must contain 2-16 entries")

len(LETTERS). Not a tokenizer property, not a measured degradation point, not a property of the readout — the length of a string literal, read once at validation time. Add ten characters to it and the cap is 26.

The committed predictions say the same thing from the other side. Every prediction records its answer_token_ids, and across all 814 of them the ids run 32, 33, 34 … 39 for A through H — consecutive, with no gaps, which is what a byte-level vocabulary does with printable ASCII. On that arithmetic P is 47 and Z is 57. Eight letters is as far as the committed rows go, so the rest is inference rather than measurement, but it does not need to be settled: _slot_ids already refuses to run unless every letter it is handed is one exact round-trip token. The guard is there so the constant does not have to be conservative. Sixteen is a choice, and a round one.

So "decision models cap out at N options" is not a claim about decision models. It is a claim about one file.

5 · It cannot plan

TypeSafe documents this itself, twice. Entry 4 on the jaggedness page:

And the closing reminder on the same page lists "System Two tasks: more layers of indirections" among the things to avoid — which is the model's own category name being used as a warning label.

An independent implementer says the same thing without hedging. Vercel Labs' json-render added a Jev composer this week — covered separately — and its Limits section reads: "Root selection, grouping, and deciding when to stop require planning, which is a documented weakness of Jev," linking to the jaggedness page. That is a team shipping the integration writing down, in the README, the thing the integration is bad at.

The measured version is thin but pointed. Chopra's WikiRouter task is Wikipedia navigation with a ten-hop budget: Jev reached the destination on 1 of 120 routes, and "stopped on 118". His prototype managed 4 and Laya 0, and he says himself that these counts do not establish a ranking. They do not. What they establish is that the best performer in every other row of that table falls over on the one task that requires carrying a plan across steps.

The derivation is the eight-bit argument from the top. A forward pass produces one distribution over the caller's options and nothing else. There is no scratchpad, no chain of thought, no place to write down "I am three hops in and my target is Rubber Duck" except the state the caller reassembles on the next call. Multi-step reasoning is not weak in the model; it is absent from the model and located in the harness, which is where the constructive half of this article picks up.

Label: partly derived. Bounded output forbids the scratchpad — that is forced. But "struggles with indirection" also describes plenty of ordinary encoders, and I cannot separate the two from the outside with the evidence available.

The six that do not belong

TypeSafe's jaggedness page has eight entries. Two of them — Generation, Indirection — are the structural ones above. The other six are:

Every one of these describes a prompted LLM equally well. Context rot has its own literature. Prompt injection is the defining unsolved problem of the entire agent field. "It answers what you wrote, not what you meant" is a description of language models in general.

They are on the page because the page is honest, not because they follow from the shape, and reading them as consequences of the architecture is the mistake that makes people conclude the architecture is a dead end. It is worth separating the two, because the structural limits have workarounds that are design rules and the ordinary ones have workarounds that are just prompt engineering.

So: one limit, or several?

One limit, two prices, and a lot of misfiling.

The one limit is real and it is the one worth knowing: the output is an element of the caller's set. It forbids generation, it forbids a scratchpad, and those two are true of every model in this category regardless of how it is built.

The two prices are a genuine fork, and the most useful thing in this article for anyone choosing between these models. The shape does not say how options reach the scorer, and the two answers fail in opposite directions:

• Isolate the options and order becomes inexpressible — you get exact order-invariance for free — but a question spanning two options has no answer to find.
• Share the context and relational questions become at least answerable, but position acquires a prior worth 1.71 logits, and 27.8% of a reversed set changes its mind.

You can find out which family a hosted model is in with one afternoon and a reversed list. That single experiment tells you more about what it will do in production than any accuracy number on its card.

And the misfiling: the cardinality ceilings are implementation constants, and six of the eight documented failure modes are ordinary model weaknesses that happen to be written down by an unusually honest vendor.

The constructive half: the bounded model decides, something else composes

The limits are not a verdict. They are a specification, and this week three unrelated projects shipped the same answer to it.

The rows are the design rule. Somebody enumerates a finite answer set; the model chooses within it; somebody else composes the result. In WindTunnel the page enumerates (every live WebMCP tool, re-listed each step) and Mercury composes. In jev-semgrep, which has its own piece, the file enumerates — thirty lines per request, one Noul per (line, meaning) pair. The enumeration is a nested forEach:

// jev-semgrep · semgrep.mjs @ 21120e9 — the enumerate step, inside evaluate(chunk)
const id = i => `L${String(i).padStart(3, '0')}`;
const state = Object.fromEntries(chunk.map((l, i) => [id(i), l.text.slice(0, 2000)]));
const questions = {};
chunk.forEach((_, i) => meanings.forEach((text, m) => {
  questions[`${id(i)}_${m}`] = { type: 'noul', instructions: `Does line ${id(i)} match the meaning: "${text}"?` };
}));
// … one POST to /v1/systemone, then:
return chunk.map((_, i) => meanings.map((_, m) => answers[`${id(i)}_${m}`].noul));

Thirty lines times however many meanings you passed, each an independent yes/no with its own probability. Nothing is ranked, nothing is retrieved, nothing is embedded. And the composition is one line of JavaScript:

// jev-semgrep · semgrep.mjs @ 21120e9 — the boolean algebra, entirely outside the model
if (!expr.some(term => term.every(([m, not]) =>
      (not ? p[m] < tNeg : p[m] >= tPos)))) continue;

Disjunctive normal form, evaluated in the harness over independent calibrated probabilities. The README states the design principle better than I could: "Because each meaning yields an independent probability, logical AND and NOT are plain boolean operations, not a trick with set differences or negative queries." An embedding cannot do this, because an embedding of a line is fixed before it ever sees your query; a cross-encoder judging a proposition per line can, and then && is just &&.

In json-render the developer enumerates — seventeen component types plus per-request value and binding candidates — and code assembles the tree: "Jev does not author the serialized JSON; code assembles it from the choices." Those are the lines that do the assembling:

// json-render · packages/core/src/experimental-composition-batch.ts @ 3ad3818 — abridged
const answers = await call("select", shared, questions);   // one batched evaluation
if (answers.root!.choice === "unavailable") { yield complete("unavailable"); return; }
const root = candidates.find((c) => c.id === answers.root!.choice)!;
const selected = [root];
// … one membership answer per candidate group appends to `selected`
 
spec = { root: "node_0", elements: {}, state: structuredClone(options.initialState ?? {}) };
const defaultSlot = rootSlots.includes("default") ? "default" : rootSlots[0]!;
selected.forEach((candidate, i) => {
  const id = `node_${i}`;
  spec!.elements[id] = { ...structuredClone(candidate.element), children: [] };
  if (i) attach(spec!, id, { id: spec!.root, slot: defaultSlot });
});
validate(spec);

The model's entire contribution is answers: a map from question id to a chosen string. Every character of JSON that reaches the renderer is written by the block underneath it, out of objects the catalog already held. Note that unavailable is itself a candidate, which is how "nothing offered fits this request" comes back as a choice instead of as a malformed tree.

The correction: nobody converged

I wanted to write that the field arrived at this independently in a week. It did not, and the truth is better. TypeSafe published the rule first, in the shape of a failure-mode page and four cookbooks, and these projects are following documentation:

• Generation: "extract possible options using regex or a generative model and let jev-1.13 pick the correct extraction."
• Counting: "iterate in code over the candidates and ask one question for each, then add up the answers yourself." That is jev-semgrep, described a week early.
• Dates: "Extraction is a judgment, so give it to the model. Arithmetic is not, so keep it in code." Every part of a date is a small closed set, so extraction becomes a Choice.
• Re-ranking: one question per query-candidate pair, and code does the sorting.

expand

A vendor publishing a page titled "jaggedness" that is, read sideways, the architecture guide for its own product is a genuinely good piece of technical writing, and it was published against the immediate commercial interest of making the model sound capable. Credit where it is due.

Where the difficulty went

The rule has a cost, and json-render is the only one of the three that states it plainly:

That is the whole trade in one sentence. You do not remove the open-ended part of the problem by putting a bounded model in the middle of it. You move it to whoever builds the list, and you get, in exchange, a guarantee: the answer is one of the things you already decided you were willing to do. WindTunnel's harness puts it as an invariant — the action menu is rebuilt every step from what is on the page, capped, checked for duplicates, and a returned index can only ever name something the runtime already validated.

So the design rule, stated for the next person who asks why it cannot just do X:

Put the bounded model where the answer set is already finite, and where something else owns the composition. If you find yourself enumerating candidates that a generative model had to invent first, you have not removed the generative model — you have given it a proofreader. That is often exactly what you want, and it is worth being clear-eyed that it is what you bought.