The 245x factors: 15x of interface, 16x of price

Two articles ago I argued, from TypeSafe's docs, that Jev must be a per-option scalar scorer. Then an independent benchmark put it at 0 of 100 on relational choice, which is what that architecture predicts and nothing else does. Both pieces were about a constraint: a model that picks from a caller-supplied option list and cannot emit a string.

WindTunnel is the first place I have seen someone ship a real system into that constraint, measure it against twenty other configurations on the same tasks, and publish everything needed to check the arithmetic. Jev + Mercury 2.5 over WebMCP solves 49 of 49 tasks and is at rank one on a 21-configuration board. The claims that travelled are the cost multipliers: 112x cheaper than GPT-6 Astra on code execution, 245x cheaper than Astra on screenshots.

Both numbers are real. Both are also products of two independent ratios, and the board contains the cells needed to pull them apart. That is what most of this piece is.

The denominator, first

49 tasks, 3 attempts each, 8 self-hosted applications, 600 s per-attempt cap. A task is solved when a majority of its three attempts pass. Scoring is outcome-based: the harness reads the application's own state — is the order in the database, does the appointment exist — so no model judges a run.

The task set is 49 and not 50. tasks/learnhouse.yaml carries a tenth task, lh-4, marked excluded: api-only-data:

# tasks/learnhouse.yaml — max_steps elided
- id: lh-4
  excluded: api-only-data
  tier: answer
  prompt: What tags does the Evaluation Foundations course carry?
  predicate:
    type: answer
    contains:
      - webmcp

Course tags are not rendered on the page. A WebMCP tool can return them; a screenshot agent cannot see them. Dropping it removes a task that WebMCP would have won by construction, and the benchmark's authors dropped it. Worth noticing before any of the rest.

expand

Ten of the 21 configurations solve 49/49 — all nine WebMCP rows plus GPT-6 Astra on code execution. So a 100% task-solve rate separates nothing here. Cost and time do, and that is where the argument lives. Jev + Mercury's attempt-level number is 141/147; the six failures are two abstentions, two stalls and two answers that missed their predicate. None of them hit a budget or a timeout.

A model that cannot write a string

Jev takes state plus typed questions and returns a choice with a probability distribution. It does not generate text. So the action menu has to be the answer space, and under WebMCP the menu is whatever tools the page has registered. From experiments/jev/frozen/arms/decision-agent.mjs:

const menu = [...snapshot.actions, FINISH, ABSTAIN];
if (menu.length > 255) throw new Error('action menu exceeds 255 choices');
const state = decisionState(task, menu, history, snapshot.observation, audit);
const selection = await select(state, 'Choose the next action to advance the user task. …',
  menu.map(a => ({ value: a.id, label: `${a.id}: ${a.description ?? ''}` })));

FINISH and ABSTAIN are synthetic options bolted onto the site's real tools, because "I am done" and "I am stuck" have to be choosable too. The 255 cap is Jev's Choice limit; decisionState caps the serialized state at 48,000 characters and drops history oldest-first to stay under it.

What Jev cannot do is fill in search_products(query=…). That job goes to Mercury 2.5, Inception Labs' diffusion LLM, called at reasoning_effort: 'instant' with the selected tool's JSON Schema forced as the only callable function (experiments/jev/webmcp/mercury.mjs):

const body = { model: 'mercury-2.5', reasoning_effort: 'instant', max_completion_tokens: 4096,
  messages: [{ role: 'system', content: INSTRUCTIONS + … },
             { role: 'user', content: JSON.stringify({ state, selected: { description, name } }) }],
  tools: [{ type: 'function', function: { name: 'submit_arguments', parameters: outboundSchema } }],
  tool_choice: { type: 'function', function: { name: 'submit_arguments' } } };

The system prompt is explicit that the choice is already made: "The action and its target are already chosen. Do not choose a different action." Mercury never selects; Jev never types. One detail that matters for the cost arithmetic later: if the chosen tool's schema has no properties and nothing required, the scaffold skips the Mercury call entirely and passes {}.

This is the two families argument cashed out in a shipping system. Option isolation buys order-invariance and costs you the ability to say anything the caller did not already write down. The fix is not to make the decision model talk — it is to put a talker next to it and give it no decisions.

Here is one real attempt, end to end.

Two things in that trace are worth more than the totals. At step 1 Mercury invents a category that does not exist, and the tool answers with an error and the four valid categories; step 2 uses one. At step 4 the checkout tool refuses and returns the two shipping options; step 5 passes one back. Neither recovery required the agent to be clever. It required the tool to reply with the legal values — which is a property of the code on the website, not of either model.

What est_cost_usd counts

Before any multiplier: the cost column is model tokens at a pinned price table, and nothing else. docs/SPEC.md says "provider list prices, with cached input priced at each provider's cached rate". The table lives in harness/lib.mjs and is serialized into every run.json so a published result can be re-costed later:

// $ per million: input, output, cached read, cache write
export const PRICES = [
  ["typesafe-ai/jev",  [0.042, 0,  0.042, 0.042]],
  ["gpt-6-astra",      [10,    50, 1,     12.5 ]],
  …
];

Mercury is priced separately inside the arm, at $0.20 / $0.02 / $0.75 per million (input / cached / output). Jev goes in at $0.042 per million for every token type, with output at zero — the run records both as standard_prices_per_million, and the zero is at least consistent with the shape of the thing, whose output is a choice id and a probability vector rather than a stream.

Every number on the board is an estimate of this kind: token counts multiplied by a list price, never a provider invoice. The code has a path that uses a gateway's billed figure when one comes back; this run used TypeSafe's API directly, so all 294 Jev rows carry cost_estimated: true.

Nothing else enters the column. Not the browser, not the Docker capsule, not the site, and — this is the one to keep hold of — not a line of the tool code that makes WebMCP work at all.

I recomputed the three GPT-6 Astra medians from the four token columns and that price table. All three reproduce to the cent, so the reported column really is the stated formula:

cu-openai   × gpt-6-astra   reported $0.2607800   recomputed $0.2607800
code-openai × gpt-6-astra   reported $0.1186760   recomputed $0.1186760
wm-gpt      × gpt-6-astra   reported $0.0171000   recomputed $0.0171000

The decomposition

Every headline multiplier compares Jev + Mercury on WebMCP against GPT-6 Astra on something else. Two things change at once. The board contains the cell that holds one of them fixed — Astra has its own WebMCP row — so the product comes apart.

16.07x of that is price, and it is the same term in both headlines. Hold the interface at WebMCP and swap the models: Astra $0.017100, Jev + Mercury $0.001064. What is left — 15.25x against screenshots, 6.94x against code execution — is what WebMCP itself buys with the model held fixed.

The price term does not come from reading less. Jev + Mercury's median attempt processes 9,793 tokens against Astra's 2,575 — 3.8x more — and still costs 16x less, because its blended rate is $0.109 per million tokens against Astra's $7.01. Call it 61x on the medians, 64x on the run totals. The decision model is cheap the way a classifier is cheap.

The interface term does come from reading less, mostly. Astra's screenshot row processes 20,560 median tokens against its own WebMCP row's 2,575 — 8.0x — and the remaining 1.9x is rate mix. That mix has a mechanism worth naming:

A WebMCP prompt is a stable list of tool schemas, so half of it comes back as a cache read at a tenth of the input price. A screenshot prompt carries new image bytes every turn, so the prefix never repeats and essentially nothing caches: one part in a thousand. That is a property of the modality, not of Jev, and it is doing real work in the 245x.

It also makes the headline sensitive to one price-table line. Cache writes are billed at 1.25x input, and they are 93.5% of the Astra computer-use bill. Price them at plain input instead and the same medians give 199x and 93x. The 1.25x is disclosed in a comment above the table; I cannot independently verify it, and it is the single assumption the multiplier leans on hardest.

The cheap fast model is 78% of the bill

The framing everywhere is that argument-writing is the easy part, delegated to something small. That is true about the cognition and false about the invoice.

Across the 147 WebMCP attempts Jev costs $0.0451 and Mercury costs $0.1564. The decision model is 22.4% of the spend. Under the DOM setup the split inverts to 75.7% Jev — and the reason is in the call rates, not the prices. Mercury runs on 0.93 calls per step under WebMCP and 0.25 under DOM. Almost every tool call needs arguments; almost every click needs only an index.

So "cheap decision model plus cheap fast writer" is really "nearly-free decision model plus the actual cost centre", and the lever on a WebMCP agent's bill is the argument writer, not the chooser.

One more thing the transcripts settle. The announcement sells Mercury at "1,000+ tokens/sec", which is a decode rate. End to end in this harness its median call is 614 ms for a median of 40 output tokens — about 65 tok/s wall-clock, because a forty-token JSON object is all prefill and round trip. Jev's median call is 403 ms. Two calls a step, a median of two steps, and the attempt's 3.21 s median agent time is about 63% model latency. The fast model's headline speed mostly does not survive being called this way, and the configuration is still the fastest row on the board.

25/49 to 49/49, and an 18% that is not what it looks like

The second claim is the interesting one: adding WebMCP nearly doubled solved tasks and reduced model cost by 18%. Doubling accuracy while spending less is the kind of result that deserves a second look.

It checks out, and it is not a per-attempt saving. Per median attempt, WebMCP is 37% more expensive — $0.001064 against $0.000778 — and the README says so plainly. The 18% is a total across 147 attempts each, straight out of the release's own PROVENANCE.md: $0.201557376 against $0.245387386, which is 17.9% lower.

The difference is failure. Attempts that fail grind through their step budget:

Seventy percent of the page setup's bill buys nothing. Divide by what each actually delivered and the comparison stops being counterintuitive: $0.0041 per solved task against $0.0098 — a factor of 2.4, not 1.18. Per passing attempt it is 2.3.

Two caveats in the other direction, both from the release's own disclosures. It publishes an unknown-usage reserve — requests whose token counts never came back — excluded from the leaderboard medians: $0.0144 for WebMCP against $0.0032 for DOM. Fold both reserves and the non-scored requests back in and the 18% becomes 12.9%. And the DOM arm's failure taxonomy is not random noise:

36  agent blocked            (Jev chose the BLOCKED operation)
19  agent budget exhausted
 8  Text helper returned no valid field value; nothing typed
 6  predicate failed        (3 state, 3 answer)
 1  invalid TypeSafe response; no action executed
 1  Mercury content-filter refusal

Half of the failures are the agent deciding, explicitly, that no supported operation makes progress. It is not flailing. It is giving up — and it was given the larger step budget of the two (20 against 12 by default, 32 against 15 on the checkout task).

The compression claim, measured

The authors' hypothesis for why the page setup underperforms is that WebMCP "compresses a sequence of clicks into one tool call", shrinking the decision space. TypeSafe's own documentation points the same way without using the word — it tells you to keep each question to "the kind of judgment a highly knowledgeable person could make in a few seconds" and to decompose anything that "would require extended reasoning or weighs multiple independent factors". A journey through a checkout flow is exactly the thing it says not to ask in one question.

Split by journey length, the two curves differ in slope, not offset:

On one- and two-step answers the page setup is 2.4x cheaper and solves 13 of 21. By the four checkout-shaped tasks it is 2.0x dearer and solves 1 of 4. Across the four tiers WebMCP's median attempt cost grows 4.0x and the page setup's grows 19.5x. The median tool calls per attempt go 1 → 2 → 3 → 4; the median page actions go 1 → 3 → 8 → 22.

That is the compression claim with a number on it, and it also explains the aggregate: 37 of the 49 tasks are in the two short tiers where the page setup is genuinely cheaper per attempt, which is why WebMCP's median is higher and its total is lower.

expand

What the website pays

The cost column stops at the model. The work does not.

Getting these eight applications to expose tools took 7,302 added lines across 44 tools, committed as one reference patch per site in goldens/:

About 166 lines per tool, and the good ones are not thin wrappers. md-8's whole journey collapses into one call because somebody wrote the collapse:

// goldens/nextjs-starter-medusa.reference.patch — schema body elided
{
  name: "complete_checkout",
  description:
    "Complete guest checkout for the current cart in one step: sets the contact email and " +
    "shipping/billing address, picks the shipping option and payment provider (automatically " +
    "when the store offers exactly one of each, otherwise returns the choices so you can pass " +
    "shipping_option_id / payment_provider_id), places the order, and returns the order " +
    "confirmation (order id, display number, total, items). Use after add_to_cart; " +
    "begin_checkout is not required.",
  inputSchema: { /* 13 properties, 7 required */ },
  annotations: { readOnlyHint: false },
  async execute(args) { … }
}

Read the description again. "Otherwise returns the choices so you can pass shipping_option_id" is the step-4 error in the trace above, designed in advance. That is a tool written by someone who knew a model would be calling it.

And that tool exists because WebMCP could not finish without it. The v1.1 changelog is unusually direct about it: before September 6, all eight WebMCP configurations scored 0/3 on md-8 because the store's tools handed guest checkout back to the page. Adding complete_checkout took all eight to 3/3, 24 attempts for $0.93. The entry calls the old state "a limitation of one demo store's tool surface, not of the approach", which is fair — and the consequence is still that the tool surface under test is co-designed with the benchmark. 49/49 is a score for WebMCP-as-implemented-by-its-advocates, against a standard of tool design that almost no site in the wild currently meets.

That is not a knock on the result. It is the correct reading of it: the number measures the interface and the quality of the tools behind it, and here those were built by people who wanted the interface to win.

The 49th task rests on a 79-millisecond wait

The Jev WebMCP arm's tool_version is decision-scaffold-v5-mercury-readiness-v1, and the readiness half is a harness behaviour no other WebMCP configuration has. Every arm shares prepareWebMCPPage, which waits up to 30 s for the site's first registerTool call — hydration is slow and that is fair to everyone. What this arm adds is a wait on every subsequent observation: if the live tool list comes back empty mid-run, poll every 25 ms for up to a second before deciding anything.

It is disclosed, and better than disclosed — the run audits every observation. Across 147 attempts and 413 observations it fired four times, for 275 ms total:

ea-7   step 5   0 → 4 tools   28 ms   ready-after-wait   failed anyway
hev-8  step 2   0 → 1 tool    85 ms   ready-after-wait   pass
hev-8  step 2   0 → 1 tool    83 ms   ready-after-wait   pass
hev-8  step 2   0 → 1 tool    79 ms   ready-after-wait   pass

Three of the four are the same task, all three at the same step, and all three passed. hev-8 is "Log in to the organizer dashboard … and report the current status of the WebMCP Community Workshop event" — three steps, two tool calls, and the empty list lands on step 2, immediately after the login redirect. Hi.Events is also the application the shared arm's own comment singles out: "Heavy SSR apps (hi-events) hydrate well past 5s".

This next part is reasoned, not measured. Without that poll I think hev-8 is 0/3 and the headline is 48/49. The shared WebMCP arms throw no live WebMCP tools registered the moment a mid-run list comes back empty. This scaffold would not throw; it would assemble a menu of exactly two options, __finish__ and __abstain__, and ask Jev to choose between reporting a status it has not read and giving up. Both are scored failures, and the task's predicate rejects the hedge explicitly — not_contains: ["could not determine", …].

It is the same race that cost the Medusa store a task: the golden's own comment says complete_checkout was moved into the always-on group because registering it with the cart-conditioned one "left a gap during the begin_checkout navigation in which an agent's next call found no such tool." Tools that come and go with the route are a real property of real single-page applications, and the 1 s poll is the right engineering answer. It is also an advantage one configuration has and eight do not.

Credit, which this release has earned

Five things here were published against their author's interest.

• The weakest baseline is theirs. "Our modified Ultrafast setup solved 25/49 tasks — that is a result for our particular implementation and benchmark, not a universal limit on Jev or Browser Use." Followed by an invitation to send patches. That is a benchmark author publishing the least flattering version of their own comparison and then offering to have it beaten.
• The discount was not taken. Mercury's arm records a mercury_promotion_multiplier: 0.2 and a promotional_est_cost_usd on every call, and the leaderboard uses the full standard price. Applying the promotion would have made the headline roughly 2.6x larger.
• The reserve is published. Requests with unusable usage data are counted, named and excluded — which is what let me put the 18% at 12.9% two sections ago. Most releases round that to zero silently.
• The rescore went one way and they said so. Independent review found the canonical merge predated a predicate fix; re-scoring produced "19 promotions and 0 demotions", and the corrected rows are flagged in the CSV rather than quietly replaced.
• The free win was dropped. lh-4 asks for data that exists only behind the API. A tool returns it; a screenshot cannot see it. It sits in the task file marked excluded, which is why the denominator is 49.

It reproduces

The benchmark is Apache-2.0 and the Jev release ships its runner, its source hashes and 294 redacted attempt transcripts. Everything below needs npm ci, a Chromium and a Python 3.12 venv — and no API key, no Docker and no money. All of it passed on first go:

$ python3 scripts/verify-jev-release.py
PASS: 294 matching attempts, 14044 transcript events, 30 source hashes;
      redaction regression checks and artifact scans.
 
$ node experiments/jev/run.mjs --interface webmcp
{"interface":"webmcp","tasks":49,"attempts":147,"mode":"check-only"}
$ node experiments/jev/run.mjs --interface dom
{"interface":"dom","tasks":49,"attempts":147,"mode":"check-only"}
 
$ node experiments/jev/webmcp/fixture-check.mjs
PASS: portable WebMCP method selects, fills, executes and finishes with both
      measured providers; no predicate leakage.
$ node experiments/jev/webmcp/signal-seam-check.mjs
PASS: actual providerFactory signal seam caps remaining attempt; … (2 real Chromium scenarios).
$ node experiments/jev/ultrafast/fixture-check.mjs
PASS: upstream policy bridge, real Chromium, accounting, timeout and fail-closed provider checks

Without --run the launcher re-hashes all 30 frozen runner files and all 49 task definitions and refuses to start if either moved. The fixture checks stand up a real Chromium, register a WebMCP tool on a local page, and drive the actual select-then-fill loop with scripted provider responses — so the code path in the article is the code path that ran, not a description of it. 294 and 14044 are 147 + 147 attempts and 3782 + 10262 transcript events.

I did not spend a dollar to check a single number in this piece. That should be unremarkable and currently is not.

The number that survives

The headline that survives all of this is not 245x. It is that a model which cannot produce text, paired with a small one that can, is the cheapest and fastest row on a 21-configuration board — and that most of the margin is the interface, available to any model that uses it. Astra over WebMCP also solves 49/49, at 3.8x fewer tokens than the Jev pair. The decision model wins on price per token, not on economy of reading.