# The 245x factors: 15x of interface, 16x of price > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/webmcp-windtunnel > date: 2026-09-19 > tags: explainer, agents, llm, benchmarks, architecture Two articles ago I argued, from TypeSafe's docs, that Jev must be a per-option scalar scorer. Then an independent benchmark put it at [0 of 100 on relational choice](/articles/jev-scores-zero), which is what that architecture predicts and nothing else does. Both pieces were about a constraint: **a model that picks from a caller-supplied option list and cannot emit a string.** [WindTunnel](https://github.com/nekuda-ai/WindTunnel) is the first place I have seen someone ship a real system into that constraint, measure it against twenty other configurations on the same tasks, and publish everything needed to check the arithmetic. Jev + Mercury 2.5 over WebMCP solves **49 of 49 tasks** and is at rank one on a 21-configuration board. The claims that travelled are the cost multipliers: **112x** cheaper than GPT-6 Astra on code execution, **245x** cheaper than Astra on screenshots. Both numbers are real. Both are also products of two independent ratios, and the board contains the cells needed to pull them apart. That is what most of this piece is. ## The denominator, first **49 tasks, 3 attempts each, 8 self-hosted applications, 600 s per-attempt cap.** A task is *solved* when a majority of its three attempts pass. Scoring is outcome-based: the harness reads the application's own state — is the order in the database, does the appointment exist — so no model judges a run. The task set is 49 and not 50. `tasks/learnhouse.yaml` carries a tenth task, `lh-4`, marked `excluded: api-only-data`: ```yaml # tasks/learnhouse.yaml — max_steps elided - id: lh-4 excluded: api-only-data tier: answer prompt: What tags does the Evaluation Foundations course carry? predicate: type: answer contains: - webmcp ``` Course tags are not rendered on the page. A WebMCP tool can return them; a screenshot agent cannot see them. Dropping it removes a task that WebMCP would have won by construction, and the benchmark's authors dropped it. Worth noticing before any of the rest.
**Ten of the 21 configurations solve 49/49** — all nine WebMCP rows plus GPT-6 Astra on code execution. So a 100% task-solve rate separates nothing here. Cost and time do, and that is where the argument lives. Jev + Mercury's attempt-level number is **141/147**; the six failures are two abstentions, two stalls and two answers that missed their predicate. None of them hit a budget or a timeout. ## A model that cannot write a string Jev takes state plus typed questions and returns a choice with a probability distribution. It does not generate text. So the action menu has to *be* the answer space, and under WebMCP the menu is whatever tools the page has registered. From `experiments/jev/frozen/arms/decision-agent.mjs`: ```js const menu = [...snapshot.actions, FINISH, ABSTAIN]; if (menu.length > 255) throw new Error('action menu exceeds 255 choices'); const state = decisionState(task, menu, history, snapshot.observation, audit); const selection = await select(state, 'Choose the next action to advance the user task. …', menu.map(a => ({ value: a.id, label: `${a.id}: ${a.description ?? ''}` }))); ``` `FINISH` and `ABSTAIN` are synthetic options bolted onto the site's real tools, because "I am done" and "I am stuck" have to be choosable too. The 255 cap is Jev's Choice limit; `decisionState` caps the serialized state at 48,000 characters and drops history oldest-first to stay under it. What Jev cannot do is fill in `search_products(query=…)`. That job goes to **Mercury 2.5**, Inception Labs' diffusion LLM, called at `reasoning_effort: 'instant'` with the selected tool's JSON Schema forced as the only callable function (`experiments/jev/webmcp/mercury.mjs`): ```js const body = { model: 'mercury-2.5', reasoning_effort: 'instant', max_completion_tokens: 4096, messages: [{ role: 'system', content: INSTRUCTIONS + … }, { role: 'user', content: JSON.stringify({ state, selected: { description, name } }) }], tools: [{ type: 'function', function: { name: 'submit_arguments', parameters: outboundSchema } }], tool_choice: { type: 'function', function: { name: 'submit_arguments' } } }; ``` The system prompt is explicit that the choice is already made: *"The action and its target are already chosen. Do not choose a different action."* Mercury never selects; Jev never types. One detail that matters for the cost arithmetic later: if the chosen tool's schema has no properties and nothing required, the scaffold skips the Mercury call entirely and passes `{}`. This is the [two families](/articles/cua-s1-forms#two-families-and-why-the-option-order-is-the-tell) argument cashed out in a shipping system. Option isolation buys order-invariance and costs you the ability to say anything the caller did not already write down. The fix is not to make the decision model talk — it is to put a talker next to it and give it no decisions. Here is one real attempt, end to end. Two things in that trace are worth more than the totals. At step 1 Mercury invents a category that does not exist, and the tool answers with an error *and the four valid categories*; step 2 uses one. At step 4 the checkout tool refuses and returns the two shipping options; step 5 passes one back. Neither recovery required the agent to be clever. It required the tool to reply with the legal values — which is a property of the code on the website, not of either model. ## What `est_cost_usd` counts Before any multiplier: the cost column is **model tokens at a pinned price table, and nothing else.** `docs/SPEC.md` says "provider list prices, with cached input priced at each provider's cached rate". The table lives in `harness/lib.mjs` and is serialized into every `run.json` so a published result can be re-costed later: ```js // $ per million: input, output, cached read, cache write export const PRICES = [ ["typesafe-ai/jev", [0.042, 0, 0.042, 0.042]], ["gpt-6-astra", [10, 50, 1, 12.5 ]], … ]; ``` Mercury is priced separately inside the arm, at `$0.20 / $0.02 / $0.75` per million (input / cached / output). Jev goes in at **\$0.042 per million for every token type, with output at zero** — the run records both as `standard_prices_per_million`, and the zero is at least consistent with the shape of the thing, whose output is a choice id and a probability vector rather than a stream. Every number on the board is an estimate of this kind: token counts multiplied by a list price, never a provider invoice. The code has a path that uses a gateway's billed figure when one comes back; this run used TypeSafe's API directly, so all 294 Jev rows carry `cost_estimated: true`. Nothing else enters the column. Not the browser, not the Docker capsule, not the site, and — this is the one to keep hold of — not a line of the tool code that makes WebMCP work at all. I recomputed the three GPT-6 Astra medians from the four token columns and that price table. All three reproduce to the cent, so the reported column really is the stated formula: ``` cu-openai × gpt-6-astra reported $0.2607800 recomputed $0.2607800 code-openai × gpt-6-astra reported $0.1186760 recomputed $0.1186760 wm-gpt × gpt-6-astra reported $0.0171000 recomputed $0.0171000 ``` ## The decomposition Every headline multiplier compares Jev + Mercury *on WebMCP* against GPT-6 Astra *on something else*. Two things change at once. The board contains the cell that holds one of them fixed — Astra has its own WebMCP row — so the product comes apart. **16.07x of that is price, and it is the same term in both headlines.** Hold the interface at WebMCP and swap the models: Astra \$0.017100, Jev + Mercury \$0.001064. What is left — 15.25x against screenshots, 6.94x against code execution — is what WebMCP itself buys with the model held fixed. The price term does not come from reading less. Jev + Mercury's median attempt processes **9,793 tokens against Astra's 2,575** — 3.8x *more* — and still costs 16x less, because its blended rate is \$0.109 per million tokens against Astra's \$7.01. Call it 61x on the medians, 64x on the run totals. The decision model is cheap the way a classifier is cheap. The interface term does come from reading less, mostly. Astra's screenshot row processes 20,560 median tokens against its own WebMCP row's 2,575 — **8.0x** — and the remaining 1.9x is rate mix. That mix has a mechanism worth naming: | Astra configuration | uncached input | cache reads | cache writes | output | |---|---:|---:|---:|---:| | WebMCP | 26.8% | **51.4%** | 18.9% | 2.9% | | code execution | 1.4% | 19.6% | 77.3% | 1.7% | | computer use | 2.6% | **0.1%** | 96.1% | 1.2% | A WebMCP prompt is a stable list of tool schemas, so half of it comes back as a cache read at a tenth of the input price. A screenshot prompt carries new image bytes every turn, so the prefix never repeats and **essentially nothing caches**: one part in a thousand. That is a property of the modality, not of Jev, and it is doing real work in the 245x. It also makes the headline sensitive to one price-table line. Cache writes are billed at 1.25x input, and they are **93.5%** of the Astra computer-use bill. Price them at plain input instead and the same medians give **199x** and **93x**. The 1.25x is disclosed in a comment above the table; I cannot independently verify it, and it is the single assumption the multiplier leans on hardest. The honest range, rather than one number. Against the **eleven** screen- and page-driving configurations on the board, Jev + Mercury over WebMCP is **16.3x to 245.1x** cheaper per attempt, median **58.9x**. 245x is the top of that range — the priciest model in the field on its heaviest interface. The twelfth non-WebMCP row is the same Jev pair driving the page, and it is *cheaper* per attempt, at 0.73x. Against the best conventional WebMCP row, GPT-5.6 Luna at 49/49 and 146/147, the gap is **2.3x**. **Receipts.** Jev + Mercury 2.5 over WebMCP is 16.3x to 245.1x cheaper per attempt than the eleven screen- and page-driving configurations on WindTunnel's canonical board; the median of that range is 58.9x. The published 245x is the top of the range - GPT-6 Astra, the priciest model on the board, on screenshot computer use. The cheapest screen-driving configuration, GPT-5.6 Luna on computer use, is 16.3x. | configuration | interface | solved | median $/attempt | median tokens | blended $/Mtok | x Jev+Mercury | | :--- | :--- | ---: | ---: | ---: | ---: | ---: | | Jev + Mercury 2.5 | DOM (ultrafast) | 25/49 | $0.000778 | 13,892 | $0.050 | 0.7x | | Jev + Mercury 2.5 | WebMCP | 49/49 | $0.001064 | 9,793 | $0.109 | — | | GPT-5.6 Luna | WebMCP | 49/49 | $0.002436 | 2,596 | $0.846 | 2.3x | | Gemini 3.6 Flash · Stagehand v4 | WebMCP | 49/49 | $0.004372 | 4,371 | $0.997 | 4.1x | | Gemini 3.6 Flash | WebMCP | 49/49 | $0.004396 | 4,453 | $1.002 | 4.1x | | Sonnet 5 | WebMCP | 49/49 | $0.009132 | 5,172 | $1.716 | 8.6x | | Sonnet 5 · Stagehand v4 | WebMCP | 49/49 | $0.009511 | 5,161 | $1.732 | 8.9x | | GPT-5.6 SOL | WebMCP | 49/49 | $0.012100 | 2,573 | $4.239 | 11.4x | | Claude Opus 5 | WebMCP | 49/49 | $0.014013 | 4,770 | $3.472 | 13.2x | | GPT-6 Astra | WebMCP | 49/49 | $0.017100 | 2,575 | $7.006 | 16.1x | | GPT-5.6 Luna | computer use | 45/49 | $0.017370 | 20,914 | $0.977 | 16.3x | | GPT-5.6 Luna | a11y tree | 40/49 | $0.019538 | 18,517 | $1.050 | 18.4x | | Gemini 3.6 Flash | computer use | 43/49 | $0.020155 | 23,857 | $0.801 | 18.9x | | GPT-5.6 Luna | DOM + vision | 43/49 | $0.032825 | 29,561 | $1.102 | 30.9x | | Sonnet 5 | a11y tree | 42/49 | $0.037782 | 10,762 | $3.255 | 35.5x | | GPT-5.6 SOL | computer use | 46/49 | $0.062693 | 16,235 | $4.920 | 58.9x | | Sonnet 5 | computer use | 39/49 | $0.069705 | 57,701 | $1.365 | 65.5x | | GPT-6 Astra | code execution | 49/49 | $0.118676 | 10,982 | $10.826 | 111.5x | | Claude Opus 5 | computer use | 45/49 | $0.138857 | 47,141 | $2.911 | 130.5x | | Sonnet 5 | DOM + vision | 48/49 | $0.210207 | 64,424 | $3.326 | 197.6x | | GPT-6 Astra | computer use | 45/49 | $0.260780 | 20,560 | $12.855 | 245.1x | est_cost_usd counts model tokens at pinned list prices and nothing else - no browser, no Docker capsule, no site hosting, and none of the engineering that puts the tools on the page. Jev is priced at $0.042/M for every token type with output free; Mercury at $0.20/$0.02/$0.75 per M (input/cached/output). A 5x Mercury promotional discount was recorded and deliberately not applied. > method: Ratio of medians of est_cost_usd over each configuration's 147 attempts, taken from results/canonical/results.csv at commit 5ca8644. 'Blended $/Mtok' is that configuration's total est_cost_usd divided by its total processed tokens (uncached input + cache reads + cache writes + output). Solved is majority-of-three, as the board reports it. I recomputed est_cost_usd from the four token columns and harness/lib.mjs PRICES for all three GPT-6 Astra rows; all three medians reproduce to the cent. > source: https://github.com/nekuda-ai/WindTunnel > captured: 2026-09-19 > data: https://ai.thesatyajit.com/articles/webmcp-windtunnel/data/cost-ledger.json (21 rows) ## The cheap fast model is 78% of the bill The framing everywhere is that argument-writing is the easy part, delegated to something small. That is true about the cognition and false about the invoice. **Receipts.** The decision model is not where the money goes. Across the 147 WebMCP attempts, Jev costs $0.045 and Mercury costs $0.156 - the argument writer is 77.6% of the bill. Under the DOM setup the split inverts to 75.7% Jev, because page control mostly means picking an element and only a quarter of steps need any text written at all. | setup | model | role | calls | / step | tokens | cost | share | $/Mtok | | :--- | :--- | :--- | ---: | ---: | ---: | ---: | ---: | ---: | | WebMCP | Jev 1.13.0 | picks the action | 413 | 1.00 | 1,104,774 | $0.045111 | 22.4% | $0.041 | | WebMCP | Mercury 2.5 | writes the arguments | 385 | 0.93 | 737,101 | $0.156447 | 77.6% | $0.212 | | WebMCP | both | | 798 | 1.93 | 1,841,875 | $0.201557 | 100.0% | $0.109 | | DOM (ultrafast) | Jev 1.13.0 | picks the action | 1,097 | 1.00 | 4,721,064 | $0.185858 | 75.7% | $0.039 | | DOM (ultrafast) | Mercury 2.5 | writes the arguments | 278 | 0.25 | 229,003 | $0.059530 | 24.3% | $0.260 | | DOM (ultrafast) | both | | 1,375 | 1.25 | 4,950,067 | $0.245387 | 100.0% | $0.050 | Jev is billed at $0.042 per million tokens of every kind, with output free, so its effective rate barely moves. Mercury is $0.20 input / $0.02 cached / $0.75 output. Neither rate includes the 5x Mercury promotional discount that the run recorded as promotional_est_cost_usd and left unapplied. > method: Summed provider_usage.{jev,mercury}.{calls,cost,usage} over every record in results/2026-09-18-jev-mercury/webmcp-traces.jsonl.gz and dom-traces.jsonl.gz (147 attempts each). 'per step' divides calls by the total number of agent steps in the same transcripts. The 'both' cost rows reproduce PROVENANCE.md's scored-attempt totals of $0.201557376 and $0.245387386 exactly. > source: https://github.com/nekuda-ai/WindTunnel/blob/main/results/2026-09-18-jev-mercury/PROVENANCE.md > captured: 2026-09-19 > data: https://ai.thesatyajit.com/articles/webmcp-windtunnel/data/provider-split.json (6 rows) Across the 147 WebMCP attempts Jev costs **\$0.0451** and Mercury costs **\$0.1564**. The decision model is 22.4% of the spend. Under the DOM setup the split inverts to 75.7% Jev — and the reason is in the call rates, not the prices. Mercury runs on **0.93 calls per step under WebMCP and 0.25 under DOM**. Almost every tool call needs arguments; almost every click needs only an index. So "cheap decision model plus cheap fast writer" is really "nearly-free decision model plus the actual cost centre", and the lever on a WebMCP agent's bill is the argument writer, not the chooser. One more thing the transcripts settle. The announcement sells Mercury at "1,000+ tokens/sec", which is a decode rate. End to end in this harness its **median call is 614 ms** for a median of **40 output tokens** — about 65 tok/s wall-clock, because a forty-token JSON object is all prefill and round trip. Jev's median call is 403 ms. Two calls a step, a median of two steps, and the attempt's 3.21 s median agent time is about 63% model latency. The fast model's headline speed mostly does not survive being called this way, and the configuration is still the fastest row on the board. ## 25/49 to 49/49, and an 18% that is not what it looks like The second claim is the interesting one: adding WebMCP nearly doubled solved tasks *and* reduced model cost by 18%. Doubling accuracy while spending less is the kind of result that deserves a second look. It checks out, and it is not a per-attempt saving. **Per median attempt, WebMCP is 37% more expensive** — \$0.001064 against \$0.000778 — and the README says so plainly. The 18% is a total across 147 attempts each, straight out of the release's own `PROVENANCE.md`: **\$0.201557376 against \$0.245387386**, which is 17.9% lower. The difference is failure. Attempts that fail grind through their step budget: | setup | solved | spend on passing attempts | spend on failing attempts | failures' share | |---|---:|---:|---:|---:| | WebMCP | 49/49 | \$0.187752 (141) | \$0.013805 (6) | **6.8%** | | DOM (ultrafast) | 25/49 | \$0.073198 (76) | \$0.172189 (71) | **70.2%** | Seventy percent of the page setup's bill buys nothing. Divide by what each actually delivered and the comparison stops being counterintuitive: **\$0.0041 per solved task against \$0.0098** — a factor of 2.4, not 1.18. Per passing attempt it is 2.3. Two caveats in the other direction, both from the release's own disclosures. It publishes an **unknown-usage reserve** — requests whose token counts never came back — excluded from the leaderboard medians: \$0.0144 for WebMCP against \$0.0032 for DOM. Fold both reserves and the non-scored requests back in and the 18% becomes **12.9%**. And the DOM arm's failure taxonomy is not random noise: ``` 36 agent blocked (Jev chose the BLOCKED operation) 19 agent budget exhausted 8 Text helper returned no valid field value; nothing typed 6 predicate failed (3 state, 3 answer) 1 invalid TypeSafe response; no action executed 1 Mercury content-filter refusal ``` Half of the failures are the agent deciding, explicitly, that no supported operation makes progress. It is not flailing. It is giving up — and it was given the *larger* step budget of the two (20 against 12 by default, 32 against 15 on the checkout task). ## The compression claim, measured The authors' hypothesis for why the page setup underperforms is that WebMCP *"compresses a sequence of clicks into one tool call"*, shrinking the decision space. TypeSafe's own documentation points the same way without using the word — it tells you to keep each question to *"the kind of judgment a highly knowledgeable person could make in a few seconds"* and to decompose anything that *"would require extended reasoning or weighs multiple independent factors"*. A journey through a checkout flow is exactly the thing it says not to ask in one question. Split by journey length, the two curves differ in slope, not offset: On one- and two-step answers the page setup is **2.4x cheaper** and solves 13 of 21. By the four checkout-shaped tasks it is **2.0x dearer** and solves 1 of 4. Across the four tiers WebMCP's median attempt cost grows **4.0x** and the page setup's grows **19.5x**. The median tool calls per attempt go 1 → 2 → 3 → 4; the median page actions go 1 → 3 → 8 → **22**. That is the compression claim with a number on it, and it also explains the aggregate: 37 of the 49 tasks are in the two short tiers where the page setup is genuinely cheaper per attempt, which is why WebMCP's median is higher and its total is lower.
This is not a clean interface ablation, and the release says so: *"These are complete setups with different harnesses, not an isolated interface ablation."* The WebMCP arm is nekuda's own decision scaffold; the page arm is Browser Use's Ultrafast, modified. The frozen scaffold **has** a page mode — `runPageHybrid` is right there in `decision-agent.mjs` — so the tight ablation was available and was not the thing measured. Measuring against a real shipped agent is the more useful comparison and the less clean one. Both of those are true at once. ## What the website pays The cost column stops at the model. The work does not. Getting these eight applications to expose tools took **7,302 added lines across 44 tools**, committed as one reference patch per site in `goldens/`: | site | tools | added lines | |---|---:|---:| | nextjs-starter-medusa | 8 | 1,352 | | hi-events | 7 | 1,418 | | easyappointments | 7 | 1,074 | | idurar-erp-crm | 7 | 951 | | learnhouse | 6 | 874 | | bulletproof-react | 5 | 620 | | directory-9d8 | 3 | 593 | | tailwind-nextjs-blog | 1 | 420 | About 166 lines per tool, and the good ones are not thin wrappers. `md-8`'s whole journey collapses into one call because somebody wrote the collapse: ```ts // goldens/nextjs-starter-medusa.reference.patch — schema body elided { name: "complete_checkout", description: "Complete guest checkout for the current cart in one step: sets the contact email and " + "shipping/billing address, picks the shipping option and payment provider (automatically " + "when the store offers exactly one of each, otherwise returns the choices so you can pass " + "shipping_option_id / payment_provider_id), places the order, and returns the order " + "confirmation (order id, display number, total, items). Use after add_to_cart; " + "begin_checkout is not required.", inputSchema: { /* 13 properties, 7 required */ }, annotations: { readOnlyHint: false }, async execute(args) { … } } ``` Read the description again. *"Otherwise returns the choices so you can pass shipping_option_id"* is the step-4 error in the trace above, designed in advance. That is a tool written by someone who knew a model would be calling it. And that tool exists because WebMCP could not finish without it. The v1.1 changelog is unusually direct about it: before September 6, all eight WebMCP configurations scored **0/3** on `md-8` because the store's tools handed guest checkout back to the page. Adding `complete_checkout` took all eight to **3/3**, 24 attempts for \$0.93. The entry calls the old state "a limitation of one demo store's tool surface, not of the approach", which is fair — and the consequence is still that **the tool surface under test is co-designed with the benchmark**. 49/49 is a score for WebMCP-as-implemented-by-its-advocates, against a standard of tool design that almost no site in the wild currently meets. That is not a knock on the result. It is the correct reading of it: the number measures the interface *and* the quality of the tools behind it, and here those were built by people who wanted the interface to win. ## The 49th task rests on a 79-millisecond wait The Jev WebMCP arm's `tool_version` is `decision-scaffold-v5-mercury-readiness-v1`, and the readiness half is a harness behaviour no other WebMCP configuration has. Every arm shares `prepareWebMCPPage`, which waits up to 30 s for the site's first `registerTool` call — hydration is slow and that is fair to everyone. What this arm adds is a wait on every *subsequent* observation: if the live tool list comes back empty mid-run, poll every 25 ms for up to a second before deciding anything. It is disclosed, and better than disclosed — the run audits every observation. Across 147 attempts and 413 observations it fired **four times**, for 275 ms total: ``` ea-7 step 5 0 → 4 tools 28 ms ready-after-wait failed anyway hev-8 step 2 0 → 1 tool 85 ms ready-after-wait pass hev-8 step 2 0 → 1 tool 83 ms ready-after-wait pass hev-8 step 2 0 → 1 tool 79 ms ready-after-wait pass ``` Three of the four are the same task, all three at the same step, and all three passed. `hev-8` is *"Log in to the organizer dashboard … and report the current status of the WebMCP Community Workshop event"* — three steps, two tool calls, and the empty list lands on step 2, immediately after the login redirect. Hi.Events is also the application the shared arm's own comment singles out: *"Heavy SSR apps (hi-events) hydrate well past 5s"*. **This next part is reasoned, not measured.** Without that poll I think `hev-8` is 0/3 and the headline is 48/49. The shared WebMCP arms throw `no live WebMCP tools registered` the moment a mid-run list comes back empty. This scaffold would not throw; it would assemble a menu of exactly two options, `__finish__` and `__abstain__`, and ask Jev to choose between reporting a status it has not read and giving up. Both are scored failures, and the task's predicate rejects the hedge explicitly — `not_contains: ["could not determine", …]`. It is the same race that cost the Medusa store a task: the golden's own comment says `complete_checkout` was moved into the always-on group because registering it with the cart-conditioned one *"left a gap during the begin_checkout navigation in which an agent's next call found no such tool."* Tools that come and go with the route are a real property of real single-page applications, and the 1 s poll is the right engineering answer. It is also an advantage one configuration has and eight do not. ## Credit, which this release has earned Five things here were published against their author's interest. - **The weakest baseline is theirs.** *"Our modified Ultrafast setup solved 25/49 tasks — that is a result for our particular implementation and benchmark, not a universal limit on Jev or Browser Use."* Followed by an invitation to send patches. That is a benchmark author publishing the least flattering version of their own comparison and then offering to have it beaten. - **The discount was not taken.** Mercury's arm records a `mercury_promotion_multiplier: 0.2` and a `promotional_est_cost_usd` on every call, and the leaderboard uses the full standard price. Applying the promotion would have made the headline roughly 2.6x larger. - **The reserve is published.** Requests with unusable usage data are counted, named and excluded — which is what let me put the 18% at 12.9% two sections ago. Most releases round that to zero silently. - **The rescore went one way and they said so.** Independent review found the canonical merge predated a predicate fix; re-scoring produced *"19 promotions and 0 demotions"*, and the corrected rows are flagged in the CSV rather than quietly replaced. - **The free win was dropped.** `lh-4` asks for data that exists only behind the API. A tool returns it; a screenshot cannot see it. It sits in the task file marked `excluded`, which is why the denominator is 49. ## It reproduces The benchmark is Apache-2.0 and the Jev release ships its runner, its source hashes and 294 redacted attempt transcripts. Everything below needs `npm ci`, a Chromium and a Python 3.12 venv — and no API key, no Docker and no money. All of it passed on first go: ```bash $ python3 scripts/verify-jev-release.py PASS: 294 matching attempts, 14044 transcript events, 30 source hashes; redaction regression checks and artifact scans. $ node experiments/jev/run.mjs --interface webmcp {"interface":"webmcp","tasks":49,"attempts":147,"mode":"check-only"} $ node experiments/jev/run.mjs --interface dom {"interface":"dom","tasks":49,"attempts":147,"mode":"check-only"} $ node experiments/jev/webmcp/fixture-check.mjs PASS: portable WebMCP method selects, fills, executes and finishes with both measured providers; no predicate leakage. $ node experiments/jev/webmcp/signal-seam-check.mjs PASS: actual providerFactory signal seam caps remaining attempt; … (2 real Chromium scenarios). $ node experiments/jev/ultrafast/fixture-check.mjs PASS: upstream policy bridge, real Chromium, accounting, timeout and fail-closed provider checks ``` Without `--run` the launcher re-hashes all 30 frozen runner files and all 49 task definitions and refuses to start if either moved. The fixture checks stand up a real Chromium, register a WebMCP tool on a local page, and drive the actual select-then-fill loop with scripted provider responses — so the code path in the article is the code path that ran, not a description of it. `294` and `14044` are `147 + 147` attempts and `3782 + 10262` transcript events. I did not spend a dollar to check a single number in this piece. That should be unremarkable and currently is not. ## The number that survives The headline that survives all of this is not 245x. It is that a model which cannot produce text, paired with a small one that can, is the cheapest *and* fastest row on a 21-configuration board — and that most of the margin is the interface, available to any model that uses it. Astra over WebMCP also solves 49/49, at 3.8x fewer tokens than the Jev pair. The decision model wins on price per token, not on economy of reading. Both factors are ratios of published medians, so the arithmetic is not in doubt — the reading is. Run any third model on both its own WebMCP row and its own computer-use row and see whether the interface term stays near 15x. If it collapses for cheap models (Luna's own pair is 7.1x) then "what WebMCP buys" is not a constant and the factorization is descriptive of Astra rather than of the interface. 93.5% of that row's dollars are cache-write tokens. If GPT-6 Astra does not in fact bill a write premium — the table's only support is a source comment — the honest multipliers are 199x and 93x, not 245x and 112x. A provider invoice for the same token mix settles it in one line. 77.6% of the WebMCP spend, from summing provider_usage over 147 transcripts. Swap Mercury for a cheaper argument writer — or, better, let the site declare which tools take no arguments so the scaffold's existing skip path fires more often — and re-measure. If total cost barely moves, the split is an artifact of these tools' schemas rather than of the architecture. Reasoned, and settled by one re-run: `hev-8` three times with the readiness timeout set to 0, which `createReadinessAdapter` already takes as a parameter. If it still passes 3/3, the empty tool list at step 2 was recoverable some other way and this caveat is wrong. About a dollar of API credit and ten minutes of Docker. The `complete_checkout` tool was added because WebMCP could not otherwise finish md-8, and took all eight WebMCP configurations from 0/3 to 3/3. Point the harness at a site whose tools were written by someone with no stake in the benchmark — or degrade the goldens to thin wrappers over existing endpoints, keeping the interface identical — and re-run. If 49/49 survives thin tools, the score is about WebMCP; if it does not, it is about these tools. It is two different agent implementations, which the release states. `runPageHybrid` in the frozen scaffold runs the identical select-then-fill loop against page controls. Running that arm over the same 49 tasks would separate "WebMCP beats page control" from "this scaffold beats that harness", and it is the single cheapest experiment left on the table.