{
  "claim": "In the only third-party benchmark that runs a decision model end to end, the decision model makes the decisions and a generative model does almost everything else. 385 of the 413 actions Jev selected could not be executed until another model had written their arguments, and 379 of those 385 required at least one string that was not in any list.",
  "method": "Computed from the 147 redacted WebMCP attempt records published with WindTunnel board v1.2 (results/2026-09-18-jev-mercury/webmcp-traces.jsonl.gz). Calls, token counts and costs are summed from each record's provider_usage; the free-text count is taken by inspecting the JSON Schema in each Mercury request body for a string property with no enum and no const. Costs are the harness's own estimates at each provider's list price and are not comparable as a measure of work.",
  "source": "https://github.com/nekuda-ai/WindTunnel",
  "captured": "2026-09-19",
  "note": "Jev reads half again as many input tokens as Mercury and costs under a third as much, which is the whole commercial argument for the shape. It is also doing less than half the job: every argument, every field value and all 143 final answers were written by the other model.",
  "columns": [
    {
      "key": "metric",
      "label": "across 147 scored attempts"
    },
    {
      "key": "jev",
      "label": "Jev 1.13.0",
      "align": "right"
    },
    {
      "key": "mercury",
      "label": "Mercury 2.5",
      "align": "right"
    }
  ],
  "rows": [
    {
      "metric": "provider calls",
      "jev": "413",
      "mercury": "385"
    },
    {
      "metric": "input tokens",
      "jev": "1,074,068",
      "mercury": "712,698"
    },
    {
      "metric": "output tokens",
      "jev": "30,706",
      "mercury": "18,382"
    },
    {
      "metric": "estimated cost",
      "jev": "$0.045111",
      "mercury": "$0.156447"
    },
    {
      "metric": "share of the bill",
      "jev": "22.4%",
      "mercury": "77.6%"
    },
    {
      "metric": "calls filling at least one unconstrained string",
      "jev": "n/a",
      "mercury": "379 of 385"
    },
    {
      "metric": "options per decision (min / median / max)",
      "jev": "3 / 7 / 10",
      "mercury": "n/a"
    }
  ]
}