{
  "claim": "Jev does not return the same probabilities twice for the same request. Across 1,300 items sent three times each with byte-identical bodies, 873 of them (67.2%) came back with a different probability vector on at least one repeat, and the chosen answer itself changed on 43 (3.3%). Every response echoed the pinned version jev-1.13.0, so no model change is involved, and the API exposes no seed, temperature or sampling parameter that could be blamed.",
  "method": "Recomputed from the raw per-attempt logs published in nibzard/decision-model-benchmark, run v1.1 (2026-09-18T09:56:45Z), which records the provider's full response body for every attempt. The runner's protocol is repeats=3 at temperature 0 with concurrency 4; identical input_tokens across all three repeats of every item confirms the request bodies did not differ. An item counts as identical only if every option's probability is equal across all three repeats; S3's 275 items drop to 200 because the API answers `400 Too many choices` above 255 options, and a rejected request has no probabilities to compare. Probabilities arrive quantised to two decimals — all 69,300 values in the S4 suite lie exactly on the 0.01 grid — so this measures movement of at least half a grid step, not floating-point noise below it.",
  "source": "https://github.com/nibzard/decision-model-benchmark",
  "captured": "2026-09-22",
  "note": "Read the last column, not the first. Instability is not spread evenly: S3 plants the correct answer among distractors and every contender scores 100% on it, and it is also the most reproducible suite — three quarters of its items are bit-identical and not one answer moved. S5 withholds a good option on purpose, and only 2% of its items are bit-identical while 12% change their answer with nothing changed. The readout is reproducible exactly where the decision is easy. That is the same pattern option reversal shows, and it is why an aggregate accuracy number hides both.",
  "columns": [
    { "key": "suite", "label": "suite" },
    { "key": "opts", "label": "options", "align": "right" },
    { "key": "n", "label": "items", "align": "right" },
    { "key": "same", "label": "identical all 3×", "align": "right" },
    { "key": "flip", "label": "answer changed", "align": "right" },
    { "key": "dp", "label": "max |Δp| within an item (median / max)", "align": "right" }
  ],
  "rows": [
    {
      "suite": "S1 · Banking77 intent",
      "opts": "77",
      "n": "300",
      "same": "113 (37.7%)",
      "flip": "10 (3.3%)",
      "dp": "0.01 / 0.19"
    },
    {
      "suite": "S2 · SMS spam",
      "opts": "2",
      "n": "300",
      "same": "53 (17.7%)",
      "flip": "2 (0.7%)",
      "dp": "0.02 / 0.13"
    },
    {
      "suite": "S3 · planted answer, N sweep",
      "opts": "2–255",
      "n": "200",
      "same": "149 (74.5%)",
      "flip": "0 (0.0%)",
      "dp": "0.00 / 0.10"
    },
    {
      "suite": "S4 · S1 with permuted options",
      "opts": "77",
      "n": "300",
      "same": "108 (36.0%)",
      "flip": "7 (2.3%)",
      "dp": "0.02 / 0.16"
    },
    {
      "suite": "S5 · forced uncertainty",
      "opts": "6",
      "n": "200",
      "same": "4 (2.0%)",
      "flip": "24 (12.0%)",
      "dp": "0.04 / 0.15"
    },
    {
      "suite": "all five",
      "opts": "2–255",
      "n": "1,300",
      "same": "427 (32.8%)",
      "flip": "43 (3.3%)",
      "dp": "0.02 / 0.19"
    }
  ]
}