{
  "claim": "Option-order sensitivity in the Kev family is continuous, not binary, and it shrinks with backbone size: from 15.3% of permuted Choice questions changing their answer on the Qwen2.5-0.5B prototype to 1.8% on Qwen3.5-9B. Two reference points bracket it. The option-isolation arm — the same code with every option span made its own sub-branch — flips nothing at all and moves its probabilities by 2e-7, which is fp32 noise. Jev also flips nothing, in 324 permuted questions across seven runs, but its probabilities move by 0.020, five orders of magnitude more than the arm that is invariant by construction.",
  "method": "Every trial in jaredpalmer/kev at 1c35199 commits a `permutation` block, and so does every standalone benchmark run: each frozen suite ships a `permuted` variant of every Choice question (the option keys shuffled under a per-record seed), and kev.benchmark joins each one back to its parent, realigns the probabilities by option key, and records both the argmax flip rate and the mean per-question maximum change in probability. The Kev rows below pool those blocks by base model over every trial result.json that carries a provenance config, across both the in-distribution (decision-*) and out-of-domain (transfer-*) partitions, excluding six-item smoke runs and the two research arms (option_isolation, perm_kl) — the isolation arm is listed separately on its own row. The Jev row pools the seven standalone report.json runs against the hosted model through the Vercel AI Gateway. Items repeat across trials of the same suite, so 'items' counts question-evaluations, not distinct questions: this measures a population of checkpoints, not a population of questions.",
  "source": "https://github.com/jaredpalmer/kev",
  "captured": "2026-09-22",
  "note": "Jev's zero is what the site has read three times as evidence of per-option scoring. Its non-zero probability movement is the part nobody has reported, and it has a dull alternative explanation nobody has ruled out either: a hosted endpoint can be non-deterministic on identical input. The control is one line — send the same option order twice and measure the same quantity.",
  "columns": [
    {
      "key": "model",
      "label": "model / arm",
      "align": "left"
    },
    {
      "key": "evals",
      "label": "evals",
      "align": "right",
      "mono": true
    },
    {
      "key": "items",
      "label": "permuted items",
      "align": "right",
      "mono": true
    },
    {
      "key": "flips",
      "label": "answer flips",
      "align": "right",
      "mono": true
    },
    {
      "key": "rate",
      "label": "flip rate",
      "align": "right",
      "mono": true
    },
    {
      "key": "delta",
      "label": "mean max Δp",
      "align": "right",
      "mono": true
    }
  ],
  "rows": [
    {
      "model": "Kev, option_isolation arm (0.6B / 4B / 8B)",
      "evals": 18,
      "items": 864,
      "flips": 0,
      "rate": "0.0000",
      "delta": "0.0000002"
    },
    {
      "model": "Jev 1.13, hosted",
      "evals": 7,
      "items": 324,
      "flips": 0,
      "rate": "0.0000",
      "delta": "0.0202"
    },
    {
      "model": "Kev, Qwen3.5-9B",
      "evals": 14,
      "items": 672,
      "flips": 12,
      "rate": "0.0179",
      "delta": "0.0326"
    },
    {
      "model": "Kev, Qwen3-8B",
      "evals": 16,
      "items": 768,
      "flips": 20,
      "rate": "0.0260",
      "delta": "0.0311"
    },
    {
      "model": "Kev, Qwen3.5-4B",
      "evals": 32,
      "items": 1536,
      "flips": 59,
      "rate": "0.0384",
      "delta": "0.0369"
    },
    {
      "model": "Kev, Qwen3.6-35B-A3B",
      "evals": 4,
      "items": 192,
      "flips": 11,
      "rate": "0.0573",
      "delta": "0.0460"
    },
    {
      "model": "Kev, Qwen3.5-0.8B",
      "evals": 10,
      "items": 480,
      "flips": 29,
      "rate": "0.0604",
      "delta": "0.0544"
    },
    {
      "model": "Kev, Qwen3-4B",
      "evals": 83,
      "items": 3972,
      "flips": 241,
      "rate": "0.0607",
      "delta": "0.0535"
    },
    {
      "model": "Kev, Qwen3-0.6B",
      "evals": 47,
      "items": 2244,
      "flips": 152,
      "rate": "0.0677",
      "delta": "0.0646"
    },
    {
      "model": "Kev, Qwen2.5-0.5B prototype",
      "evals": 10,
      "items": 504,
      "flips": 77,
      "rate": "0.1528",
      "delta": "0.0895"
    }
  ]
}