{
  "claim": "One complaint about Jev contains three separate assertions. They are not equally supported, they need different experiments, and two of them turn out to be true while the third is a misreading of what was claimed. Filing them as one grievance is what makes the whole thing look like a verdict rather than three findings.",
  "method": "Each row records what the assertion actually says, the strongest evidence that bears on it, and the verdict that evidence supports. Rows 1 and 2 are recomputed in this article from published raw logs: the hosted-model numbers come from nibzard/decision-model-benchmark run v1.1, the local-model numbers from openjev's committed predictions at ca3ba65. Row 3 is a reading of TypeSafe's own wording against its own documentation, not a measurement.",
  "captured": "2026-09-22",
  "note": "The reason to separate them is that they have different fixes. Order sensitivity is trained out with option-order shuffling and worked around by averaging over permutations. Run-to-run variation is a serving property and is fixed, if at all, by the vendor's inference stack. And the type-safety guarantee needs no fix, because it is not what anybody was actually complaining about.",
  "columns": [
    { "key": "claim", "label": "the assertion" },
    { "key": "kind", "label": "what kind of claim it is" },
    { "key": "evidence", "label": "strongest evidence" },
    { "key": "verdict", "label": "verdict" }
  ],
  "rows": [
    {
      "claim": "The order of the choices drastically changes the output probabilities.",
      "kind": "Empirical, about one model. Settled by reordering one array.",
      "evidence": "Hosted Jev, 100 Banking77 items × 3 permutations × 3 repeats: the answer changes on 12, and the probability vector moves by a median of 0.04 and up to 0.69. Local vocabulary readout (openjev, 36 reversal pairs): 27.8% flip, slot A worth +1.71 logits.",
      "verdict": "True, and measured on both a hosted and an open model."
    },
    {
      "claim": "Exact same prompts give you different probabilities when you run it multiple times.",
      "kind": "Empirical, about a serving stack. Needs a repeat arm, not a reorder arm.",
      "evidence": "Hosted Jev, 1,300 items sent three times each with identical bodies and the pinned version echoed on every response: 67.2% returned a different probability vector, 3.3% changed their answer.",
      "verdict": "True. The cause is not identified and cannot be from outside."
    },
    {
      "claim": "…which is why I am confused by the no-hallucination claim.",
      "kind": "A claim about a claim. Settled by reading what was asserted.",
      "evidence": "TypeSafe asserts schema closure — \"Schema matching is guaranteed\" — and separately documents that \"Calibration… does not guarantee that an individual answer is correct.\" Neither sentence says the answer is a function of the question alone.",
      "verdict": "The guarantee holds; the reading that order sensitivity would break was never made."
    }
  ]
}