{
  "claim": "The Typed Decisions table in AgentJev's README is three different measurements stacked in one grid, and the README says so. Two rows were run in this repository, five were copied from the dataset card. The Brier column survives that mixing — recomputing it here with AgentJev's own code reproduces the card's uniform baseline to three decimals. The ECE column does not: the same code on the same uniform predictor returns 0.046 where the card's table says 0.169, so the 0.1687 that sits next to Jev's 0.144 is not the same statistic.",
  "method": "Rebuilt the 400-case / 2,000-question test split from LocalLLaMA/typed-decisions at revision ea93064, using typed_decisions/prepare_data.py's own convert() and jev_service.contract.prepare(), and checked that every question id and gold target matches the committed test_calibrated_predictions.json. Then ran typed_decisions/experiment.py's metrics() over a uniform predictor (1/C on every option) and over a per-question label-frequency prior fitted on the train split, and compared against the dataset card's reference rows.",
  "source": "https://huggingface.co/datasets/LocalLLaMA/typed-decisions",
  "captured": "2026-09-22",
  "note": "Uniform is the check that settles it, because 1/C on every option leaves nothing to a fitting choice. Its Brier comes out at 0.2382 against the card's 0.238. Its ECE comes out at 0.046 with a first-index tie-break and 0.015 with a random one; no bin count between 10 and 20 and no soft/hard accuracy variant I tried gets within a factor of three of the card's 0.169. My prior is a reconstruction rather than the card's own, so its 0.1978 against 0.189 is the looser of the two checks.",
  "columns": [
    { "key": "row", "label": "table row" },
    { "key": "kind", "label": "kind" },
    { "key": "top1", "label": "top-1", "align": "right" },
    { "key": "who", "label": "measured by" }
  ],
  "rows": [
    {
      "row": "AgentJev-0.6B",
      "kind": "specialist, fitted here",
      "top1": "79.25%",
      "who": "this repo — typed_decisions/experiment.py, 400 cases"
    },
    {
      "row": "Laya, published checkpoint",
      "kind": "specialist, fitted by its authors",
      "top1": "77.00%",
      "who": "this repo — laya-typed-decisions @ f9ab0b2, 19.2 s for 2,000 questions"
    },
    {
      "row": "TypeSafe Jev 1.13.0",
      "kind": "generalist, zero-shot",
      "top1": "72.7%",
      "who": "the dataset card — TypeSafe API, 2026-09-18, all 2,000 decisions, $0.016"
    },
    {
      "row": "ModernBERT-base 149M",
      "kind": "specialist",
      "top1": "64.6%",
      "who": "the dataset card — Adaptive Classifier 0.2.0, frozen encoder"
    },
    {
      "row": "MiniLM-L6 22M",
      "kind": "specialist",
      "top1": "58.7%",
      "who": "the dataset card — same harness"
    },
    {
      "row": "Prior (label frequency)",
      "kind": "reference",
      "top1": "47.0%",
      "who": "the dataset card — reads nothing"
    },
    {
      "row": "Uniform",
      "kind": "reference",
      "top1": "30.8%",
      "who": "the dataset card — 1/C on every option"
    },
    {
      "row": "Teacher self-agreement",
      "kind": "ceiling, not in the README",
      "top1": "73.5%",
      "who": "the dataset card — a fresh teacher sample against gold built from the others"
    },
    {
      "row": "Perfect scenario understanding",
      "kind": "ceiling, not in the README",
      "top1": "70.4%",
      "who": "the dataset card — the latent factors fitted to gold, cross-validated"
    }
  ]
}