{
  "claim": "All six contextual-hallucination comparisons go DIFF's way, and at 100 samples each only one of them is individually larger than two standard errors. The pattern is the evidence, not any single row.",
  "method": "Accuracy (free of hallucination, judged by GPT-4o against the reference) copied from Table 4 of arXiv 2410.05258v2; 100 samples per dataset, both models 3B trained on 350B tokens. My arithmetic: standard error of the difference assuming two independent binomial samples, sqrt(p1(1-p1)/100 + p2(1-p2)/100), and z = gap / SE. The same 100 inputs were given to both models, so a paired test would be tighter; the paper does not publish per-item judgements, so it cannot be run. Sign test for six of six in one direction: (1/2)^6 = 1/64, one-sided.",
  "source": "https://arxiv.org/abs/2410.05258",
  "captured": "2026-09-26",
  "note": "Summaries: XSum, CNN/DM, MultiNews. Question answering: Qasper (single-document), HotpotQA and 2WikiMultihopQA (multi-document), all drawn from LongBench.",
  "columns": [
    { "key": "task", "label": "dataset", "align": "left" },
    { "key": "t", "label": "Transformer", "align": "right" },
    { "key": "d", "label": "DIFF", "align": "right" },
    { "key": "gap", "label": "gap", "align": "right" },
    { "key": "se", "label": "SE (unpaired)", "align": "right" },
    { "key": "z", "label": "z", "align": "right" }
  ],
  "rows": [
    { "task": "XSum", "t": "0.44", "d": "0.53", "gap": "+0.09", "se": "0.070", "z": "1.28" },
    { "task": "CNN/DM", "t": "0.32", "d": "0.41", "gap": "+0.09", "se": "0.068", "z": "1.33" },
    { "task": "MultiNews", "t": "0.42", "d": "0.61", "gap": "+0.19", "se": "0.069", "z": "2.74" },
    { "task": "Qasper", "t": "0.28", "d": "0.39", "gap": "+0.11", "se": "0.066", "z": "1.66" },
    { "task": "HotpotQA", "t": "0.36", "d": "0.46", "gap": "+0.10", "se": "0.069", "z": "1.45" },
    { "task": "2WikiMQA", "t": "0.29", "d": "0.36", "gap": "+0.07", "se": "0.066", "z": "1.06" }
  ]
}