{
  "claim": "The same ticket, the same architecture, two plausible sixteen-option support taxonomies. The 25M model is wrong on both. The 70M model is right only when the correct option is spelled out, and by 5.7 points. The 395M model is right on both and knows it. This is the accuracy half of the ladder whose latency half is in the table above.",
  "method": "One decision = 16 (context, hypothesis) pairs through the shipped path — transformers.js 4.3.0 on ONNX Runtime Web (WASM), hypothesis template `This example is {}.`, softmax over the 16 entailment logits. Identical context in every cell: a customer charged twice for one order, unrefunded after nine days, two ignored emails. 'terse' is a generic support taxonomy containing three near-synonyms of the right answer (a billing problem / a request for a refund / a duplicate charge dispute); 'explicit' is the option list the widget ships with. No template tuning: the stock zero-shot form was used throughout, including where it made a model look bad.",
  "source": "Xenova/mobilebert-uncased-mnli, Xenova/nli-deberta-v3-xsmall, onnx-community/ModernBERT-large-zeroshot-v2.0-ONNX — all read from the Hub, run in headless Chromium 141",
  "captured": "2026-09-19",
  "note": "N = 1 decision per cell. This is a demonstration of the failure mode, not an accuracy benchmark, and it is reported that way: nobody should conclude a ranking of these three models from six decisions. What it does establish is that at the sizes that fit comfortably in a browser, the answer depends on how the caller phrased the options — which is a property of the whole category, not of these checkpoints.",
  "columns": [
    { "key": "model", "label": "model", "align": "left" },
    { "key": "mib", "label": "MiB", "align": "right", "mono": true },
    { "key": "terse", "label": "terse list — top answer", "align": "left" },
    { "key": "tp", "label": "p", "align": "right", "mono": true },
    { "key": "expl", "label": "explicit list — top answer", "align": "left" },
    { "key": "ep", "label": "p", "align": "right", "mono": true }
  ],
  "rows": [
    {
      "model": "MobileBERT-MNLI 25M q8",
      "mib": 26.4,
      "terse": "a complaint about support response time ✗",
      "tp": 0.2349,
      "expl": "a chargeback already filed with the bank ✗",
      "ep": 0.2507
    },
    {
      "model": "DeBERTa-v3-xsmall NLI 70M q8",
      "mib": 91.5,
      "terse": "a question about pricing ✗",
      "tp": 0.3078,
      "expl": "a duplicate charge that needs refunding ✓",
      "ep": 0.4525
    },
    {
      "model": "ModernBERT-large zeroshot 395M q4f16",
      "mib": 287.0,
      "terse": "a duplicate charge dispute ✓",
      "tp": 0.6592,
      "expl": "a duplicate charge that needs refunding ✓",
      "ep": 0.9524
    }
  ]
}