{
  "claim": "Six projects now answer TypeSafe's System One contract from an open model. Every one of them has reproduced the serving half — a typed request in, a probability per option out, in one pass. On the half the category is named after, they split cleanly: the three that train or fit something publish a calibration number, and the three that are pure serving tricks publish none. Two of those three say so in their own documentation, which is the honest way to ship a thing like this.",
  "method": "Base model, parameter count and licence read from each project's own repository, model card or config: LFM2.5's BASE_MODEL_MANIFEST.json and the repo's 708,984,464-byte bf16 checkpoint; DiffusionGemma's model.safetensors.index.json plus the safetensors headers pulled with HTTP range requests and summed by hand (25,250,986,812 text parameters, 3,840,375,612 active at top_k_experts 8 of 128, plus a 572,794,416-parameter vision tower); gliner2-doom's head.safetensors headers read directly; kev's MODEL_CARD.md; jeff's README and bench/RESULTS.md. The readout column describes what the code actually computes, read out of the source file named in the article, not what the README calls it.",
  "source": "https://github.com/sgl-project/sglang",
  "captured": "2026-09-19",
  "note": "\"Calibration published\" means a number a reader can check — an ECE, a Brier score, a reliability curve — not a temperature, a confidence field or a spread over repeated draws. jeff's and kev's figures are their own, measured on their own eval sets, and both are worse than the hosted Jev they were measured against; both published them anyway. The DiffusionGemma row reports an entropy statistic and a standard error over noise draws, which describe the spread of the model's own answers rather than whether its probabilities track being right. deepseek-v4.1-flash-jev has no published artifact at all, which is the article's point rather than a criticism: a serving configuration has nothing to upload.",
  "columns": [
    { "key": "project", "label": "project" },
    { "key": "base", "label": "base model" },
    { "key": "params", "label": "params", "align": "right" },
    { "key": "licence", "label": "licence" },
    { "key": "readout", "label": "what the readout actually is" },
    { "key": "cal", "label": "calibration published" }
  ],
  "rows": [
    {
      "project": "deepseek-v4.1-flash-jev",
      "base": "DeepSeek-V4.1-Flash",
      "params": "undisclosed",
      "licence": "n/a — no artifact",
      "readout": "sglang /v1/score — logprobs of caller-chosen label_token_ids at one position",
      "cal": "none"
    },
    {
      "project": "LFM2.5-350M-RLCD",
      "base": "LiquidAI/LFM2.5-350M, byte-for-byte unchanged",
      "params": "354.5M",
      "licence": "LFM Open v1.0 (weights) · MIT (code)",
      "readout": "full-sequence log-likelihood per candidate, no length normalisation",
      "cal": "none — \"no calibration evaluation was performed\""
    },
    {
      "project": "vLLM PR #57250 (open)",
      "base": "google/diffusiongemma-26B-A4B-it",
      "params": "25.25B / 3.84B active",
      "licence": "Gemma terms (weights) · Apache-2.0 (PR)",
      "readout": "canvas-slot logprobs over label ids; each label must be one token",
      "cal": "none — entropy and a spread over noise draws"
    },
    {
      "project": "gliner2-doom",
      "base": "fastino/gliner2.5-multi-v1, frozen",
      "params": "983,553 (head only)",
      "licence": "no LICENSE file · Apache-2.0 base",
      "readout": "option-attention head, final layer [1, 256] — one scalar per option",
      "cal": "none — a fixed temperature 0.6 in the head JSON"
    },
    {
      "project": "jeff",
      "base": "knowledgator/gliformer-large-v1, frozen",
      "params": "400M (card) · 2.30 GB checkpoint",
      "licence": "MIT (code) · Apache-2.0 (weights)",
      "readout": "independent sigmoids per option, renormalised, temperature 3.2",
      "cal": "ECE 0.217 hard / 0.29 standard; Brier 0.745 / 0.50"
    },
    {
      "project": "kev-0.5b",
      "base": "Qwen/Qwen2.5-0.5B, frozen",
      "params": "494M frozen + 9.3M trained",
      "licence": "Apache-2.0",
      "readout": "LoRA + pointer head, softmax over per-option scalars",
      "cal": "ECE 0.065; 0.031 after temperature 1.47"
    }
  ]
}