{
  "claim": "The 419 human ratings in R4T's rebuttal repo do not reproduce the pattern of its Table 1. In both base-model families the two arms split the two metrics — one wins alignment decisively, the other wins diversity — where the paper's LLM judge has R4T ahead on both. The alignment gap is the larger and the more certain of the two.",
  "method": "github.com/pat-jj/r4t-rebuttal ships two spreadsheets of raw Google Forms responses: 35 forms x 6 respondents for Gemma (210 rows) and 35 forms x 6 (one form has 5) for Qwen (209 rows), each row rating Result 1 and Result 2 on the paper's own Alignment and Diversity rubrics, 1-5. Per query I take the mean of the responses for each arm, then compare the two arms across the 35 queries with an exact two-sided sign test on the non-tied queries. Two cells are blank (one per file, both in form 11) and are dropped pairwise. 'dedup' repeats the whole calculation after collapsing each form's identical score vectors to one.",
  "source": "https://github.com/pat-jj/r4t-rebuttal",
  "captured": "2026-09-18",
  "note": "Which arm is which system is not stated anywhere in the repo. I infer Result 1 = zero-shot baseline from pixel matching against the paper's Figure 3 (see the article). The split-decision finding does not depend on that inference: whichever arm is R4T, it wins exactly one of the two metrics, and the paper's Table 1 has it winning both. Independence is weaker than 6-per-query suggests: the median form carries 4 distinct score vectors, not 6, and 34 (Gemma) / 41 (Qwen) consecutive within-form submissions repeat the previous row's four scores less than 10 seconds later — the fastest pair 0.07 s apart.",
  "columns": [
    { "key": "family", "label": "base model" },
    { "key": "metric", "label": "metric" },
    { "key": "arm1", "label": "Result 1 mean", "align": "right" },
    { "key": "arm2", "label": "Result 2 mean", "align": "right" },
    { "key": "wins", "label": "queries won (R1/R2/tie)", "align": "right" },
    { "key": "p", "label": "sign test p", "align": "right" },
    { "key": "dedup", "label": "dedup p", "align": "right" }
  ],
  "rows": [
    { "family": "Gemma3-4B", "metric": "alignment", "arm1": 4.243, "arm2": 3.771, "wins": "26 / 4 / 5", "p": "5.9e-05", "dedup": "9.1e-04" },
    { "family": "Gemma3-4B", "metric": "diversity", "arm1": 3.481, "arm2": 3.748, "wins": "11 / 20 / 4", "p": "0.15", "dedup": "0.34" },
    { "family": "Qwen3-4B", "metric": "alignment", "arm1": 4.446, "arm2": 3.729, "wins": "31 / 3 / 1", "p": "7.7e-07", "dedup": "1.4e-06" },
    { "family": "Qwen3-4B", "metric": "diversity", "arm1": 2.736, "arm2": 3.259, "wins": "11 / 23 / 1", "p": "0.058", "dedup": "0.035" }
  ]
}