# Retrieve-for-Train: the 20x is in the prose, not the figure
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/retrieve-for-train
> date: 2026-09-18
> tags: retrieval, diffusion, reinforcement-learning, synthetic-data, benchmarks, explainer
The framing going around, from Google Research's own post and the aggregators that picked it up: Retrieve-for-Train "slashes AI search latency by 20x," a "12× to 20× faster query fan-out." I went looking for the paper behind it, because the interesting part of R4T is not the speedup — it is the claim that you can use reinforcement learning as a *compiler*. Run RL once against a reward you cannot differentiate, harvest what the policy learned as ordinary (input, target) pairs, train a small supervised model on those, and delete the policy. If that works it is reusable well outside retrieval.
So: the ID first. I resolved it through the arXiv API rather than trusting any number I was handed. `arXiv:2603.06397v1`, submitted 6 March 2026, primary `cs.IR`, cross-listed `cs.LG`, titled *Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion*, eleven authors — Pengcheng Jiang, Judith Yue Li, Moonkyung Ryu, R. Lily Hu, Kun Su, Zhong Yi Wan, Liam Hebert, Hao Peng, Jiawei Han, Dima Kuzmin, Craig Boutilier — Google and UIUC. That checks out. Two footnotes on provenance: the arXiv record carries no `Comments` field, so the ICML 2026 attribution comes from the blog post and the first author's homepage, not from arXiv; and there is only a v1, never revised. There is also no code. The first author ships repos for his other retrieval work — `pat-jj/DeepRetrieval`, `pat-jj/s3`, both with hundreds of stars — and none for this one. Music is a proprietary internal dataset, training ran on TPU pods named in-house, so that is unsurprising rather than suspicious. But it does mean "read the code and check it against the paper" is not available here.
What *is* available, and what nobody seems to have opened, is `pat-jj/r4t-rebuttal`: four files, no README, 5.8 MB, last touched 31 March 2026. It contains a human evaluation the paper does not report. More on that at the end, because it is the most interesting thing in this whole exercise.
## RL as a compiler
The problem R4T picks is real and stated well. Most retrieval supervision is pointwise: a query, a correct document. But plenty of retrieval is *set-valued* — give me ten outfits for "bohemian festival style," give me a playlist — and the thing you actually want is a property of the set, like diversity or coverage, which does not decompose into per-item labels and does not have a unique right answer. RL can optimize that, because a reward only has to be computable, not differentiable. The catch is that you then have to *run* a 4B model at query time, autoregressively, once per user query.
R4T's move is to not deploy it. Three stages:
Stage one trains a fan-out language model (FOLM) — Gemma3-4B or Qwen3-4B — with GRPO plus forward *and* reverse KL penalties, against a composite reward. Stage two freezes it and samples it a lot. Stage three trains a 53.9M-parameter diffusion transformer to map a query embedding straight to a block of content embeddings in one shot, which get resolved to database items by nearest neighbor. The 4B model never runs again.
The reward is worth writing out, because everything downstream turns on its weighting. For the open-ended task,
$$
\mathcal{R}_{\text{abs}}(q, Q) = \lambda_g\, r_{\text{ground}}(Q) + \lambda_d\, r_{\text{div}}(Q) + \lambda_a\, r_{\text{align}}(q, Q)
$$
with $\lambda_g = 0.6$ and $\lambda_d = \lambda_a = 0.2$. Groundedness penalizes the distance from each sub-query embedding to its nearest database item — it keeps the policy on the manifold of things that actually exist in the index. Diversity is the Vendi Score over the top-1 item retrieved per sub-query. Alignment is mean cosine similarity back to the original query.
The ablation around this is the best part of the paper and it is honest about a failure mode. Train on groundedness alone and the policy collapses to nonsense strings — the paper quotes its own output, `"line ending line ending line ending"` — which happen to sit very close to one particular database item. Add alignment and it collapses *faster*, into paraphrases of the input query. Only with all three does GRPO stay stable. That is a clean demonstration that the three terms are mutual anti-hacking anchors, and it is the kind of negative result most papers leave out.
The paper contradicts itself about stage two, and it took the figure to settle it. The introduction says stage two works "by harvesting and filtering successful trajectories"; §2.1 says it collects "high-reward trajectories." Then §2.4 says the opposite: "We use the FOLM generations as supervision, allowing the downstream model to learn the **full distribution** of behaviors induced by the RL process." Figure 1 is the tiebreaker — step 2 draws query → FOLM → targets with no property check anywhere in the path, while step 1 right above it draws the check explicitly. So there is no reward filter on the synthetic data; the diffusion model learns the policy's whole output distribution, high-reward samples and low-reward samples alike. That is a defensible design (it preserves entropy, which matters here), but "filtering successful trajectories" describes a different algorithm than the one the figure and §2.4 describe.
## The 20x is in the prose, not the figure
Here is the headline number, and it is the one thing in the paper I could check exactly rather than approximately.
Figure 5 ships as a matplotlib SVG at `arxiv.org/html/2603.06397v1/query_fanout_efficiency.svg`. Matplotlib writes every marker as a `` with a `transform="matrix(1,0,0,-1,tx,ty)"` placing it at the data point, and every gridline as a horizontal `` at a labeled tick. So the plotted values are recoverable exactly — not estimated off a screenshot, read out of the vector. This is one marker, verbatim from the file:
```xml
```
Both sit at `x = 523.045`, the batch-1024 tick. Orange is autoregressive, blue is diffusion. The six main-axes gridlines run `75.04, 131.04, 187.03, 243.03, 299.02, 355.02` against labels 0 through 50 seconds, so one second is 5.59956 units and the orange marker reads 49.766 s, the blue 3.930 s.
I checked that calibration twice before trusting it. It reproduces both autoregressive numbers the paper states in words — "approximately 1.46 seconds even for a batch of 8" against my 1.456, and "nearly 50 seconds at a batch size of 1024" against my 49.766. And the figure's zoom inset is a *separate* set of axes with its own gridlines and its own scale; calibrated independently it returns 1.456 / 1.518 / 1.652 / 1.852 and 0.090 / 0.098 / 0.157 / 0.230 for batches 8 through 64 — identical to the main panel, to three decimals.
Now the prose. It says the diffusion model "process[es] the small batch in just 0.07 seconds and the largest batch in 4.21 seconds," and concludes with "a consistent 12×-20× speedup." Neither of those diffusion numbers is what the figure plots. The figure has 0.090 s at batch 8, not 0.07, and 3.930 s at batch 1024, not 4.21. And `1.46 / 0.07 = 20.9` is precisely where the 20x comes from — it is an arithmetic consequence of the one number the figure contradicts.
Divide the figure's own two series and the speedup runs from **8.05x at batch 64 to 16.18x at batch 8**. Five of the eight measured batch sizes fall *below* the claimed 12x floor. None reaches 20x. The word doing the most work is "consistent": the ratio is not monotone, it dips by half in the middle of the range, and the figure caption's gloss — "maintaining sub-second performance for small batches and achieving an order-of-magnitude speedup at larger batch sizes" — has the direction backwards, since the largest ratio is at the *smallest* batch.
**Receipts.** The speedup plotted in R4T's own Figure 5 ranges from 8.1x to 16.2x and falls below 12x at five of the eight batch sizes measured. The paper's prose reports "a consistent 12x-20x speedup"; the 20x end does not appear anywhere in the figure.
| batch size | autoregressive (s) | diffusion (s) | speedup | inside claimed 12x-20x? |
| ---: | ---: | ---: | ---: | :--- |
| 8 | 1.456 | 0.09 | 16.18x | yes |
| 16 | 1.518 | 0.098 | 15.49x | yes |
| 32 | 1.652 | 0.157 | 10.52x | no — below |
| 64 | 1.852 | 0.23 | 8.05x | no — below |
| 128 | 5.632 | 0.527 | 10.69x | no — below |
| 256 | 11.448 | 1.091 | 10.49x | no — below |
| 512 | 24.531 | 2.141 | 11.46x | no — below |
| 1,024 | 49.766 | 3.93 | 12.66x | yes |
The prose's two diffusion numbers do not match its own figure: it reports 0.07 s at batch 8 where the figure plots 0.090 s, and 4.21 s at batch 1024 where the figure plots 3.93 s. 1.46/0.07 = 20.9x is where the headline 20x comes from; 1.456/0.090 = 16.2x is what the figure supports. No hardware, serving stack or token budget is stated for either series, and the retrieval calls the autoregressive path has to make are not separated out.
> method: Figure 5 ships as a matplotlib SVG (arxiv.org/html/2603.06397v1/query_fanout_efficiency.svg). Every marker is a at the data point and every gridline is a horizontal at a labelled tick, so ty calibrated against the six main-axes gridlines (0-50 s) recovers the plotted value. The calibration reproduces the paper's own two stated autoregressive numbers exactly -- 1.456 s at batch 8 against its "approximately 1.46 seconds", and 49.77 s at batch 1024 against its "nearly 50 seconds" -- and the figure's zoom inset, an independent set of axes, returns the same four values for batches 8-64.
> source: https://arxiv.org/html/2603.06397v1
> captured: 2026-09-18
> data: https://ai.thesatyajit.com/articles/retrieve-for-train/data/fig5-latency.json (8 rows)
To be clear about what this does and does not overturn. A ten-times wall-clock gap between a 4B autoregressive decode and a 53.9M single-shot sampler is a real engineering result and the right shape for production. Nothing here says the method is slow. What it says is that the number in every headline about this paper traces to a prose sentence that the paper's own figure does not support, and that if you want the honest one-line version it is "roughly 8x to 16x, depending heavily on batch size."
There is a second gap underneath, which is that **no hardware is named anywhere in §3.4**. Appendix F tells you the FOLM trained on a TPUv6e-16 and the diffusion model on a TPUv6e-16, but the latency benchmark says only "wall-clock latency is measured across varying batch sizes." No accelerator, no serving stack, no mention of whether the autoregressive side used continuous batching or a naive loop, and no statement of whether the `k` retrieval round-trips — which the autoregressive path must make and the diffusion path also makes — are inside or outside the measurement. Appendix D shows the fan-out prompt requires a `` block before the JSON, so the baseline's token count is dominated by unbounded reasoning tokens that are never reported.
## "Single-pass" means 256 passes
The efficiency framing rests on a phrase repeated throughout: the diffusion retriever "generates all `k = 10` retrieval directions in a single forward pass," non-autoregressively, in one shot. Table 3 in Appendix F, the hyperparameter table:
| Parameter | Value |
|---|---|
| Seq. Length (L) | 12 |
| Embed. Dim (d) | 128 |
| Hidden / MLP Dim | 1024 |
| Layers / Heads | 6 / 16 |
| Scheme | Variance Exploding |
| **Steps** | **256** |
| CFG Strength | 0.1 |
| Total Steps (training) | 10 × 10⁶ |
| Batch Size (training) | 512 |
`Steps: 256` is the inference-time SDE solver. §2.5 is careful and says so — "we solve the probability flow stochastic differential equation to generate $\mathbf{Z}_0$" — but §3.4, the section that makes the speed argument, says "a single forward pass." Those are different things by a factor of 256. What is single-pass about R4T-Diffusion is the *pipeline*: one call to the retrieval stack instead of a serialized generate-then-retrieve loop. The denoiser itself runs 256 times per query.
That reframes where the win comes from. A rough arithmetic check, mine and clearly labeled as an estimate: at 2 FLOPs per parameter per token, 53.9M parameters over `L = 12` rows for 256 solver steps is about **331 GFLOP per query**. A 4B model decoding ~150 tokens of reasoning plus ten short sub-queries is about 1.2 TFLOP. That is roughly 3.6x in arithmetic against 8-16x in wall clock. The gap between those two ratios is the point: most of R4T-Diffusion's advantage is that 256 small steps are *parallel across the batch* while 150 token steps are *serial per query*. It is a serialization win, not primarily a compute win — which is a better and more durable argument than the one the paper makes, and the paper does not make it.
One number in that table I cannot interpret and will not pretend to. `CFG Strength: 0.1`, with `Cond. Drop: 0.1`. Classifier-free guidance has two common parameterisations in the literature and the paper defines neither; under one of them 0.1 is essentially no guidance, under the other it is a tenth of the usual strength. Either way it is at the far weak end for a model whose entire job is to be conditioned on the query embedding. It may well be correct — embedding-space diffusion is not image diffusion and the usual scales need not transfer. I flag it because it is the single hyperparameter that most directly governs how much the query influences the output, and it is printed without units or a definition.
And `Seq. Length (L) = 12` sits oddly against `k = 10`. Every baseline in the paper generates "up to 10 queries" (Appendix D's prompt says so literally); §3.1 says "for all fan-out baselines and R4T, we report results with `k = 10` sub-queries"; Appendix B.2 defines Recall@5K as "500 candidates for each of the `k = 10` sub-queries (or generated embeddings), yielding a pool of 5,000 items." But the diffusion model emits 12 rows. 500 × 12 = 6,000, not 5,000. Either the 12 rows get sliced to 10 somewhere (the paper never says so) or the metric named `@5K` is computed over a pool 20% larger for the method than for the baselines. It is probably the former. It should have been one sentence.
## The 60% of the objective the shipped model cannot be scored on
Look at Table 1 again with the reward weights in mind. `R4T-Diffusion` — the artifact the entire framework exists to produce — has `\` in the Groundedness column, and therefore `\` in the Average column too. The paper's own footnote: "Groundedness is not applicable to R4T-Diffusion... due to the absence of intermediate sub-query."
That is internally consistent. Groundedness is defined on sub-query embeddings (Eq. 3), and the LLM judge scores it as "how accurately each retrieved collection corresponds to its generating sub-query" (Appendix B.1). No sub-query, nothing to score. But follow it through: groundedness carries $\lambda_g = 0.6$, the largest weight in the reward, and it exists specifically to stop the policy drifting off the database manifold. So for the deployed model, every reported number comes from the 40% of the objective that survives the change of representation, and the term with the most weight — the one the ablation shows is the most gameable — is unmeasured at deployment by construction.
There is a cheap fix the paper does not take. The diffusion model emits embeddings that are then nearest-neighbor matched to database items; the distance to that nearest neighbor is exactly the quantity Eq. 3 averages. A content-side groundedness number was available for free. It is not reported.
## Two judges, one of them a contestant
Section 3.1: "we employ LLM-as-a-Judge evaluation using **Gemini-2.5-Pro** to assess three dimensions."
Appendix B.1: "We therefore adopt LLM-as-a-Judge evaluation using **Gemini-2.5-Flash**, which supports multimodal inputs."
Those are different models, and the reader cannot tell which produced Table 1. It matters more than a typo normally would, because Gemini-2.5-Flash is also one of the three baseline systems *inside* Table 1 — its own zero-shot fan-out is the first block of scores in the table it may have graded. If Appendix B.1 is the accurate one, a model is grading its own fan-out against the method's. If §3.1 is the accurate one, the appendix that documents the protocol is wrong about the protocol. The paper also uses Gemini-2.5-Pro to *generate* the broad queries for Table 2 (its caption says so), so both models genuinely appear in the pipeline — which is exactly why the judge's identity needed to be stated once, consistently.
While in Table 1: the scale is never defined either. Appendix B.1 says each dimension is scored "on a 5-point Likert scale (1=Poor … 5=Excellent)." Table 1 prints 22.4, 76.8, 63.1. There is no stated mapping from one to the other, and the two obvious candidates — $100s/5$ and $100(s-1)/4$ — give materially different reconstructions. So Table 1's numbers can be compared to each other and their direction can be trusted; their *level* cannot be compared to anything outside the table. I will come back to this.
## Where R4T wins, and the two places it does not
Credit where it is due: on the open-ended task the margins are large and consistent. Gemma3-4B on Polyvore goes 38.5 average zero-shot, 40.9 with Best-of-5, and 49.1 with R4T-FOLM. Best-of-N is a genuinely strong baseline — it selects with the same reward the method trains on — and R4T beats it with one inference pass instead of five. That is the central claim and Table 1 supports it.
But "R4T consistently dominates the accuracy-efficiency frontier," as §3.2 puts it, is not what Table 2 shows. Three things in that table run against the sentence:
- **The Vendi Score column has R4T last.** `R4T-FOLM (Qwen)` scores **27.5**, the lowest diversity of any method in the table — below Gemini-2.5-Flash's 33.4 and well below the plain Qwen3-4B zero-shot baseline's **46.4**, which is the highest score in the column. The best diversity number in Table 2 belongs to an untrained baseline.
- **The deployed model trails a baseline on the primary coverage metric.** `R4T-Diffusion (Gemma)` gets Recall@5K of **15.0** against Gemini-2.5-Flash's **15.7**. It wins on Hit@5K (54.1 vs 52.1), and its Qwen sibling clears both — but on one of two configurations the shipped artifact is behind a zero-shot proprietary baseline on the headline recall number.
- **Distillation costs quality on the task that has ground truth.** R4T-FOLM (Qwen) gets 20.9 Recall; R4T-Diffusion (Qwen) gets 16.5. That is a 21% relative drop from teacher to student, and the same direction holds for Gemma (16.9 → 15.0). On the open-ended task, R4T-Diffusion is below R4T-FOLM on three of the four applicable cells too.
To be fair to the paper: §3.2 discusses the diversity trade-off directly and explains it well, citing the entropy-collapse literature on RL-tuned LMs, and argues that lower recall against a non-exhaustive reference set is not obviously worse. Both points are reasonable. The complaint is narrower — "consistently dominates" is a claim about every cell, and the table has cells that say otherwise.
## The ledger
Appendix F is unusually forthcoming about hardware, so the ledger is mostly assemblable. Stage 1: TPUv6e-16, lr `1e-7`, global batch 512, group size 8. Stage 2: TPUv6e-4. Stage 3: TPUv6e-16.
What is not assembled anywhere is the total. Appendix C.1 says the OAR corpus is 43,874 unique broad queries after deduplication, split 8:1:1, and that supervision is synthesized at "temperature 0.9 and a sample size of 128" per query. The split does not divide evenly — 43,874 × 0.8 is 35,099.2 — so the exact train count is unrecoverable, and the text does not say whether the 128 samples are drawn for the train split or all queries. Take both ends:
| | train split only | all queries |
|---|---|---|
| queries sampled | ~35,099 | 43,874 |
| fan-outs generated (× 128) | 4,492,672 | 5,615,872 |
| retrieval calls (× k = 10) | 44,926,720 | 56,158,720 |
| epochs over that set at 10M × 512 | ~1,140 | ~912 |
So: somewhere between 45 and 56 *million* retriever round-trips to build the dataset, and then between roughly 900 and 1,140 full passes over it to train the 53.9M model — 5.12 billion sample presentations, from a target distribution over a candidate pool of 21,888 Polyvore collections. The upfront cost is real, and Appendix A says so in general terms ("the upfront training overhead may be substantial"). It never puts a number on it, and the number is the whole trade the method is asking you to accept: amortize all of that against inference, and you win if and only if you serve enough queries.
Nine hundred–plus epochs over a synthetic set drawn from a 21,888-item pool also raises a question the paper does not address: how much of R4T-Diffusion is generalization and how much is memorization of the FOLM's outputs on the training queries? The test split exists — the paper just never reports a train/test gap.
## The rebuttal repo nobody opened
Appendix A lists LLM-as-a-Judge as a limitation: "such evaluations may still introduce biases inherited from the judge model... Incorporating human evaluation or hybrid evaluation protocols would strengthen future studies."
They ran one. It is sitting in a repo with no README, no stars, and no link from the paper:
```
pat-jj/r4t-rebuttal/ # created 2026-03-31, 4 files, no README
polyvore_gemma_responses.xlsx # 35 sheets x 6 responses = 210 ratings
polyvore_qwen_responses.xlsx # 35 sheets, 209 ratings (one form has 5)
instructions_and_score_definitions.pdf
example - polyvore_qwen_15 - Google Forms.pdf
```
The instructions PDF carries the same two rubrics as Appendix B.1 — Query-Collection Alignment and Collection Diversity, with each definition copied verbatim from the appendix and a written anchor added for every point from 1 (Poor) to 5 (Excellent). There is no groundedness question, consistent with Table 1's blank column for R4T-Diffusion. Every sheet has the same header:
```
Timestamp | Result 1 Alignment Score | Result 2 Alignment Score | Result 1 Diversity Score | Result 2 Diversity Score
2026-03-30 03:21:49.096 | 3.0 | 1.0 | 2.0 | 1.0
2026-03-30 03:25:38.324 | 3.0 | 1.0 | 1.0 | 1.0
```
419 ratings across 70 forms. And nowhere in the repo does anything say which system is Result 1 and which is Result 2.
So I worked it out from pixels. The example form is `polyvore_qwen_15`, on the query "bohemian festival style" — the same query the paper uses for Figure 3, its qualitative comparison of R4T against Qwen3-4B zero-shot. Extract the two 3000×1200 grids from the PDF, cut each into its ten 600×600 tiles, and correlate every tile against the thirty cells of Figure 3.
Result 1's first six tiles are six near-copies of that Coachella/FRINGE collage. They match the Qwen3-4B rows at a normalized cross-correlation of **0.87** and the R4T rows at **0.30-0.44**. Nine of Result 1's ten tiles land on a Figure 3 cell above 0.81. Result 1 also contains 18 near-identical tile pairs out of 45 — it is the redundant set. Result 2 matches *no* Figure 3 cell above 0.80 and has **zero** near-identical pairs. So Result 1 is the zero-shot fan-out baseline and Result 2 is an R4T variant. Given that the study omits groundedness — the one dimension R4T-Diffusion cannot be scored on — my best guess is Result 2 is R4T-Diffusion, but the repo does not say and I am not going to pretend it does.
Here is what the ratings say.
Per query, averaged over its six respondents, then compared across the 35 queries with an exact two-sided sign test on the non-tied queries:
| base model | metric | Result 1 | Result 2 | queries won | sign test |
|---|---|---|---|---|---|
| Gemma3-4B | alignment | **4.24** | 3.77 | 26 / 4 / 5 tie | p = 5.9e-05 |
| Gemma3-4B | diversity | 3.48 | **3.75** | 11 / 20 / 4 tie | p = 0.15 |
| Qwen3-4B | alignment | **4.45** | 3.73 | 31 / 3 / 1 tie | p = 7.7e-07 |
| Qwen3-4B | diversity | 2.74 | **3.26** | 11 / 23 / 1 tie | p = 0.058 |
The two arms split the metrics. One takes alignment decisively — 31 of 35 queries in the Qwen file — and the other takes diversity by a smaller, less certain margin. Table 1 has R4T ahead of zero-shot on *both* on Polyvore, in both families: Gemma alignment 31.2 → 37.6 and diversity 56.0 → 74.3, Qwen alignment 23.4 → 27.4 and diversity 37.0 → 65.0.
The nice thing about this finding is that it does not depend on my arm mapping at all. Whichever arm is R4T, it wins one metric and loses the other, and the paper's LLM judge has it winning both. If my mapping is right, R4T's alignment advantage inverts under human raters. If it is backwards, R4T's diversity advantage inverts instead. There is no assignment of labels under which these 419 ratings reproduce Table 1's pattern.
**Receipts.** The 419 human ratings in R4T's rebuttal repo do not reproduce the pattern of its Table 1. In both base-model families the two arms split the two metrics — one wins alignment decisively, the other wins diversity — where the paper's LLM judge has R4T ahead on both. The alignment gap is the larger and the more certain of the two.
| base model | metric | Result 1 mean | Result 2 mean | queries won (R1/R2/tie) | sign test p | dedup p |
| :--- | :--- | ---: | ---: | ---: | ---: | ---: |
| Gemma3-4B | alignment | 4.243 | 3.771 | 26 / 4 / 5 | 5.9e-05 | 9.1e-04 |
| Gemma3-4B | diversity | 3.481 | 3.748 | 11 / 20 / 4 | 0.15 | 0.34 |
| Qwen3-4B | alignment | 4.446 | 3.729 | 31 / 3 / 1 | 7.7e-07 | 1.4e-06 |
| Qwen3-4B | diversity | 2.736 | 3.259 | 11 / 23 / 1 | 0.058 | 0.035 |
Which arm is which system is not stated anywhere in the repo. I infer Result 1 = zero-shot baseline from pixel matching against the paper's Figure 3 (see the article). The split-decision finding does not depend on that inference: whichever arm is R4T, it wins exactly one of the two metrics, and the paper's Table 1 has it winning both. Independence is weaker than 6-per-query suggests: the median form carries 4 distinct score vectors, not 6, and 34 (Gemma) / 41 (Qwen) consecutive within-form submissions repeat the previous row's four scores less than 10 seconds later — the fastest pair 0.07 s apart.
> method: github.com/pat-jj/r4t-rebuttal ships two spreadsheets of raw Google Forms responses: 35 forms x 6 respondents for Gemma (210 rows) and 35 forms x 6 (one form has 5) for Qwen (209 rows), each row rating Result 1 and Result 2 on the paper's own Alignment and Diversity rubrics, 1-5. Per query I take the mean of the responses for each arm, then compare the two arms across the 35 queries with an exact two-sided sign test on the non-tied queries. Two cells are blank (one per file, both in form 11) and are dropped pairwise. 'dedup' repeats the whole calculation after collapsing each form's identical score vectors to one.
> source: https://github.com/pat-jj/r4t-rebuttal
> captured: 2026-09-18
> data: https://ai.thesatyajit.com/articles/retrieve-for-train/data/human-eval.json (4 rows)
Three honest caveats, and the first one cuts against my own use of this data as much as anyone's.
**The six respondents per query are not six independent opinions.** The median form carries four distinct score vectors, not six. In 13 of the 35 Qwen forms, all six responses collapse to three or fewer distinct vectors. Thirty-four (Gemma) and forty-one (Qwen) consecutive within-form submissions repeat the previous row's four scores in under ten seconds — the closest pair 0.07 seconds apart, which is not two people reading a form. The effective sample is closer to four per query than six. I re-ran everything after collapsing identical vectors within each form; the alignment result survives comfortably (Qwen p = 1.4e-06, Gemma p = 9.1e-04) and the Gemma diversity result stays insignificant either way.
**Level and direction are different claims.** Because Table 1's 0-100 scale is never defined, I am comparing the *direction* of the two evaluations, not their magnitudes. Direction is the whole finding, so this does not weaken it — but nobody should read a 4.45 against a 23.4 as a gap.
**This is a rebuttal artifact, not a published result.** I do not know what question the authors were answering, what they concluded, or whether v1 of the paper predates it — it does, by 25 days. Running a human study during rebuttal and publishing the raw responses is more transparency than most authors offer. The problem is that nobody, including the authors, appears to have gone back and reconciled it with Table 1, and the arXiv record was never updated.
## What I would actually take from this
The compiler framing is the durable part, and I think it generalizes. Any objective you can compute but not differentiate — set diversity, coverage, a business metric, a simulator's verdict — can in principle be pushed through a policy once and harvested as supervised targets for something small and fast. That is a real pattern, cleanly demonstrated here, and the reward-hacking ablation is the evidence that the hard part is specifying the reward rather than the compiling.
What I would not take from it is any specific number without going back to the figure first. The 20x does not exist in the data. The single pass is 256 passes. The shipped model is unscored on the largest term of its own objective, the judge is named twice with two different names, and the human study that would settle the judge question is sitting in a repo the paper does not link to, reversing one of its two headline directions.
None of that is misconduct. All of it is the ordinary gap between what a paper measures and what its abstract says, widened at each retelling until the aggregators are writing "20x" as a fact. The figure was always there. It is an SVG; the numbers are right in the file.
A v2 of the paper with a corrected Figure 5, or an author statement that the
figure was generated from a different run than the prose. My reading is
checkable in about ten lines of Python against the published SVG — if my
gridline calibration is wrong, the same calibration would not reproduce both
autoregressive numbers the paper states in words, and the zoom inset would
not independently agree to three decimals. Both do.
A README, a commit message, or an author confirming the opposite. The
evidence is a 0.87 correlation to the Qwen3-4B rows of Figure 3 against
0.30-0.44 to the R4T rows, plus 18 of 45 near-identical tile pairs within
Result 1 and none within Result 2. Note that overturning this does not
rescue Table 1 — it swaps which of the two metrics the human raters reverse.
A mapping from the 5-point Likert scale to Table 1's 0-100 numbers under
which both evaluations put the same system ahead on both metrics. No such
mapping exists for a monotone transform — the ratings put different arms
ahead on the two metrics and Table 1 puts one arm ahead on both — but if the
35 rebuttal queries are not the OAR test queries, or the two arms are not the
two systems Table 1 compares, the comparison is void.
A measured FLOP or energy count for both paths. My 331 GFLOP versus ~1.2
TFLOP is a back-of-envelope at 2 FLOPs per parameter per token, assuming
~150 generated tokens for the autoregressive baseline — a number the paper
never reports, and the one my estimate is most sensitive to. If the FOLM's
{""} blocks run to 800 tokens, the arithmetic gap alone
reaches the wall-clock gap and this claim is wrong.
A code release, or an appendix stating a reward threshold. Right now §2.4 and
Figure 1 say no filter and the introduction says "filtering successful
trajectories"; I am siding with the figure, which draws the property check in
step 1 and omits it from step 2. A released synthesize.py with a
threshold in it settles this against me in one line.