Interference Search: the 23 out of 30 has no language model in it

mdjsonmcpcopy↗ai

2026-09-26 · 20 min · llm · reasoning · tree-search · verifiers · evaluation · benchmarks · explainer

Bad Theory Labs has been in these pages three times. BTL-3 was a LoRA adapter, honestly labelled. BTL-4 had a benchmark table that did not close. Tinfield 1 was 1.85% of a model. Each time the useful move was to stop reading the announcement and read the artifact.

This time there is more artifact than ever. On 2026-09-25 at 14:13 UTC, @Badtheorylabs posted:

On the same hard problems, a 1.7B model thinking the normal way solved 3 out of 30. Interference Search(our new architecture)solved 23.

It describes the method as "many paths at once, the ones that meet merge into one, the dead ends cancel out, and everything left moves forward together", and adds: "We watched the normal model find the right answer at token 1,313, check it 9 more times, wander off and run out of budget without ever answering. Ours got there in 3 steps." It links nothing and ends "More technical details soon."

They came 72 minutes later. The repository Badtheorylabs/interference-search was created at 15:25 UTC with a six-page paper, Apache-2.0 code, the trained judge, the raw results, a research log that keeps its failures, and a paper/CLAIMS.md mapping each number to a file. The dossier is on the lab's site. It is the best-documented thing this lab has shipped. The post is the least careful sentence in it.

What the 3 and the 23 are

"Out of 30" invites AIME. It isn't. The 30 are Countdown puzzles: four numbers and a target; combine all four with +, −, × and ÷, each used once, every intermediate result a positive whole number. The repository's generator makes them "hard" by rejecting any instance where a single operation on two of the numbers lands within 25 of the target, and keeping only instances with at most two distinct solutions. The paper's running example is [24, 98, 19, 3] → 361, solved by (24 × 19) − 98 + 3.

The two numbers in the post come from different rows of the paper's Table 4 (reported):

The judge's size is my arithmetic on interference_search/judge.py: two layers, width 64, four heads, 29 features per number, 102,145 parameters (reasoned; the README says "100k" and the 419,573-byte weight file agrees). The model is mlx-community/Qwen3-1.7B-4bit.

The repository says so plainly:

Those are different systems: the fast one uses the environment to list moves and a 100k-parameter judge to rank them, and never runs the language model.

So does the paper: "The strongest Countdown numbers use a small trained judge and an environment that lists moves; they do not run the language model." The post drops that sentence, and what is left credits a Countdown solver's win over a 1.7B model to an architecture.

Between the 3 and the 23 the paper has a ladder. Each rung keeps the search and changes only the judge.

Asking the model of each state "can you reach exactly the target? Answer Yes or No." and ranking by P(yes) solves 4 (reported, AUC 0.58). The same width-6 frontier with a shuffle for a judge solves 14.4% of 30 other hard problems from the same generator in expectation, about 4.3 of 30 (measured). Asking the model is indistinguishable from not asking it.

A linear probe on the model's hidden state carries real signal: 15 (reported, AUC 0.87). A linear probe on plain number features, trained on the same 1,376 states, does better, with 22 (AUC 0.94). The author ran that control himself and printed what it did to his reading: "I had first read the probe result as the model knowing more than it can say; the feature control does not support that reading." The trained set transformer gets 23. The same judge at width 16 and 200 judged positions gets 30.

So the model contributes nothing to the 23. Its best showing in Countdown is 15 of 30, as features under a probe, reading about 4,290 prompt tokens per problem and writing none. That took 15.2 s a problem on an M2 against 57.4 s for thinking in text (reported; that timing run of the text arm solved 1 of 30, not 3).

How Interference Search works

expand

A domain supplies a start state, a proposer, a merge key, a judge, a goal test and a dead-end test. Each round of interference_search/core.py:

1. Expands every live state. In Countdown the environment lists every legal move, for free.
2. Executes. A branch holds the state it reached, not a description of it. A child equal to the target ends the search, also free.
3. Merges. Children with the same key (the sorted multiset of numbers) become one entry, pooling their parents' ranks as votes of 1/(rank+1). Keys already expanded are dropped.
4. Judges. Every merged state is scored, and each score costs one unit of budget.
5. Advances. The best W move down a level together. W is the budget over three times the number count: 16 at a budget of 200 on four numbers.

The baseline, "one line of thought", chains the same parts: judge the children of the current state, sample one with probability proportional to its score squared (temperature 0.5), continue, and restart when the chain dies.

In classical terms this is beam search over a transposition table with a learned value function. The research log says so: "a value-ranked frontier is close to value-guided beam search, which is prior art". The paper names transposition tables (Zobrist, 1970) and AlphaZero's value network as its roots and claims the combination and its ablations as new. The "interference" is steps 3 and 4. Nothing cancels like an amplitude; a merged state is judged once, and a dropped state takes every path through it along.

Here is the loop on the showcase problem, with the judge's own recorded scores:

The first level has 19 distinct states, 2 of them alive. At width 12 the judge keeps 12, whose 110 moves merge into 82 distinct states, and only one, 95 · 456, can finish (measured; my enumerator reproduces the lab's trace exactly). The judge ranks it 4th of 82 and the third level finds 456 − 95 = 361. The recorded judge solves this problem from width 4, a random order in about 27% of shuffles at width 12, the exact solver at width 1.

That also explains "Ours got there in 3 steps". Four numbers take exactly three moves to become one, so a frontier that succeeds without restarting always reports 3. Latency tracking depth rather than mistakes is the design point, but 3 is a property of the puzzle, not a speed.

expand

What "the same budget" buys at four numbers

The paper's headline equal-compute claim is Figure 3 and Table 3. With the same judge and 200 judged positions per problem, the frontier solves 30 and the line solves 21. The line needs 1,500 positions to solve all 30, and then takes 23.7 sequential steps on average against the frontier's 3.

expand

Read the unit. Only judging costs; listing moves and checking the goal are free, and the budget is checked at the top of each round. On four numbers it never stops the frontier. It sets the width.

How big is the space? I enumerated every state of the 30 hard problems whose numbers the lab published in its Phase 0 and Phase 1 raw results: same generator as the paper's 30, different seed. The paper's own 30 are not listed, though they can be regenerated. A hard four-number problem has 114 distinct non-terminal states on average, from 73 to 151, on 555 complete move sequences, 4.6 of which reach the target (measured; "at most two solutions" counts distinct expressions, which can be reached in several orders).

A breadth-first pass over every merged state, with no judge at all, touches 72 to 150 states and solves every problem. The 200-position budget is larger than the whole search space. At that budget the frontier judges the entire first level and the children of 16 of its roughly 18 states, which is close to everything.

Here are the curves, with two references I ran through the same loop, cost unit and width rule:

Three things follow.

The judge works. With a random order the same frontier solves about 14 of 30 at 200 (measured, on the other 30); with the exact solver it solves all 30 from 80. The trained judge is at 23 by 80 and 30 by 200, a good value function for 102,145 parameters trained on smaller problems. How small can a verifier be? found a 0.63M-parameter model verifying Countdown at 0.85 for the same reason: the tell is local arithmetic in a short input.

The shape beats a weak baseline. A chain judges about 18 first-level children and 10 second-level ones per dive, so 200 buys it about seven weighted random dives (reasoned). Keeping 16 states per level of a 114-state tree beats seven dives. The comparison is matched and real. It is also the expected result, and it says nothing about language models.

Merging does nothing here. The benchmark JSON behind Table 3 also has a frontier with merging off. It solves 29 at 150 and 30 at 200 and 500, the same as Interference Search; the two differ only at 50, 7 against 4 (reported, results/countdown/countdown_benchmark.json). Four numbers leave little to merge: at the second level, 176 paths collapse into 96 states, 1.8x (measured; 179 into 97 on the paper's 20 random instances). The headline result is breadth plus a good judge. The part the method is named after is not in it.

Where the name is earned: six numbers

expand

At six numbers the picture changes, and this is the paper's strongest result. At 100 expansions the merged frontier solves 77%, the frontier without merging 50%, the line 40% and best-first search 10% (reported). Duplicates fill an unmerged frontier's width, and best-first keeps picking shallow states that look safe, which is the case for advancing level by level. At seven numbers the line catches up at large budgets (87% for both at 400), and the paper says so.

The figure the dossier puts next to this, "63x paths per state", needs a correction. Table 1's caption places it at "the deepest non-terminal level". But compression.py prints every level down to a single number, and 831,176 paths into 13,229 states is its last row: distinct final values, which the search never judges, only compares with the target. At the deepest level the judge sees, two numbers left, the lab's own file gives 264,496 paths into 16,479 states: 16.1x at six numbers, 4.8x at five and 1.8x at four (arithmetic on results/countdown/compression.txt). The ratio still grows with size, so the argument survives, with a number about a quarter the size.

The judge generalizes, within a boundary the paper states. Trained on four and five numbers, at a threshold of 0.7 it solves every random six-number problem at 9.3% of the work, but only 43% of hard ones (at most 24 solution paths). The safe setting there is 0.2: 98% at 17.9%. On ten seven-number problems it solves all ten at 7.9% (all reported).

The model finds answers; it does not commit to them

The post's anecdote checks out, with footnotes. In results/llm/cot_trace.json the model writes 17 complete attempts, 11 of them repeats. The first correct one is at token 1,313, and the correct expression appears 9 times in all, so "check it 9 more times" is 8 more. That trace ran to 2,048 tokens, not the 1,500 of the 3-of-30 run, and it is one sample on one of four showcase problems (measured).

The more useful data is the lab's Phase 0 and Phase 1, on hard problems from the same generator. In Phase 1, eight independent streams of 2,048 tokens ran on 24 problems while the harness scanned the text for a correct expression. One turned up on 22 of 24: 20 within 1,024 tokens per stream, 8 within the first 256 (measured from phase1_qwen3-1.7b.jsonl, arm A). In Phase 0, where samples had to commit an answer, 11 of 48 were right and 3 of 6 problems had at least one right sample (measured).

That is the baseline the post implies and the paper does not run on its 30: best-of-N with the same free checker the search gets. Majority vote is beside the point when answers are checked exactly; pass@k with the checker is the fair comparison. Phase 1 used eight times the tokens, so it does not settle it. It does say the model's failure here is committing and stopping, not searching, which is how the author reads his own trace.

Where the model does the proposing, nothing worked. On a six-problem smoke test the model choosing moves from an explicit state, with memory of dead states, solved 0. The log's conclusion: "a small model cannot be prompted into state-based search; its competence lives in its native thinking mode." The other negative results are the best part of the paper. Telling streams which expressions had failed primed them: 33.6% repeats within 80 tokens of a note, against about 21% without. Rewinding a stream that wrote a refuted expression regenerated it up to 16 times. Parallel streams that could attend to each other scored 0.511, against 0.508 for isolated ones.

Code: the only matched-budget test with the model proposing

In code the model does the proposing: each state is a program plus what it returns on each test, the interpreter executes, and programs that behave identically on the tests merge. The lab screened the first 150 MBPP problems, kept the 69 that Qwen3-1.7B fails on its first greedy try, and used 30 of them, with 1,500 generated tokens per problem (reported).

83% of the programs merged away as behaviour duplicates. Nine against eight is one problem in 30, and the paper says so: "In code the gain over independent sampling is within noise." Per-problem outcomes were not saved, so no paired test is possible. This is the only place "parallel thinking and execution" meets a language model at an equal token budget, and the margin is one problem.

The post's last claim, "Subagents were a terrible way to tackle this", has no experiment behind it. The word appears nowhere in the paper, the code or the log.

Where this sits in test-time compute

Inference compute goes three ways. Sample independent chains and vote or verify: self-consistency, best-of-N. Search partial solutions with a value function: Tree of Thoughts, RAP, verifier-guided beam search. Or train the model to branch and join itself: APR, which the paper says reports large gains over serial search on this same puzzle, and Parallel-R1. Stream of Search chose Countdown for the same reason this paper did: every state can be labelled exactly.

On Countdown, Interference Search is the second family with the language model removed from both proposing and judging. On code it is the second family with the model proposing and tests judging. Its distinctive part, merging states that different paths reach, pays when paths collide, which on these puzzles means six numbers, not four.

The closest precedent here is ChessLFM, where 408 of 552 Elo came from a minimax wrapper and a bug fix rather than training. Same shape: a search loop and a small value function do the lifting, and the headline credits the model. For the latent alternative the paper cites, several frontiers held inside continuous thoughts, see LOTUS.

Can anyone reproduce it?

Mostly, and that is to the lab's credit.

• Released: the library, every script, the judge weights, raw results for most claims, the log, paper/CLAIMS.md, a Dockerfile and scripts/reproduce_countdown.sh, which reruns every Countdown number on CPU in about ten minutes against the committed results. Those numbers are deterministic.
• Not on Linux: the language-model experiments use MLX and need Apple silicon.
• Gaps: the 3-of-30 text arm has no committed raw output (CLAIMS.md points at the script), and a later timing run with the same settings solved 1 of 30. The 22-of-30 feature probe has none either. Code outcomes were not saved per problem. Everything is one seed; the lab's own noise estimate is about ±2.5 problems on 30.

The log runs from 2026-09-22 to 2026-09-24. The whole project took three days.

What is good here

• The claim boundary is on the website: one seed, 30 problems, "the method is classical and claims no quantum speedup", "Nothing here trains the language model yet."
• The author overturned his own reading with a control he did not have to run, and printed both.
• The failures are kept, with numbers.
• The six-number ablation is a real finding. It isolates merging from level-by-level advance and shows each earning its keep.

The take

As published, Interference Search is value-guided beam search over a transposition table, measured carefully on a puzzle small enough to enumerate. The language model is in the headline and not in the search. The paper says so. The post doesn't.

The 3-to-23 comparison is not a result about reasoning. Before quoting it I would want two runs: pass@k for Qwen3-1.7B on the same 30 with the same free checker, at 1,500 total tokens and at 16,384; and an Interference Search arm in which the model proposes the moves. The author names his next step: train the model to reason inside the frontier, then test on SWE-bench and Terminal-Bench at matched compute. That is the experiment that would make the post's sentence true.

Sources: the post, read through fxtwitter because x.com returns 402 here; the paper and dossier; and the repository at commit afedcc2, whose paper/PAPER.pdf is byte-identical to the site's. All read on 2026-09-26; none of the lab's code was executed. Measured numbers come from my own Countdown enumerator, which reproduces the lab's recorded showcase traces state for state, and from the committed countdown_benchmark.json, compression.txt, probe_results.json, cot_trace.json, cot_timing.json, phase0_qwen3-1.7b.jsonl, phase1_qwen3-1.7b.jsonl and viz_slim.json. Timestamps are from the post, the GitHub API and the PDF metadata. The four figures are the paper's, redistributed under Apache-2.0 with the terms in NOTICE.txt; the three interactives are mine. Earlier reads of this lab: BTL-3, BTL-4 and Tinfield 1.