Set Diffusion: one knob from autoregression to diffusion

There's a spectrum hiding behind two families of language model that usually get treated as opposites. Autoregressive models generate strictly left-to-right, one token per step — sequential, but they support the KV cache that makes serving cheap. Diffusion LMs (like iLLaDA) denoise many tokens in parallel and in any order — flexible, but fixed-length, and they can't cache (every step needs full bidirectional context). Set Diffusion (Arriola & Kuleshov, Cornell — the block diffusion authors) makes the spectrum explicit and shows you can slide along it with a single idea: change which token sets you decode together, and in what order.

The spectrum, precisely

The prior bridge between these worlds was block diffusion (BD3-LM): generate fixed-size contiguous blocks left-to-right, with diffusion inside each block. That buys variable length and a per-block KV cache, but the block is rigid — it can only extend left-to-right, so no infilling, and the cache can only update once a whole block finishes decoding (the within-block denoising needs bidirectional context until then).

expand

Set Diffusion's move is to stop thinking in blocks and think in sets. A set is an arbitrary-position, arbitrary-length subset of token positions you decode together. Factorize the likelihood over a sequence of disjoint sets that cover the whole sequence, and the familiar models fall out as special cases:

• Autoregression — every set is a single token, in left-to-right order.
• Order-agnostic diffusion — one set of the whole sequence, decoded in random order.
• Block diffusion — fixed contiguous blocks, left-to-right.

They're not different architectures; they're different set schedules for the same object. That's the whole conceptual payoff — and once you see it, the interesting question is how to pick a schedule between the corners.

Two knobs, one window

Set diffusion exposes two knobs the block-size parameter conflates: set size (how many tokens you commit per step — parallelism) and ordering bias (how left-to-right you stay — quality). In practice both are controlled by one schedule parameter, a window width w: each position gets an active generation window of width w, and the widths determine how much decoding overlaps.

The paper makes the endpoints rigorous. As w → 1/L the windows stop overlapping, tokens generate one at a time in order, and the training objective becomes the tight autoregressive ELBO — best perplexity, no parallelism. As w → 1 every position shares one schedule and you recover order-agnostic diffusion — maximally parallel and any-order. Set diffusion lives in between: a sliding window that decodes a few tokens per step, mostly in order but flexible enough to fill gaps. Smaller w buys perplexity; larger w buys parallelism and any-order decoding. One dial, the whole spectrum.

expand

Why sets get to keep the KV cache

The systems win is that generation is set-causal: each set attends to itself and to all previously decoded sets, but not to future ones. Because the ordering across sets is causal, finished sets never need reprocessing — their keys and values are cached and reused, and the cache updates after every inference step. That's the thing pure diffusion can't do (it needs full bidirectional context, so nothing is ever "final" enough to cache) and the thing block diffusion does only per block (bidirectional context within the block blocks earlier caching). The ablation is stark: turn the causal mask and KV caching off and GSM8K accuracy collapses from 26.6 to 6.4 while throughput drops too — the causal set structure is buying both.

The flexibility also gives infilling for free. Because sets are flexible-position, the schedule can select gap tokens and condition them on the clean tokens on both sides — something block diffusion's strict left-to-right blocks structurally cannot do.

The numbers

At GPT-2-small scale (110M params), the headline is that set diffusion beats block diffusion on both axes at once — accuracy and speed. On GSM8K it tops the whole diffusion field:

expand

— and it does so at higher throughput than any of those block-diffusion settings (60.4 vs 55.4 tok/s). It won't be lost on you that an ordinary AR transformer scores higher still (75.7) and, at this small scale with full diffusion sampling steps, is even a bit faster; the point of a diffusion LM isn't to beat AR on a left-to-right benchmark, it's to keep AR's efficiency while offering any-order generation. Which is exactly where the second result lands — infilling, filling a gap given text on both sides, where set diffusion clearly beats block diffusion:

At ~25% faster decoding, no less. Across the rest of the suite it's the same shape: on OpenWebText it matches block diffusion's perplexity at 22% higher throughput (and runs ~13× faster than cacheless MDLM), on LM1B it posts the best diffusion perplexity and the highest diffusion throughput, and on CNN/DailyMail it's competitive on ROUGE at up to 10% faster. A strictly better speed-quality frontier than block diffusion, plus the infilling block diffusion gives up.

The take

What I like about Set Diffusion is that it's a reframing that pays off, not a new mechanism bolted on. "Interpolate between AR and diffusion by varying block size" was already a good idea (that's block diffusion); the insight here is that block size was the wrong knob — the right one is the order in which token sets are generated, and once you factorize over flexible sets instead of rigid blocks you get a strictly larger design space that still contains AR, still contains diffusion, and adds a KV-cacheable, any-order, infilling-capable middle that block diffusion couldn't reach.

The honest caveats: everything is at 110M parameters, an AR model still wins the straight left-to-right benchmarks, and the ideal window schedule is currently hand-tuned (learning it is future work). But as a clean statement of what the AR↔diffusion spectrum actually is, and a practical model that sits usefully in its middle, it's the most satisfying diffusion-LM paper I've read since block diffusion itself — which makes sense, given it's the same group closing the loop on their own idea.

Built on Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion (Marianne Arriola, Volodymyr Kuleshov; Cornell, ICML 2026), which generalizes the same authors' Block Diffusion. Code and weights are at kuleshov-group/setdlms. Benchmark figures are quoted from the paper's tables (110M-parameter models).