# Set Diffusion: one knob from autoregression to diffusion

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/set-diffusion
> date: 2026-07-03
> tags: llm, diffusion, language-models, inference-optimization, explainer
There's a spectrum hiding behind two families of language model that usually get treated as
opposites. **Autoregressive** models generate strictly left-to-right, one token per step —
sequential, but they support the [KV cache](/articles/how-llm-inference-works) that makes serving
cheap. **Diffusion** LMs ([like iLLaDA](/articles/illada-diffusion-language-model)) denoise many
tokens in parallel and in any order — flexible, but fixed-length, and they *can't* cache (every step
needs full bidirectional context). **Set Diffusion** (Arriola & Kuleshov, Cornell — the block
diffusion authors) makes the spectrum explicit and shows you can slide along it with a single idea:
change *which token sets you decode together, and in what order.*

<Spectrum />

## The spectrum, precisely

The prior bridge between these worlds was **block diffusion** (BD3-LM): generate fixed-size
contiguous blocks left-to-right, with diffusion inside each block. That buys variable length and a
per-block KV cache, but the block is rigid — it can only extend left-to-right, so no infilling, and
the cache can only update once a whole block finishes decoding (the within-block denoising needs
bidirectional context until then).

<Figure
  src="/articles/set-diffusion/fig1.png"
  alt="Two decoding traces over wall-clock time. Left, Block Diffusion fills fixed sequential blocks and only updates the KV cache after each block completes. Right, Set Diffusion reveals flexible-position, position-biased token sets and updates the KV cache after every step, decoding mostly left-to-right but filling later positions when likely."
  caption="Set diffusion decodes flexible-position token sets and refreshes the KV cache after every step, while block diffusion is locked to fixed sequential blocks that can only update the cache once a whole block finishes (paper, Figure 1)."
/>

Set Diffusion's move is to stop thinking in blocks and think in **sets**. A *set* is an
arbitrary-position, arbitrary-length subset of token positions you decode together. Factorize the
likelihood over a sequence of disjoint sets that cover the whole sequence, and the familiar models
fall out as special cases:

- **Autoregression** — every set is a single token, in left-to-right order.
- **Order-agnostic diffusion** — one set of the whole sequence, decoded in random order.
- **Block diffusion** — fixed contiguous blocks, left-to-right.

They're not different architectures; they're different **set schedules** for the same object. That's
the whole conceptual payoff — and once you see it, the interesting question is how to pick a schedule
*between* the corners.

## Two knobs, one window

Set diffusion exposes two knobs the block-size parameter conflates: **set size** (how many tokens you
commit per step — parallelism) and **ordering bias** (how left-to-right you stay — quality). In
practice both are controlled by one schedule parameter, a window width `w`: each position gets an
active generation window of width `w`, and the widths determine how much decoding overlaps.

<WindowKnob />

The paper makes the endpoints rigorous. As `w → 1/L` the windows stop overlapping, tokens generate
one at a time in order, and the training objective *becomes the tight autoregressive ELBO* — best
perplexity, no parallelism. As `w → 1` every position shares one schedule and you recover
order-agnostic diffusion — maximally parallel and any-order. Set diffusion lives in between: a sliding
window that decodes a few tokens per step, mostly in order but flexible enough to fill gaps. Smaller
`w` buys perplexity; larger `w` buys parallelism and any-order decoding. One dial, the whole spectrum.

<Figure
  src="/articles/set-diffusion/fig2.png"
  alt="Four panels of per-token reveal-time CDFs for a length-4 sequence, plotting the probability a token has been revealed against normalized ordering time. AR shows fully staggered step CDFs (strict left-to-right); two sliding-window SetDLM panels show progressively overlapping ramps as the window widens; MDLM shows all tokens sharing one linear schedule (order-agnostic)."
  caption="Per-token reveal-time CDFs for L=4: the decoding window w slides the ordering bias from strict left-to-right AR, through sliding-window set diffusion, to fully order-agnostic MDLM diffusion (paper, Figure 3)."
/>

## Why sets get to keep the KV cache

The systems win is that generation is **set-causal**: each set attends to itself and to all
*previously decoded* sets, but not to future ones. Because the ordering across sets is causal,
finished sets never need reprocessing — their keys and values are cached and reused, and the cache
**updates after every inference step**. That's the thing pure diffusion can't do (it needs full
bidirectional context, so nothing is ever "final" enough to cache) and the thing block diffusion does
only *per block* (bidirectional context *within* the block blocks earlier caching). The ablation is
stark: turn the causal mask and KV caching off and GSM8K accuracy collapses from **26.6 to 6.4** while
throughput drops too — the causal set structure is buying both.

The flexibility also gives **infilling** for free. Because sets are flexible-position, the schedule can
select gap tokens and condition them on the clean tokens on *both* sides — something block diffusion's
strict left-to-right blocks structurally cannot do.

## The numbers

At GPT-2-small scale (110M params), the headline is that set diffusion beats block diffusion on *both*
axes at once — accuracy and speed. On GSM8K it tops the whole diffusion field:

<BenchBars
  title="GSM8K — 0-shot pass@1 (%)"
  unit="%"
  bars={[
    { label: "SW-SetDLM (S≤8)", value: 66.41, highlight: true },
    { label: "BD3-LM (block, S=4)", value: 63.53 },
    { label: "BD3-LM (block, S=8)", value: 56.94 },
    { label: "BD3-LM (block, S=16)", value: 50.49 },
    { label: "MDLM (diffusion)", value: 6.37 },
  ]}
/>

<Figure
  src="/articles/set-diffusion/fig3.png"
  alt="Speed-accuracy scatter for GSM8K: x-axis decoding throughput in tokens per second, y-axis accuracy. Filled markers for set diffusion (S<=8 and S<=32) sit above and to the right of the open block-diffusion markers (S=4 and S=16) across the frontier, while a single AR star sits highest in accuracy at low throughput. An arrow labels the up-and-right direction as improved tradeoff."
  caption="On GSM8K, set diffusion (filled markers) traces a strictly better speed-accuracy frontier than block diffusion (open markers), while a plain autoregressive model still tops accuracy at low throughput (paper, Figure 5)."
/>

— and it does so at higher throughput than any of those block-diffusion settings (60.4 vs 55.4
tok/s). It won't be lost on you that an ordinary AR transformer scores higher still (75.7) and, at this
small scale with full diffusion sampling steps, is even a bit faster; the point of a diffusion LM isn't
to beat AR on a left-to-right benchmark, it's to keep AR's efficiency *while* offering any-order
generation. Which is exactly where the second result lands — **infilling**, filling a gap given text on
both sides, where set diffusion clearly beats block diffusion:

<BenchBars
  title="ROCStories infilling — ROUGE-1 (fill 3 of 5 sentences)"
  unit=""
  bars={[
    { label: "SW-SetDLM (S≤32)", value: 18.1, highlight: true },
    { label: "BD3-LM (block, S=16)", value: 15.8 },
  ]}
/>

At ~25% faster decoding, no less. Across the rest of the suite it's the same shape: on OpenWebText
it matches block diffusion's perplexity at **22% higher throughput** (and runs ~13× faster than
cacheless MDLM), on LM1B it posts the best diffusion perplexity *and* the highest diffusion throughput,
and on CNN/DailyMail it's competitive on ROUGE at up to 10% faster. A strictly better speed-quality
frontier than block diffusion, plus the infilling block diffusion gives up.

## The take

What I like about Set Diffusion is that it's a *reframing* that pays off, not a new mechanism bolted
on. "Interpolate between AR and diffusion by varying block size" was already a good idea (that's block
diffusion); the insight here is that block size was the wrong knob — the right one is the **order in
which token sets are generated**, and once you factorize over flexible sets instead of rigid blocks you
get a strictly larger design space that still contains AR, still contains diffusion, and adds a
KV-cacheable, any-order, infilling-capable middle that block diffusion couldn't reach.

The honest caveats: everything is at 110M parameters, an AR model still wins the straight
left-to-right benchmarks, and the ideal window schedule is currently hand-tuned (learning it is future
work). But as a clean statement of *what the AR↔diffusion spectrum actually is*, and a practical model
that sits usefully in its middle, it's the most satisfying diffusion-LM paper I've read since block
diffusion itself — which makes sense, given it's the same group closing the loop on their own idea.

---

*Built on [Set Diffusion: Interpolating Token Orderings Between Autoregression and Diffusion](https://arxiv.org/abs/2607.01775)
(Marianne Arriola, Volodymyr Kuleshov; Cornell, ICML 2026), which generalizes the same authors'
[Block Diffusion](https://arxiv.org/abs/2503.09573). Code and weights are at
[kuleshov-group/setdlms](https://github.com/kuleshov-group/setdlms). Benchmark figures are quoted from
the paper's tables (110M-parameter models).*