DFlash 2: the drafter already knew the answer, it just picked the wrong one

2026-08-23 · 10 min · speculative-decoding · inference · block-diffusion · serving · explainer

There is one number in DFlash 2 that makes the rest of the post feel inevitable.

At the first position of a draft block, a five-layer DFlash drafter's top pick is correct 85.4% of the time. The correct token is somewhere in its top sixteen candidates 99.5% of the time.

The drafter is not failing to know the answer. It is failing to choose it. And the candidates it needs are already sitting in a tensor it already computed.


Who	Inco AI · code at z-lab/dflash · 18 Aug 2026
What	two additions to parallel block drafting: a pairwise path selector and a two-tap dynamic convolution
Cost	+1.3% draft–verify cycle latency, ~3% more drafter parameters
Gain	+1.05 accepted tokens per pass over DFlash (21%), +0.48 over DSpark
Losslessness	greedy output matches the target exactly; sampling preserves its distribution
Out today	drafters for Qwen3.8-27B and Meta's Muse Glimmer
Throughput	2.7–3.4× autoregressive on Qwen3.8-27B, 3.1–4.6× on Muse Glimmer

z-lab/Qwen3.8-27B-DFlash2hugging face · snapshot 2026-09-08

parameters: 1.92B
repo size: 3.85 GB
architecture: DFlash2DraftModel
license: apache-2.0
downloads: 262.1K
likes: 270
files: 5

parameters by dtype

BF16 1.92B

The setup, briefly

Speculative decoding exists because decode is memory-bound: the cost of a forward pass is dominated by streaming weights out of DRAM, not by the arithmetic. If a cheap drafter guesses the next k tokens, the target model can verify all of them in one pass, amortizing that weight load across however many turn out to be right.

For years the drafter itself stayed autoregressive — one token at a time, which is a strange thing to accept when the entire premise is that sequential decoding is the problem. DFlash, released in January and now in SGLang, vLLM, TensorRT-LLM and llama.cpp, made the draft one-pass too: every position of the block predicted in parallel, block-diffusion style. NVIDIA measured up to 15× throughput with it on Blackwell, Google reported 3× more tokens per second on TPUs, and CoreWeave's production Kimi K2.7 Code endpoint runs it by default. Meta ships one with Muse Glimmer, as do Poolside, Xiaomi and NVIDIA with their own models.

Predicting in parallel buys the speed. It also creates the problem this post is about.

Every pick is plausible; nothing makes them fit

When each position is predicted independently, each token is reasonable on its own terms and nothing coordinates them. The characteristic failure is a stutter: two neighbouring positions both independently deciding the most likely next word is decoding, producing "good for decoding decoding". Verification kills the block at the first mismatch, and the whole tail is discarded.

S_t(a, b) = U_t(b) + ⟨A(a) ⊙ H(h_t), B(b)⟩3 of 3 drafted tokens survive · 4 emitted this pass

The pairwise term prices decoding → decoding at −1.6 and speculative → decoding at +1.1, so the walk gives up 0.05 of logit at position 0 to buy 2.7 across the pair. The whole block survives and the pass emits four tokens.

The thing to notice is what the selector does not do. It never asks the backbone for another forward pass, never touches the LM head again, and never rewrites a full-vocabulary distribution. It scores adjacent pairs of tokens that were already computed, all of them at once, and then walks the result. All the parallelism survives; only the final walk is sequential, and it is a walk over numbers already in registers.

That is why it costs two million parameters and 0.6% of cycle latency while beating a sequential correction head that costs seventy-eight million and 9.6%. The candidates were always there. Nobody had bothered to connect them.

Recent methods buy coherence by bolting a sequential head onto the drafter — DSpark and Domino both rewrite each position's full-vocabulary distribution conditioned on the token before it. That works. It also reintroduces exactly the autoregressive step the parallel design was built to remove.

DFlash 2's question is whether that is necessary, and the answer comes from measuring the drafter's own candidate lists.

five-layer Qwen3-4B DFlash · GSM8K · conditional on every earlier position being rightacceptance 4.27

selector qualitytop pick

DFlash — take each position's top pick

4.27

+ DSpark-style sequential correction (+77.8M params, +9.6% latency)

4.49

+ DFlash 2 pairwise path selector (+2.0M params, +0.6% latency)

4.61

oracle: always pick correctly from the top 16

6.79

your slider

4.27

The two dashed lines are the entire argument. At the first draft position the top pick is right 85.4% of the time — but the right token is somewhere in the top sixteen 99.5% of the time. The drafter is not failing to know the answer. It is failing to choose it.

Turn those rates into acceptance length and the gap is 4.27 against 6.79, more than two and a half tokens per verification pass sitting in a shortlist the drafter already computed. DFlash 2 claims about 0.34 of it with two million parameters. DSpark’s sequential rewrite claims 0.22 with seventy-eight million. Choosing really is cheaper than predicting — and the oracle says most of the headroom is still there.

Turn those conditional recall rates into acceptance length — a block is accepted up to its first mistake, plus the verifier's own token — and the top-pick row gives 4.27 while the top-16 oracle gives 6.79. Two and a half tokens per pass of pure selection headroom, requiring no new predictions at all.

So DFlash 2 keeps the top 16 candidates at every position and scores every adjacent pair:

S_t(a, b) = U_t(b) + \langle A(a) \odot H(h_t),\; B(b) \rangle

The first term is DFlash's own logit for b. The second asks how well b follows a: A and B give each token a compact 256-dimensional embedding, matched under a context gate H(h_t) that decides which parts of the match count — a low-rank bilinear attention over adjacent candidates. Every pair at every position is scored in one shot, with no extra backbone or LM-head pass. The only sequential work left is a walk over precomputed scores: greedy follows the best successor, sampling draws from the same scores, and rejection sampling restores the exact target distribution.

A left-to-right pipeline diagram. On the left, a context of tokens ending in the target-decoded token 'for', followed by three mask tokens. These enter the DFlash2 Backbone: five layers of attention and MLP with dynamic short convolutions before and after each, then the target LM head, producing drafted candidates. Those feed the Parallel Path Selector, drawn as a trellis of top-16 candidate boxes at positions one, two and three with a green path traced through one candidate per column, labelled 'top-16 per position, associative scan to one path'. On the right, the emitted sequence: for, speculative, decoding, end-of-sequence. — The whole design. The backbone drafts a block in one pass with two-tap convolutions inside each layer; the selector traces one coherent path through the candidates it produced. (Inco AI, Qwen3.8-27B-DFlash2 model card.)

Suffix decay is a backbone problem, and a local one

The selector cannot fix everything, and the recall table says why: even the oracle decays, from 99.5% at the first position to 87.8% at the last. No amount of choosing helps when the candidates themselves have run out. That is a backbone problem.

Depth fixes it — a fifteen-layer drafter holds the end of the block far better than a five-layer one. But the curves are nearly identical at position zero, which means ten extra attention blocks are adding capacity everywhere including where none was needed, at 15.2% more cycle latency. Indiscriminate.

The targeted fix comes from looking at where the drafter's attention actually goes. It has two jobs: read the context before the block, and model dependencies inside it. And it progressively abandons the second — within-block attention mass falls from 30% in layer 1 to 8% in layer 5, and what remains concentrates in a shrinking handful of heads.

Qwen3-4B Recall@1 on GSM8K at T=0 · conditional on every earlier positiona kernel reaching one token back ≈ ten more layers

DFlash 5L−12.53 ptsthe shipped baseline

DFlash 15L−7.69 pts3× the parameters · +15.2% cycle latency

DFlash 5L + two-tap conv−8.22 pts+3% parameters · +0.7% cycle latency

the diagnosis — share of attention the drafter spends inside its own block

30%

26%

20%

11%

After the convolution is added, the average across layers 4 and 5 falls from 9.4% to 0.5% — attention hands the local job over and goes back to reading context.

Look at position zero first: all four curves are within a point of each other there. Everything the extra depth buys, it buys at the end of the block — which is the tell that the problem was never general capacity. Attention in a parallel drafter is doing two jobs, and by layer five it has almost stopped doing the second one.

So the fix is a two-tap kernel: each position mixes its own representation with its predecessor’s, before and after every attention and MLP sublayer, with coefficients that adapt to the content. Blocks are four to sixteen tokens long and the tight dependencies are between neighbours, so reaching exactly one position back recovers most of what ten extra Transformer layers buy — at 3% of the parameters and a twentieth of the latency. Everything still computes in parallel; the convolution is block-local and stateless, so it drops in without touching attention, the LM head, or verification.

So split the jobs. A block is only 4 to 16 tokens long and the tightest dependencies are between neighbours, so the natural operator is a very short convolution: two taps, one on the current position and one reaching a single position back, with content-adaptive weights.

\operatorname{Conv}_k(x)_t = k_{t,0} \odot x_t + k_{t,1} \odot x_{t-1}

One sits before and after each attention and MLP sublayer of every drafter layer. Each coefficient combines a learned base kernel with a small correction from the current hidden state, shared across every 16 channels. The first position reads the last verified token's representation; every later position reads its predecessor's. Information crosses the block while all positions still compute in parallel — the convolution is block-local and stateless, so it drops in without touching attention, the LM head, or verification.

With 16.5M added parameters (3%), five layers plus convolution lands on the fifteen-layer curve. And afterwards the average within-block attention across layers 4 and 5 falls from 9.4% to 0.5% — attention hands the local job over and goes back to reading context, which is a satisfying confirmation that the diagnosis was right rather than merely a lucky architecture change.

What the two together are worth

On Qwen3.5-4B, mean acceptance length across five benchmarks: MTP 4.54, DFlash 4.92, DSpark 5.49, DFlash 2 5.97. That is +1.05 tokens over DFlash — 21% — and +0.48 over DSpark, for a combined 1.3% of cycle latency. Position by position on MATH-500, DFlash 2 holds near 86% all the way to the fifteenth draft position while every baseline finishes 6 to 9 points below it.

The two shipped drafters tell the same story against each model's official speculation path. On Qwen3.8-27B, DFlash 2 averages 4.80 against the model's built-in MTP at 4.28 and a community DSpark drafter at 3.62. On Muse Glimmer, 5.70 against the official DFlash drafter Meta ships at 4.44.

Why acceptance length is not a leaderboard statistic

Here is where I would push back on how these results usually get read — and, to be fair, where the model card is more honest than the blog post.

Qwen3.8-27B · SGLang on one H200 · seven draft tokens per step8 of 15 cells slower than no speculation

GSM8Kautoregressive 1,329.8 tok/s

1,381.11.04×

1,506.51.13×

1,922.51.45×

MATH-500autoregressive 1,505.8 tok/s

1,415.60.94×

1,429.00.95×

1,951.81.30×

HumanEvalautoregressive 1,546.5 tok/s

1,296.80.84×

1,330.10.86×

1,799.01.16×

MBPPautoregressive 1,507.7 tok/s

1,314.90.87×

1,361.30.90×

1,886.81.25×

MT-Benchautoregressive 1,507.4 tok/s

1,159.70.77×

1,115.50.74×

1,525.31.01×

Qwen3.8's built-in MTP · accepts 4.28community DSpark drafter · accepts 3.62DFlash 2 · accepts 4.80no speculation at all

Start at concurrency 1, where every speculative method looks like a straightforward win, then step to 32. Four of MTP’s five tasks and four of DSpark’s fall below the dashed line: at that batch size the speculation is costing more compute than the memory traffic it saves, and turning it off would be faster. MT-Bench is the worst cell — MTP at 0.77×, DSpark at 0.74×.

Nothing about that is a bug. Batching already amortizes the weight loads that speculation exists to amortize, so as arithmetic intensity climbs, verifying seven tokens to keep three or four is simply waste. Which is why acceptance length is not a leaderboard statistic but the thing that decides whether the technique still applies at all: DFlash 2 carries a full extra token per pass over DSpark on this model, and that is the entire difference between staying above water at 32 and not.

At batch size 1, every speculative method is a clear win and the ranking barely matters. At concurrency 32, four of MTP's five tasks and four of DSpark's are slower than not speculating at all. MT-Bench is the worst cell: 0.77× for MTP, 0.74× for DSpark. Turning the feature off would make those servers faster.

None of that is a bug. Batching already amortizes the weight loads that speculation exists to amortize, so as arithmetic intensity climbs, verifying seven tokens to keep three is just waste. Which reframes what an extra accepted token per pass is for: not a better score, but the difference between the technique still applying at your serving concurrency and not. DFlash 2 stays above water in all fifteen cells. It is the only one that does.

What I would want to know next

The oracle is still at 6.79. Pairwise scoring is, by the authors' own description, the simplest selector they could think of, and it claims about 0.34 of a 2.5-token gap. A trigram term, a wider beam, or a learned scan over three-token windows all seem obviously worth trying — and the fact that a low-rank bilinear form over adjacent pairs gets this far mostly suggests nobody has looked hard yet.

The 16 is a hyperparameter nobody varied. Top-16 per position at block size 8 means 16 × 8 pairwise blocks to score. There is no ablation on that width in the post, and it is the one knob that trades selector cost against how much of the oracle is reachable at all. Recall@16 is 99.5% at position 0; what is Recall@4, and does the cheaper selector get most of the same lift?

The convolution's win is measured on one family. The two-tap kernel closing the gap to fifteen layers is a strong result, but it is shown on Qwen3-4B, and "suffix decay is local" is an architectural claim about how much within-block dependency a drafter needs. Models with different block sizes — Muse Glimmer runs 16, twice Qwen3.8-27B's 8 — are exactly where a one-position reach should start to strain.

Lossless is doing real work in the pitch, and it should be checked. Greedy output matching the target exactly is a property of the verification rule, not of the drafter, so it holds by construction. Sampling preserving the target distribution is a stronger claim, resting on rejection sampling over the selector's scores rather than over the drafter's raw logits. The post asserts it; I would like to see the distributional test.

The line I keep coming back to

Choosing is cheaper than predicting.

That is a general principle wearing a speculative-decoding costume, and it applies well beyond drafters. A model that has already computed a distribution over candidates has done the expensive part. The mistake is treating the argmax as the answer when the distribution was the answer — and then, having thrown away the rest, paying a second full model to reconstruct what you discarded.

DFlash 2 gets an extra token per verification pass for 1.3% more latency because it stopped throwing the shortlist away. In seven months DFlash went from a paper to three and a half million downloads and an ecosystem of vendor-shipped drafters; the sequel is a two-million-parameter bilinear form and a two-tap kernel. Inference really is nowhere near its floor.