Speculative decoding on AMD GPUs: verifying the hedge

Most speculative-decoding posts lead with a number: 6× here, 2.5× there. AMD and Embedded LLM's vLLM writeup — 242 minutes of reading time, five drafting methods, nine target models, two AMD Instinct GPU generations — leads with a hedge instead:

Speculative decoding allows vLLM to verify multiple drafted tokens in a single target-model pass. In our experiments, its effect on output-token throughput varied across drafting methods and proposal lengths, and also depended on the model family, draft checkpoint, workload, and acceptance behavior.

That is an unusual sentence for a vendor blog post to lead with. It reads like a disclaimer bolted onto a marketing page. It also happens to be exactly correct, and the appendix backs it with per-model, per-workload, per-proposal-length tables — which means the hedge is checkable rather than merely diplomatic. I read the whole thing, pulled the raw numbers out of its interactive appendix, cross-checked every headline figure below against those tables at least twice, and fetched the actual released draft checkpoints from Hugging Face to verify a few architecture claims independently rather than take the prose's word for them.

The vocabulary, from the post's own worked example

Every number below is one of three things: a throughput ratio (generated tokens per second, against a non-speculative baseline on the same hardware and prompt set), a mean accepted length (MAL — how many tokens get committed per verification round, on average), or an acceptance rate (AR — what fraction of proposed tokens survive). The post ties them together with a small worked example:

expand

Three committed tokens from one verification pass — two drafted, one from the target — is the entire mechanism. Every method below is a different answer to how the draft component decides what to propose, and that answer turns out to matter more than anything else in this post.

Five ways to draft, drawn as what they are

The post's own comparison figure is worth reproducing directly, because it is the cleanest single image in a 242-minute post:

expand

The three sequential methods differ only in what feeds the sequential loop: native MTP reuses the target model's own auxiliary path and hidden state; Gemma 4 MTP is a separately-checkpointed assistant that shares the target's KV cache; EAGLE-3 is a dedicated draft network conditioned on fused early/mid/late target hidden states, feeding its own output back into itself. DFlash breaks the pattern entirely — one forward pass predicts every masked position in the block at once, using target-derived features as extra key/value context in every draft layer rather than as a one-time input. DSpark keeps DFlash's parallel backbone but bolts on a lightweight left-to-right "Markov head" that adjusts each position's logits using the token chosen immediately before it, buying back some of the coherence a fully independent parallel prediction loses.

Put real numbers on that mechanism split and a pattern shows up immediately: on gemma-4-31B-it, the two sequential methods with checkpoints available (Gemma 4 MTP, EAGLE-3) hit their peak acceptance well before their peak throughput proposal length — Gemma 4 MTP's best tested setting was only N=4, and it still reached 85.2% acceptance, the highest of any method tested on this model. DFlash needed N=7 to reach its own peak, at a much lower 68.0% acceptance — it is trading acceptance rate for proposal length, and the parallel backbone's block prediction still comes out ahead on raw throughput (2.34× vs 2.12–2.20×) because verifying seven candidates per round beats verifying four or five, even at a lower hit rate per candidate.

The case where it actively loses

Here is the finding the post's hedge is protecting, stated as plainly as the appendix states it. On Qwen3-8B with EAGLE-3 on MATH500, throughput is below the non-speculative baseline at every single tested proposal length:

Baseline is 3,530 tok/s. Even the best tested setting, N=7, tops out at 88% of the speed of just not speculating. And this is not a low-acceptance failure — at N=1 the draft's single proposed token is accepted 89% of the time, which on paper sounds like a strong drafter. It loses anyway, and loses worst at the shortest setting: 0.44× at N=1 is a 56% throughput cut, worse than doing nothing.

The reason is baseline speed, not drafting quality. Qwen3-8B is a small dense model whose non-speculative decode is already fast — 3,530 tok/s on MATH500, the highest baseline in the entire study alongside its own GSM8K run at 3,698. EAGLE-3 adds a real forward pass through its own draft network every round, plus the verification overhead, and at N=1 that fixed cost buys back only one extra candidate token — not enough to amortize against a baseline this quick. The same model's EAGLE-3 numbers on GSM8K, HumanEval, and MBPP do clear baseline once N climbs past 3 or 4, so it is not that EAGLE-3 fails on Qwen3-8B categorically; MATH500's longer, more circuitous reasoning traces apparently give the draft network less to work with per round even while its single-token acceptance stays high. This is precisely the shape of finding a single aggregate number would hide: "EAGLE-3 gives up to 2.27× on this model" (true, for GSM8K) and "EAGLE-3 never beats baseline on this model" (also true, for MATH500) are both accurate statements about the same method on the same target model, and the difference is entirely the workload.

It isn't the only place this shows up, either — it's the extreme end of a pattern. DSpark on the same Qwen3-8B/MATH500 pair opens at 0.73× (N=3) and only reaches 0.96× at its longest tested setting (N=15), never clearing baseline. Native MTP on Qwen3.6-35B-A3B, a completely different model family, opens at 0.87× on HumanEval and 0.89× on MBPP at N=1 — a sequential method, on a mixture-of-experts model, failing at the shortest possible proposal length for the same underlying reason: one extra draft step's fixed cost isn't recovered by one extra candidate token when the baseline itself is fast.

Same method, same benchmark, 2.6× of spread across models

DFlash has the widest model coverage of any method tested here — eight of the nine targets — which makes it the fair one to hold fixed while varying the target model:

The Qwen3.6 pair is the cleanest illustration in the whole post of why "drafting method X gives Y×" is an incomplete sentence. Qwen3.6-27B is dense; Qwen3.6-35B-A3B is a mixture-of-experts model in the same generation, same lab, same naming scheme. DFlash's peak on the 27B dense model is 1.59× (N=11); on the 35B-A3B MoE it's 2.06× (N=7) — and the post says this outright: "The difference from the Qwen3.6-27B measurements shows that results can vary between models in the same family." A sparse model activates a smaller fraction of its parameters per token, which narrows the per-token compute gap between the tiny draft network and the target — exactly the kind of interaction a single leaderboard number cannot represent.

Qwen3-8B sits at the bottom for a related reason to the EAGLE-3 case above: it's the fastest non-speculative baseline in the study (3,226–3,698 tok/s across its four workloads), so there is the least available headroom for any drafting overhead to hide inside. gemma-4-26B-A4B-it, at the top, is a mixture-of-experts model whose 26B total parameters mask a much smaller active-parameter compute cost per token, similar to the Qwen3.6-35B-A3B story — plus DFlash's fused-context drafter here draws from a real, published checkpoint (z-lab/gemma-4-26B-A4B-it-DFlash) rather than an internal one, so the comparison is against exactly what a reader could download and run.

Proposal length has a peak, and parallel methods can crash past it

The post's tuning-considerations section says a larger proposal window "may improve throughput" but that "acceptance may decrease at later draft positions... causing throughput to flatten or regress." That is a real, measured curve, not a caveat — Qwen3.5-122B-A10B has full sweeps for both a sequential method (native MTP, N=1 through 7) and a parallel one (DFlash, N=3/7/11/15) across all four of its benchmarked workloads:

The shapes are genuinely different, not just scaled versions of each other. Native MTP's curve is concave and monotonic across the tested range on all four workloads — each additional proposed token buys less than the last, but it never gives any of it back, because a sequential drafter only ever pays for the tokens it actually proposes. DFlash pays a fixed verification cost for the entire block it committed to, whether the tail of that block turns out to be right or not, so its curve rises to a peak around N=7 and then declines — on HumanEval and MBPP, past baseline into a net loss by N=15. Reading the per-position acceptance rows in the post's appendix explains why: DFlash's acceptance at position 1 sits around 90% on this model, and by position 15 it has fallen under 20%. Verifying fifteen candidate positions when only the first four or five are likely to survive is pure waste — drafting compute and verification-pass tokens spent on a suffix that is going to be discarded anyway.

This is also a place the post's own "match the sweep to the workload" advice is directly demonstrated rather than merely asserted: DFlash's peak sits at N=7 on GSM8K, MATH500, and HumanEval for this model, but MBPP's curve dips at N=7 (1.05×) before recovering at N=11 (1.34×) — a noisier, less clean sweet spot than the other three, and a real reason to benchmark a target proposal length on the actual workload rather than copy a setting from a different dataset's sweep.

DSpark is tested with its own headline feature turned off

This is the finding in the post that's easiest to miss unless you go looking for it, and it's an example of exactly the discipline this site tries to apply — not restating a claim, but checking what it's actually built on. Buried in the DSpark section is one sentence:

I pulled DeepSeek's own DSpark paper to see what that confidence head is actually for. Its abstract doesn't treat it as a footnote — it's the headline: "DSpark employs confidence-scheduled verification, dynamically tailoring the verification length for each request based on estimated prefix survival probabilities... Compared to the established production baseline (MTP-1), DSpark accelerates per-user generation speeds by 60 to 85 percent at matched throughput levels." That production number, from DeepSeek-V4 under live traffic, is exactly the mechanism this ROCm benchmark disabled. And it isn't a checkpoint limitation — I pulled the actual released config for deepseek-ai/dspark_qwen3_8b_block7 from Hugging Face, and it carries "enable_confidence_head": true and "confidence_head_with_markov": true right in the checkpoint metadata. The head exists in the weights this benchmark loaded; the serving path just wasn't wired to use it. vLLM even published a companion post on exactly this feature eight days before this one — "Adaptive Verification in vLLM: DSpark confidence-scheduled verification" — which means the throughput numbers for DSpark in this post are testing the weaker half of a method whose stronger half the same team wrote about separately. The site's own DSpark piece covers that confidence-scheduled verifier in full; the numbers in this article's charts are DSpark with it switched off.

The DFlash paper makes a similarly bold claim worth the same treatment: "over 6× lossless acceleration... up to 2.5× higher speedup than... EAGLE-3." The one place in this post where DFlash and EAGLE-3 are benchmarked on the same target, dataset, and hardware — gemma-4-31B-it, MATH500 — DFlash reaches 2.34× against EAGLE-3's 2.12×. That's a real edge, about 10% (2.34 divided by 2.12), not 2.5×. Nothing here contradicts the paper — it's near-certainly measuring a different model, task mix, and possibly a different reference point for "EAGLE-3" — but it's a clean demonstration of why a paper's own headline number and a downstream serving benchmark on a specific target model rarely land on the same figure, and why this post's hedge is the more trustworthy of the two documents.

One more independently-checkable detail: Gemma 4 MTP's separate assistant checkpoint, google/gemma-4-26B-A4B-it-assistant, is a real download — I pulled its config from Hugging Face. It's 4 hidden layers, hidden size 1024, about 420M parameters total, next to a 26B-A4B (mixture-of-experts, ~4B active) target. That's the actual size gap behind "Gemma 4 uses a separately packaged MTP draft component" — a ~420M-parameter assistant riding along with a target two orders of magnitude larger.

Training a speculator you don't already have

The post explicitly doesn't go deep here — "this guide does not cover speculator training in depth" — and points to three external resources instead. It also doesn't ship a figure for this section, unlike the mechanism comparison above, so here's the workflow it describes, as a diagram:

The three hidden-state collection modes are a real three-way tradeoff, not boilerplate: online generation avoids ever writing a large cache to disk but competes for the same GPUs training needs; offline generation frees every GPU for training but demands the storage up front; hybrid pays the generation cost exactly once, on the first epoch, then reuses it. The post is specific that what gets collected differs by method — EAGLE-3 wants hidden states from a few selected layers for autoregressive drafting, DFlash and DSpark want target features to condition a block predictor, and native MTP training isn't speculator training at all — it fine-tunes the target's own MTP component, so it only works if the target already ships compatible MTP layers to begin with.

The one number I could ground independently here comes from the actual released z-lab/Qwen3-8B-DFlash-b16 checkpoint rather than the post's prose: its config lists target_layer_ids: [1, 9, 17, 25, 33] against Qwen3-8B's 36 layers, and block_size: 16 — matching the post's own worked tuning example ("when block_size = 16, the maximum proposal length is normally num_speculative_tokens = 15") exactly, independently, from the checkpoint metadata rather than the text describing it.

What I'd take from this

The hedge holds up. Every specific number the post's summary leads with — 2.87× for DFlash on gemma-4-26B-A4B-it, 2.83× for Gemma 4 MTP on the same target, 2.68× for DFlash on Kimi-K2.5, 2.20× as the ceiling for native MTP across the three Qwen3.5/3.6 models where it was tested — checked out against the appendix tables on the first and second pass. So did the quieter half: Qwen3-8B's EAGLE-3 numbers never clearing baseline on MATH500, DSpark's confidence head sitting unused in a checkpoint that ships with it enabled, and a 2.6× spread in DFlash's own peak ratio across nine models that are all, notionally, running "the same method."

None of that is a knock on speculative decoding, or on AMD's ROCm stack specifically — the site's own pieces on EAGLE-3, DFlash 2, and DSpark show real, reproducible gains for each method on its own terms, and the vLLM engine deep-dive covers why the v1 scheduler treats speculative decoding as one more consumer of a shared token budget rather than a bolted-on mode. What this post adds is the part those pieces don't: what happens when you hold the serving stack and hardware fixed and vary the target model nine times. The answer is that a drafting method's throughput number is a function with at least four inputs — model family, workload, proposal length, and which half of the checkpoint's own features you actually turned on — and reporting only the best cell you found is the one move this post, to its credit, declined to make.