2026-07-20 · 6 min · inference · speculative-decoding · llm · systems · explainer
Autoregressive decoding is the tax every LLM pays: one forward pass, one token, and each pass is memory-bound — you stream all the weights through the chip to produce a single token. Speculative decoding is the trick that beats it. A small, cheap draft model guesses the next few tokens; the big target model verifies all of them in one parallel forward pass and keeps the longest prefix it agrees with. The output is provably identical to normal sampling — you just get several tokens per expensive pass instead of one. EAGLE-3 (Li et al.) is the current peak of the EAGLE family, and its contribution is subtle: it makes the draft model itself keep getting better with more training data. Up to 6.5× faster, losslessly.
Draft and verify
The whole game is acceptance: how often the target agrees with the draft's guesses. A token only counts if every token before it was accepted too, so its odds decay geometrically — which is why deeper drafts hit diminishing returns and the real lever is raising the acceptance rate. Drag it:
Acceptance is what everything hinges on. Because a token only counts if the whole prefix before it was accepted, its odds decay as α^i — so a deeper draft has sharply diminishing returns, and the real lever is pushing α up. That is exactly what EAGLE-3 does: a better-trained draft raises acceptance, and τ is (almost) the speedup. Verification stays exact — the target only ever emits tokens it would have sampled itself.
The mean accepted length τ — tokens produced per target forward pass — is essentially the speedup. Verification is exact: the target only ever emits a token it would have sampled itself (rejected tokens are resampled from the corrected distribution), so nothing about quality changes. Speculative decoding is a pure latency win with no accuracy cost.
What EAGLE-3 changes
EAGLE's draft model was clever: instead of predicting tokens directly, it autoregressively predicted the target's top-layer feature and reused the target's own LM head. But that came with a feature-prediction loss that constrained the draft — and the authors found that scaling up its training data barely helped. EAGLE-3 makes two changes.
First, it drops feature prediction for direct token prediction, and trains the draft the way it will actually run — a technique they call Training-Time Test (TTT). The problem it solves is concrete: EAGLE's first drafted token got accepted often, but its second collapsed, because at step two the draft feeds on its own step-one output, which drifts away from the features it was trained on. TTT folds that multi-step rollout into training so the draft learns to consume its own predictions.
Second, freed from the feature constraint, the draft fuses the target's low, middle, and high features instead of only the top layer — concatenated and projected down — giving it a richer basis for predicting tokens two and three ahead.
A scaling law for inference acceleration
Here's the payoff, and it's the reason the paper exists. With the feature constraint removed, the draft model's speedup keeps climbing as you give it more training data — a relationship never seen for EAGLE, which flatlines. EAGLE-3 was trained on roughly 8× more data than EAGLE, and the curve is still going up.
That reframes draft-model training as a data-scaling problem instead of a fixed architectural trick — the same lesson the rest of the field keeps relearning.
The numbers
On Vicuna-13B (greedy), EAGLE-3 averages 5.51× speedup across five tasks, peaking at 6.47× on HumanEval with an accepted length of 7.54 — a clear step over EAGLE-2, and multiples over Medusa and vanilla speculative sampling:
The part that matters for real serving is that the win survives batching. Most speculative methods degrade past batch ~16 (the extra draft compute stops paying off when the GPU is already busy); in SGLang, EAGLE-3 still delivers +38% throughput at batch 64, and at batch 1 it hits 373 tok/s vs 158 for plain SGLang — a 2.36× serving speedup. It's also compatible with EAGLE-2's dynamic draft tree, so the two stack.
From scratch: tiny-speculators
If you want to see the machinery without the framework, tiny-speculators
(junuxyz) is a from-scratch EAGLE-3 trainer, verifier = Qwen3-8B, in five clear stages: prepare
ShareGPT data → extract the verifier's early/mid/late/final hidden states via vLLM → train the draft
with the 3-step TTT rollout (using PyTorch FlexAttention for the non-causal TTT mask) → export
to vLLM's speculators format. The draft is a single decoder layer that fuses three hidden states
(projected 3H → H), concatenates the sampled token's embedding, and predicts the next token.
It's honest about being a small educational build. Its 60k-sample checkpoint on HumanEval reaches a 27.4% draft-token acceptance rate and a mean accepted length of 1.82, cutting single-request p50 latency from 2.61s to 1.78s — while openly noting that at high concurrency it stayed below plain serving. That gap between a from-scratch draft and the paper's 6.5× is exactly the value of the scaling law: acceptance is a data problem, and EAGLE-3's whole point is that more of it keeps helping.
The take
Speculative decoding was already the standard latency trick; the interesting move in EAGLE-3 is turning the draft model into something that scales. Drop the constraint that stopped it learning (feature prediction), train it on its own multi-step outputs (Training-Time Test), give it more of the target's internal features to look at, and its acceptance — and therefore the speedup — rises with data instead of plateauing. It pairs naturally with the site's pieces on multi-token prediction and DeepSeek's DSpark: the same idea — predict more per step, verify exactly — attacked from three directions.
Source: EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (Li, Wei, Zhang, Zhang) and the tiny-speculators repo. Figures are the paper's; the interactive is mine.