EAGLE-4: speculation that prices itself, and where the 5.21x comes from

On 20 October LightSeek, a day-0 engine partner, posted: "EAGLE-4 is here—a game changer for agentic coding, with massive decoding speedups." Its pitch: "Train with TorchSpec. Serve with TokenSpeed." The replies quoted 5.21x back, and asked the right questions: is that only Qwen on B300s, and does acceptance hold when the drafter sees tool output?

Three primary sources, and they do not report the same numbers. The EAGLE-4 paper and the eagle-hq/EAGLE-4 repository come from Eagle AI (@EagleCorp on X; the paper's contact is EAGLE co-author Hongyang Zhang). The Eagle launch post summarises the paper. LightSeek's post covers training in TorchSpec and serving in TokenSpeed. I read all three, the harness and TokenSpeed source, and the drafter checkpoints' safetensors headers (range requests, no weights downloaded).

Every number below is labelled: measured (I computed it from a file or from code), reported (the publisher's figure, not re-run), or reasoned (my arithmetic on the other two). I ran no benchmark: the Qwen drafter alone is 6.18 GB and its target is a 27B model, so nothing on a CPU would say anything about B300 speed.

Speculation in one paragraph

A decode step of a 27B model at batch 1 is memory-bound: the GPU streams every weight to produce one token. Speculative decoding has a small drafter guess several tokens ahead. The target then checks all of them in one forward pass and keeps the longest prefix it agrees with, plus one bonus token of its own. Verification is lossless by construction. For greedy decoding the kept prefix is exactly what the target would have produced. For sampling, rejection sampling (accept $\hat x_i$ with probability $\min(1, p_i(\hat x_i)/q_i(\hat x_i))$, resample from the residual on rejection) preserves the target's distribution. The EAGLE-3 article walks through it; how LLM inference works covers the memory-bound step.

So the speedup is a ratio of two things:

$A$ is the acceptance length: tokens advanced per round, bonus token included. The denominator is what a round costs: one verify forward, $k$ draft steps, and bookkeeping. $t_\text{plain}$ is one ordinary decode step. If each drafted token survives with probability $a$, independently, along a chain of $k$, then $A = (1 - a^{k+1}) / (1 - a)$. That is a geometric series: token $i$ counts only if all tokens before it were accepted. Two consequences follow, and EAGLE-4 is built around both.

1. Acceptance length alone is not speed. A deeper draft always raises $A$, but each extra step costs $t_\text{draft}$ and adds less than the one before.
2. The cost side moves with load. At batch 1, verifying 16 tokens costs about what decoding 1 does. At batch 512, verify is compute-bound, and every speculative token competes with real work.

The EAGLE lineage

Each EAGLE generation raised the numerator.

• EAGLE-1 (arXiv 2401.15077, January 2024) drafts at the feature level. A one-layer drafter reads the target's second-to-top hidden state and predicts the next feature, which the target's own LM head turns into a token. It also reads the embedding of the token one position ahead, so it knows which branch it is on. The tree was fixed.
• EAGLE-2 (arXiv 2406.16858, June 2024) made the tree dynamic. A node's score is the product of the draft probabilities along its path. That cheaply estimates the chance the target accepts it.
• EAGLE-3 (arXiv 2503.01840, March 2025) dropped feature regression for direct token prediction. It fuses low, middle and high target layers, and it trains with training-time test: the drafter learns from its own multi-step outputs, so deep drafts look at training time the way they will at inference. Draft quality began to scale with data. EAGLE-3.1 (Eldenk et al., arXiv 2605.09992) added post-normalisation and per-hidden-state RMSNorm against what it calls attention drift.

None of them priced the denominator. EAGLE-2's tree is sized by confidence, not by what the hardware charges for it. That is the gap EAGLE-4 fills.

What EAGLE-4 changes

expand

The drafter itself barely changes. EAGLE-4 keeps EAGLE-3.1's normalised single layer. I checked that against the released Qwen3.8-27B drafter's config.json and its safetensors header (measured):

Summing the layers.0.* tensors gives 0.435B parameters. The paper's Table 1 says 0.44B. The same count on the public DFlash 2 and DSpark drafters gives 1.665B and 1.652B across five layers each (measured), against the paper's 1.66B and 1.65B. The paper's "about 3.8x fewer FLOPs per drafted token" is 1.66 / 0.44 = 3.77 (reasoned). One thing the paper's table leaves out by design: the whole drafter file holds 3.09B parameters, and 82% of them are the 248,320 x 5,120 embedding and LM head (measured). The layer is small. The vocabulary is not.

That vocabulary is the first real change.

The Efficient LM Head

A draft step that scores 248,320 tokens spends most of its time on the head. In the paper's profiled configuration the full head is 65.4% of a 574 µs draft forward (reported). EAGLE-3 truncated the draft vocabulary (32K tokens in LightSeek's comparison), which cannot propose a token outside it.

EAGLE-4 splits the job instead. A rank-128 factorisation $\tilde z = A(Bh)$ scores the whole vocabulary cheaply and picks the top 128 candidates. The full head then computes exact logits for just those 128 rows, $z_C = W_C h$. The low-rank head only retrieves; the scores stay exact. Per token that is about 33M multiply-adds instead of 1,271M (reasoned from the tensor shapes above).

expand

The paper reports a 55.6% cut. It also says a plain truncated-SVD initialisation at rank 128 lost 0.48 tokens of acceptance. Training the two factors for retrieval alone, with the rest of the drafter frozen, puts the full head's top-1 token in the 128-token shortlist 98.2% of the time, against 91.5% for SVD (reported). On TokenSpeed, LightSeek measured the head's share of draft GPU time at 66% with the full head and 19% with the efficient one for chains, 67% and 18% for trees (reported). That means a draft step costs about (1 − 0.19) / (1 − 0.66) = 2.38x less with the shortlist head (reasoned). That factor is the "full LM head" switch in the calculator below.

EAGLE-Harness: price every step

The second change is the one in the title. The paper's rule is one inequality applied at three points of every round:

$\Delta A$ is the expected extra accepted tokens, from EAGLE-2's path-probability product. $\Delta C$ is the extra time, from a table profiled once per hardware setup across batch sizes, context lengths and token counts.

1. After each draft depth, stop if the next level is not worth its draft cost. The threshold rises with depth because deep estimates are less reliable.
2. Once per round, add nodes to the verify set, best path score first, while each one is worth its marginal verify cost.
3. The load sets the shape: up to eight branches per node when lightly loaded, at most four under load.

The repository has the constants (measured, eagle_harness/core/cost_model.py): a base threshold of 2.5, a 4% rise per depth (depth_loss_ratio = 0.04), multiplied by a batch-size ratio from 1.25 for small batches to 2.0 for huge ones. Its device policy decides every other depth: draft, launch, decide. That matches LightSeek's note that a host-side policy would need a device-to-host round trip every other depth.

At temperature above 0, stopping on the tokens drawn in this round would bias the output, so a sampled round drafts to the depth where the previous round's rule would have stopped.

The widget runs a toy version of that round. The thresholds are the repository's; the costs above one request and the draft probabilities are mine and illustrative. Push the confidence up, as in repetitive code, and the one-request tree runs eight to eleven levels deep and sends 15 nodes to the target. Drop it to open-ended prose and the tree stops at depth five or six. Switch to 64 requests and the same rule builds a chain of one to four tokens; at 512 it drafts one token and verifies it if it is likely enough. That is the paper's main argument against block drafters like DFlash 2 and DSpark: a block drafter has already paid for its whole block before it can decide to verify less of it. An autoregressive drafter decides before each step; DSpark adapts too, but only by trimming the verify prefix of a block already drafted.

What the GPU actually does in a round

LightSeek's best evidence is a measurement of a single round.

expand

The decision kernel runs on the GPU and writes the next width and a stop flag to device memory, so the host never reads a score. Each depth is its own CUDA graph, with per-round values held as GPU scalars, and each ends in a one-thread kernel that tail-launches the next depth or the finalize graph through CUDA device graph launch. The host launches a round once and never waits inside it. LightSeek is candid that this is worth 0% to 1.5% over its optimised host pipeline (reported); the win is simpler graph management.

The timeline lets me check the headline arithmetic. LightSeek reports 985 tok/s greedy at 3.56x over no speculation, with an acceptance length of about 4.75 (reported). So plain decoding on that node runs at 985 / 3.56 = 276.7 tok/s, or 3.614 ms a token (reasoned). The decode graph that verifies 16 rows costs 3.828 ms, only 6% more than a single plain step. That is the memory-bound regime in one number. A median round of 4.553 ms that advances 4.75 tokens predicts 4.75 x 3.614 / 4.553 = 3.77x (reasoned). The reported 3.56x is 6% lower. Plausible: five depths is the most common count, not the only one, and a whole-run average includes what a median round leaves out. The same model with an acceptance length of about 3.6 predicts 2.88x at temperature 1, against 2.85x reported (reasoned). The headline is consistent with its own timeline.

The calculator is that arithmetic with knobs. Costs are in plain decode steps: verify $v = 3.828 / 3.614 = 1.059$, a draft step $c = 0.123 / 3.614 = 0.034$, and 0.022 of overhead for the finalize and the 20 µs gap (reasoned). The per-token acceptance $a$ in each preset is a chain-equivalent fit chosen to reproduce the reported acceptance length, not a measurement. Tick "full LM head" on the greedy preset: speed falls from 3.80x to 3.20x, which is the case for the shortlist head in one checkbox. Then pick "Busy server", where verify and draft are no longer nearly free, and watch the best depth move left. Its costs are illustrative; neither post publishes per-round costs at high load.

Checking the speedup claims

expand

Four things are worth knowing before quoting any of these.

The paper's 3.59x and LightSeek's 3.56x are not the same measurement. The paper runs one engine with MegaKernel disabled and a vanilla baseline of 240.5 tok/s. TokenSpeed's no-speculation baseline is 276.7 tok/s (reasoned), 15% faster. EAGLE-4's absolute speed went up from 863.8 to 985 tok/s, but the ratio stayed put because the baseline improved too. A faster engine shrinks every speculative speedup it reports. That is the right way to measure it.

The MTP baseline matters. The paper's 1.62x over MTP uses a 7-1-8 chain. LightSeek says that chain is about 5% slower on TokenSpeed than MTP's best, a 5/1/6 chain, and against the best chain EAGLE-4 is 1.28x at T=1 and 1.37x greedy (reported). So the honest margin over a model's own MTP head on a tuned engine is about 1.3x, not 1.6x.

The paper's headline numbers are greedy. At T=1 the paper's own Qwen row drops to 2.59x at one request and 1.03x at 512 (reported). At 512 sampled requests EAGLE-4 is barely ahead of not speculating. LightSeek lists sampled chains for high concurrency under "Looking ahead".

The shipped package measures less than the paper. The repository's support matrix for eagle-harness 0.1.2 reports 3.35x on SGLang, 2.83x on vLLM and 3.13x on TokenSpeed at one request, falling to 1.38x, 1.19x and 1.40x at 64 (reported). It also flags SGLang on one GPU at 64 requests as 0.73x, slower than not speculating, and lists it as a known limit. Different subsets and token caps explain part of the gap, but the matrix is what pip install gets you.

Why agentic coding is the best case

LightSeek's agentic workload replays SWE-smith multi-turn sessions. The first turn carries about 50K tokens of context, each later turn adds about 800, each produces 500 output tokens, and the prefix-cache hit rate is roughly 90% (reported).

expand

Why should code agents be the best case? Three reasons, all reasoned rather than measured by anyone:

• The output repeats the context. Quoting a file it just read, re-emitting a function with one line changed, naming arguments seen ten turns ago: long runs a drafter reading the target's hidden states can predict. High per-token acceptance is where the geometric series stops decaying.
• Long context is cheap for this drafter. Its attention is a 4,096-token sliding window with append-only draft KV, so its cost stays flat while the context grows to 50K. LightSeek's long-context table shows 3.08x at 32K and 3.42x at 250K (reported), and time to first token at 250K is 5.83 s with EAGLE-4 against 5.79 s without.
• An agent is often one user. A developer driving a coding agent is batch 1 on the decode side most of the time, which is where verification is nearly free.

How much acceptance does 5.21x need? If the round costs the same 1.25 plain steps as on SPEED-Bench, 5.21x needs about 6.5 accepted tokens per round, and about 7 if the drafter goes eight deep and the round gets longer (reasoned; the calculator's agentic preset is that second case). LightSeek does not publish the agentic acceptance length, so I cannot check it. On the reply asking whether acceptance drops on tool output: nobody has published a per-turn breakdown, but a 5.21x average is hard to reach if tool-call turns accept badly.

The caveats are all in LightSeek's own text. 5.21x is greedy and one user. With sampling at temperature 1 the same workload gives 3.04x. At 16 concurrent greedy requests it is 1.94x per user, and EAGLE-4 produces 4.1x to 1.9x as many tokens per GPU across that range (reported). For smaller setups the paper reports 3.49x on one AMD MI350X and 2.71x averaged over four benchmarks on a 4-vCPU c5.xlarge with 4-bit Gemma-4-E2B-it (reported); neither is agentic coding.

Throughput at high concurrency

At 256 to 512 requests the question becomes how few tokens to waste. TokenSpeed's answer is --speculative-chain-width-table: 4 verified rows from 64 requests and 3 from 384, the '64:4,384:3' example in the flag's help text (measured, server_args.py). The same text says a batch the table covers drafts "without the confidence rule", and the default is off. At high load on TokenSpeed, width is a table calibrated per workload, not the harness deciding each round. LightSeek says width selection plus the table took 256 requests from 7.2K to 15.6K tok/s, and that its final stage reached 31.9K tok/s at 512 (reported).

What it costs

• Licence. The eagle-harness code and both drafters are under the Eagle 4 Source-Available License, "not an OSI-approved open-source license", with a separate commercial agreement. TokenSpeed is MIT; the DFlash 2 and DSpark checkpoints are Apache-2.0.
• Memory. The Qwen drafter is a 6.18 GB BF16 file (measured), mostly embedding and head.
• Bit-exactness. "Lossless" holds in exact arithmetic. The support matrix's greedy check gives "5 exact / 15 near tie" on twenty prompts for SGLang, a near tie being a log-probability gap of at most 0.13 at the first divergence. Expect the same distribution, not the same bytes.
• Profiling. Costs are measured once per hardware setup. The support matrix shows a start with an empty cost cache at 2.53x against 2.99x warm on SGLang at 32K (reported).

The take

EAGLE-4's idea is not a better drafter. The drafter is EAGLE-3.1 with a sliding window and a smarter head. The idea is that $A$ was always only half of the ratio. The harness prices each draft step and each verified node against a measured table, so one trained drafter builds a 12-deep tree for a single user and a two-token chain for a busy server. The Efficient LM Head makes each step cheap enough that the tree is worth growing. The paper's ratios, its parameter counts and LightSeek's round timeline all check out.

The 5.21x is real and narrow: one greedy user on Qwen3.8-27B, four B300s, a SWE-smith replay. For planning, use the support matrix: about 3x for one request on a tuned engine, about 2x at eight, about 1.4x at 64.

Sources: the EAGLE-4 paper and repository at commit c10363b, Eagle's launch post, LightSeek's post, TokenSpeed at 12b7fa1, and the drafter checkpoints' config.json and safetensors headers. Related on this site: Kimi K3 on GB300, where a forced acceptance length decides the headline, and Step 5 Preview, whose sliding windows are the same trick at target scale. Figures are the papers' and posts'; the two interactives are mine.