# EAGLE-4: speculation that prices itself, and where the 5.21x comes from

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/eagle-4
> date: 2026-10-23
> tags: speculative-decoding, inference, agentic-coding, benchmarks

On 20 October LightSeek, a day-0 engine partner, posted: *"EAGLE-4 is here—a game changer for agentic coding, with massive decoding speedups."* Its pitch: *"Train with TorchSpec. Serve with TokenSpeed."* The replies quoted **5.21x** back, and asked the right questions: is that only Qwen on B300s, and does acceptance hold when the drafter sees tool output?

Three primary sources, and they do not report the same numbers. The [EAGLE-4 paper](https://github.com/eagle-hq/EAGLE-4/blob/main/EAGLE4_Paper.pdf) and the [`eagle-hq/EAGLE-4`](https://github.com/eagle-hq/EAGLE-4) repository come from Eagle AI (`@EagleCorp` on X; the paper's contact is EAGLE co-author Hongyang Zhang). The [Eagle launch post](https://blog.eagle.ai/blog/introducing-eagle-4) summarises the paper. [LightSeek's post](https://lightseek.org/blog/eagle-4) covers training in TorchSpec and serving in [TokenSpeed](https://github.com/lightseekorg/tokenspeed). I read all three, the harness and TokenSpeed source, and the drafter checkpoints' safetensors headers (range requests, no weights downloaded).

Every number below is labelled: **measured** (I computed it from a file or from code), **reported** (the publisher's figure, not re-run), or **reasoned** (my arithmetic on the other two). I ran no benchmark: the Qwen drafter alone is 6.18 GB and its target is a 27B model, so nothing on a CPU would say anything about B300 speed.

## Speculation in one paragraph

A decode step of a 27B model at batch 1 is memory-bound: the GPU streams every weight to produce one token. Speculative decoding has a small **drafter** guess several tokens ahead. The **target** then checks all of them in one forward pass and keeps the longest prefix it agrees with, plus one bonus token of its own. Verification is lossless by construction. For greedy decoding the kept prefix is exactly what the target would have produced. For sampling, rejection sampling (accept $\hat x_i$ with probability $\min(1, p_i(\hat x_i)/q_i(\hat x_i))$, resample from the residual on rejection) preserves the target's distribution. The [EAGLE-3 article](/articles/eagle-3-speculative-decoding) walks through it; [how LLM inference works](/articles/how-llm-inference-works) covers the memory-bound step.

So the speedup is a ratio of two things:

$$
S = \frac{A}{t_\text{verify} + k\,t_\text{draft} + t_\text{other}} \cdot t_\text{plain}
$$

$A$ is the **acceptance length**: tokens advanced per round, bonus token included. The denominator is what a round costs: one verify forward, $k$ draft steps, and bookkeeping. $t_\text{plain}$ is one ordinary decode step. If each drafted token survives with probability $a$, independently, along a chain of $k$, then $A = (1 - a^{k+1}) / (1 - a)$. That is a geometric series: token $i$ counts only if all tokens before it were accepted. Two consequences follow, and EAGLE-4 is built around both.

1. **Acceptance length alone is not speed.** A deeper draft always raises $A$, but each extra step costs $t_\text{draft}$ and adds less than the one before.
2. **The cost side moves with load.** At batch 1, verifying 16 tokens costs about what decoding 1 does. At batch 512, verify is compute-bound, and every speculative token competes with real work.

## The EAGLE lineage

Each EAGLE generation raised the numerator.

- **EAGLE-1** ([arXiv 2401.15077](https://arxiv.org/abs/2401.15077), January 2024) drafts at the **feature level**. A one-layer drafter reads the target's second-to-top hidden state and predicts the next feature, which the target's own LM head turns into a token. It also reads the embedding of the token one position ahead, so it knows which branch it is on. The tree was fixed.
- **EAGLE-2** ([arXiv 2406.16858](https://arxiv.org/abs/2406.16858), June 2024) made the tree **dynamic**. A node's score is the product of the draft probabilities along its path. That cheaply estimates the chance the target accepts it.
- **EAGLE-3** ([arXiv 2503.01840](https://arxiv.org/abs/2503.01840), March 2025) dropped feature regression for direct token prediction. It fuses low, middle and high target layers, and it trains with **training-time test**: the drafter learns from its own multi-step outputs, so deep drafts look at training time the way they will at inference. Draft quality began to scale with data. EAGLE-3.1 (Eldenk et al., [arXiv 2605.09992](https://arxiv.org/abs/2605.09992)) added post-normalisation and per-hidden-state RMSNorm against what it calls attention drift.

None of them priced the denominator. EAGLE-2's tree is sized by confidence, not by what the hardware charges for it. That is the gap EAGLE-4 fills.

## What EAGLE-4 changes

<Figure
  src="https://media.thesatyajit.com/ai/articles/eagle-4/fig1.png?v=786bfae68f2f8153"
  alt="Diagram: the target model's layers 3, 31 and 59 feed hidden states to a drafter made of one decoder layer with 4K sliding-window attention and the EAGLE Efficient LM Head. The drafter grows a draft tree depth by depth. EAGLE-Harness compares the gain of one more depth with its measured cost, picks width or stop, and once per round ranks nodes and picks how many the target verifies in one forward pass."
  caption="EAGLE-4 in one picture: three target hidden states into a one-layer drafter, a tree grown one depth at a time, and a harness that decides after each depth whether to go on and, once per round, how many nodes the target verifies. The tree is schematic (LightSeek's EAGLE-4 post, Figure 1)."
/>

The drafter itself barely changes. EAGLE-4 keeps EAGLE-3.1's normalised single layer. I checked that against the released [Qwen3.8-27B drafter](https://huggingface.co/EAGLE-Corp/Qwen3.8-27B-EAGLE4)'s `config.json` and its safetensors header (**measured**):

| Field | Value | What it is |
|---|---|---|
| `num_hidden_layers` | 1 | one decoder layer |
| `eagle_aux_hidden_state_layer_ids` | 3, 31, 59 | the low, middle and high target layers it reads |
| `fc.weight` | 5120 x 15360 | fuses three 5,120-wide hidden states into one |
| `fc_norm.0-2` | three RMSNorms | one per captured hidden state |
| `sliding_window` | 4096 | the drafter attends to the last 4,096 positions |
| `efficient_lm_head_config` | rank 128, 128 candidates | the new shortlist head |
| `efficient_lm_head.weight` | 248320 x 128 | the low-rank factor over the full vocabulary |
| `lm_head.weight` | 248320 x 5120 | the full head, kept for exact scoring |

Summing the `layers.0.*` tensors gives **0.435B** parameters. The paper's Table 1 says 0.44B. The same count on the public [DFlash 2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2) and [DSpark](https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark) drafters gives **1.665B** and **1.652B** across five layers each (**measured**), against the paper's 1.66B and 1.65B. The paper's "about 3.8x fewer FLOPs per drafted token" is 1.66 / 0.44 = 3.77 (**reasoned**). One thing the paper's table leaves out by design: the whole drafter file holds **3.09B** parameters, and 82% of them are the 248,320 x 5,120 embedding and LM head (**measured**). The layer is small. The vocabulary is not.

That vocabulary is the first real change.

### The Efficient LM Head

A draft step that scores 248,320 tokens spends most of its time on the head. In the paper's profiled configuration the full head is **65.4%** of a **574 µs** draft forward (**reported**). EAGLE-3 truncated the draft vocabulary (32K tokens in LightSeek's comparison), which cannot propose a token outside it.

EAGLE-4 splits the job instead. A rank-128 factorisation $\tilde z = A(Bh)$ scores the whole vocabulary cheaply and picks the top 128 candidates. The full head then computes exact logits for just those 128 rows, $z_C = W_C h$. The low-rank head only retrieves; the scores stay exact. Per token that is about 33M multiply-adds instead of 1,271M (**reasoned** from the tensor shapes above).

<Figure
  src="https://media.thesatyajit.com/ai/articles/eagle-4/fig2.png?v=d276154c135fa85d"
  alt="Stacked bar chart of one draft forward pass: 574 microseconds with the full LM head, of which 65.4% is the head; 326 microseconds at rank 512 with the head at 37.9%; 255 microseconds at rank 128 with the head at 17.9%."
  caption="One draft forward pass with the full head and with the Efficient LM Head at rank 512 and 128. The rest of the pass stays about constant; the head stops dominating (EAGLE-4 paper, Figure 2)."
/>

The paper reports a 55.6% cut. It also says a plain truncated-SVD initialisation at rank 128 *lost* 0.48 tokens of acceptance. Training the two factors for retrieval alone, with the rest of the drafter frozen, puts the full head's top-1 token in the 128-token shortlist **98.2%** of the time, against 91.5% for SVD (**reported**). On TokenSpeed, LightSeek measured the head's share of draft GPU time at 66% with the full head and 19% with the efficient one for chains, 67% and 18% for trees (**reported**). That means a draft step costs about (1 − 0.19) / (1 − 0.66) = **2.38x** less with the shortlist head (**reasoned**). That factor is the "full LM head" switch in the calculator below.

### EAGLE-Harness: price every step

The second change is the one in the title. The paper's rule is one inequality applied at three points of every round:

$$
\frac{\Delta A}{\Delta C} \ge \tau
$$

$\Delta A$ is the expected extra accepted tokens, from EAGLE-2's path-probability product. $\Delta C$ is the extra time, from a table profiled once per hardware setup across batch sizes, context lengths and token counts.

1. **After each draft depth**, stop if the next level is not worth its draft cost. The threshold rises with depth because deep estimates are less reliable.
2. **Once per round**, add nodes to the verify set, best path score first, while each one is worth its marginal verify cost.
3. **The load sets the shape**: up to eight branches per node when lightly loaded, at most four under load.

The repository has the constants (**measured**, `eagle_harness/core/cost_model.py`): a base threshold of 2.5, a 4% rise per depth (`depth_loss_ratio = 0.04`), multiplied by a batch-size ratio from 1.25 for small batches to 2.0 for huge ones. Its device policy decides every other depth: draft, launch, decide. That matches LightSeek's note that a host-side policy would need a device-to-host round trip every other depth.

At temperature above 0, stopping on the tokens drawn in *this* round would bias the output, so a sampled round drafts to the depth where the previous round's rule would have stopped.

<HarnessTree />

The widget runs a toy version of that round. The thresholds are the repository's; the costs above one request and the draft probabilities are mine and illustrative. Push the confidence up, as in repetitive code, and the one-request tree runs eight to eleven levels deep and sends 15 nodes to the target. Drop it to open-ended prose and the tree stops at depth five or six. Switch to 64 requests and the same rule builds a chain of one to four tokens; at 512 it drafts one token and verifies it if it is likely enough. That is the paper's main argument against block drafters like [DFlash 2](/articles/dflash2) and [DSpark](/articles/deepseek-dspark): a block drafter has already paid for its whole block before it can decide to verify less of it. An autoregressive drafter decides before each step; DSpark adapts too, but only by trimming the verify prefix of a block already drafted.

## What the GPU actually does in a round

LightSeek's best evidence is a measurement of a single round.

<Figure
  src="https://media.thesatyajit.com/ai/articles/eagle-4/fig3.png?v=2b655b9671c8c207"
  alt="Timeline of one EAGLE-4 round on TokenSpeed: a 3,828 microsecond decode graph that verifies 16 rows, accepts and drafts depth 0, then five draft-depth graphs of about 123 microseconds each with small decision kernels between them, a finalize step and a 20 microsecond gap to the next round. Side streams for next-round inputs, KV and GDN state commits and output copies run alongside drafting. One round is 4,553 microseconds median."
  caption="One round for one request on TokenSpeed, TP4: a 3,828 µs decode graph, five draft depths at about 123 µs each, and a 4,553 µs median round. Each depth's decision kernel writes a go flag that a one-thread kernel reads to tail-launch the next graph from the GPU (LightSeek's EAGLE-4 post, Figure 5)."
/>

The decision kernel runs on the GPU and writes the next width and a stop flag to device memory, so the host never reads a score. Each depth is its own CUDA graph, with per-round values held as GPU scalars, and each ends in a one-thread kernel that tail-launches the next depth or the finalize graph through CUDA device graph launch. The host launches a round once and never waits inside it. LightSeek is candid that this is worth 0% to 1.5% over its optimised host pipeline (**reported**); the win is simpler graph management.

The timeline lets me check the headline arithmetic. LightSeek reports **985 tok/s** greedy at **3.56x** over no speculation, with an acceptance length of **about 4.75** (**reported**). So plain decoding on that node runs at 985 / 3.56 = 276.7 tok/s, or 3.614 ms a token (**reasoned**). The decode graph that verifies 16 rows costs 3.828 ms, only 6% more than a single plain step. That is the memory-bound regime in one number. A median round of 4.553 ms that advances 4.75 tokens predicts 4.75 x 3.614 / 4.553 = **3.77x** (**reasoned**). The reported 3.56x is 6% lower. Plausible: five depths is the most common count, not the only one, and a whole-run average includes what a median round leaves out. The same model with an acceptance length of about 3.6 predicts **2.88x** at temperature 1, against **2.85x** reported (**reasoned**). The headline is consistent with its own timeline.

<SpeculationCalculator />

The calculator is that arithmetic with knobs. Costs are in plain decode steps: verify $v = 3.828 / 3.614 = 1.059$, a draft step $c = 0.123 / 3.614 = 0.034$, and 0.022 of overhead for the finalize and the 20 µs gap (**reasoned**). The per-token acceptance $a$ in each preset is a chain-equivalent fit chosen to reproduce the reported acceptance length, not a measurement. Tick "full LM head" on the greedy preset: speed falls from 3.80x to 3.20x, which is the case for the shortlist head in one checkbox. Then pick "Busy server", where verify and draft are no longer nearly free, and watch the best depth move left. Its costs are illustrative; neither post publishes per-round costs at high load.

## Checking the speedup claims

| Claim | Source | Setup | What it is |
|---|---|---|---|
| **3.59x** over vanilla, 863.8 tok/s | paper, Table 2 | Qwen3.8-27B BF16, 4x B300 TP4, T=0, 1 request, SPEED-Bench qualitative, "same inference engine, MegaKernel disabled" | **reported**; vanilla 240.5 tok/s |
| **1.28x** "previous state of the art" | paper abstract | same row | 863.8 / 672.6 for DFlash 2 = 1.284 (**reasoned**) |
| **1.62x** over built-in MTP | paper abstract | same row, MTP as a 7-1-8 chain | 863.8 / 534.5 = 1.616 (**reasoned**) |
| **1.70x / 1.99x** over DFlash 2 / DSpark | paper | 512 requests | 28,340 against 16,641 and 14,229 (**reasoned** ratios); both block drafters fall below vanilla there, at 0.78x and 0.67x |
| **3.56x**, 985 tok/s | LightSeek | TokenSpeed, same model and node, T=0, 1 request, 88 conversations / 117 turns | **reported**; implied vanilla 276.7 tok/s |
| **2.85x**, 750 tok/s, **1.28x** over MTP | LightSeek | T=1; MTP as its fastest 5/1/6 chain | **reported** |
| **5.21x** | LightSeek | agentic replay, greedy, **1 request** | **reported**; **1.94x** at 16 requests |
| **3.13x**, 886.5 against 283.5 tok/s | repo support matrix, 0.1.2 | TokenSpeed TP4, 22 conversations / 31 turns, up to 1,024 new tokens | **reported**; what the released package measures |

<Figure
  src="https://media.thesatyajit.com/ai/articles/eagle-4/fig4.png?v=8aba83aec32a66e3"
  alt="Grouped bar chart of throughput relative to vanilla for Qwen3.8-27B. Concurrency 1: vanilla 1.00x, DSpark 2.57x, DFlash 2 2.80x, EAGLE-4 3.59x. Concurrency 512: vanilla 1.00x, DSpark 0.67x, DFlash 2 0.78x, EAGLE-4 1.34x."
  caption="The paper's matched comparison at 1 and 512 requests, temperature 0: EAGLE-4 at 3.59x and 1.34x of vanilla, the two block drafters below vanilla at 512. These are the paper's numbers on one engine with MegaKernel disabled (Eagle's EAGLE-4 launch post, chart of the paper's Table 2)."
/>

Four things are worth knowing before quoting any of these.

**The paper's 3.59x and LightSeek's 3.56x are not the same measurement.** The paper runs one engine with MegaKernel disabled and a vanilla baseline of 240.5 tok/s. TokenSpeed's no-speculation baseline is 276.7 tok/s (**reasoned**), 15% faster. EAGLE-4's absolute speed went up from 863.8 to 985 tok/s, but the ratio stayed put because the baseline improved too. A faster engine shrinks every speculative speedup it reports. That is the right way to measure it.

**The MTP baseline matters.** The paper's 1.62x over MTP uses a 7-1-8 chain. LightSeek says that chain is about 5% slower on TokenSpeed than MTP's best, a 5/1/6 chain, and against the best chain EAGLE-4 is 1.28x at T=1 and **1.37x** greedy (**reported**). So the honest margin over a model's own MTP head on a tuned engine is about 1.3x, not 1.6x.

**The paper's headline numbers are greedy.** At T=1 the paper's own Qwen row drops to 2.59x at one request and **1.03x** at 512 (**reported**). At 512 sampled requests EAGLE-4 is barely ahead of not speculating. LightSeek lists sampled chains for high concurrency under "Looking ahead".

**The shipped package measures less than the paper.** The repository's support matrix for `eagle-harness` 0.1.2 reports 3.35x on SGLang, 2.83x on vLLM and 3.13x on TokenSpeed at one request, falling to 1.38x, 1.19x and 1.40x at 64 (**reported**). It also flags SGLang on one GPU at 64 requests as **0.73x**, slower than not speculating, and lists it as a known limit. Different subsets and token caps explain part of the gap, but the matrix is what `pip install` gets you.

## Why agentic coding is the best case

LightSeek's agentic workload replays SWE-smith multi-turn sessions. The first turn carries about 50K tokens of context, each later turn adds about 800, each produces 500 output tokens, and the prefix-cache hit rate is roughly 90% (**reported**).

<Figure
  src="https://media.thesatyajit.com/ai/articles/eagle-4/fig5.png?v=7bc01315e3ddc17b"
  alt="Line chart of per-user decode speed on the agentic workload with greedy decoding, from 1 to 16 concurrent requests. EAGLE-4 on TokenSpeed falls from about 1,280 to about 260 tokens per second per user; no speculation falls from about 245 to about 130. Speedup labels are 5.21x, 4.21x, 3.33x, 2.56x and 1.94x. EAGLE-4 uses a dynamic tree up to 4 requests and a dynamic chain from 8."
  caption="Per-user decode speed on the agentic workload, greedy, 1 to 16 concurrent requests: 5.21x at one user falling to 1.94x at sixteen, with the harness switching from tree to chain at 8 (LightSeek's EAGLE-4 post, Figure 7)."
/>

Why should code agents be the best case? Three reasons, all **reasoned** rather than measured by anyone:

- **The output repeats the context.** Quoting a file it just read, re-emitting a function with one line changed, naming arguments seen ten turns ago: long runs a drafter reading the target's hidden states can predict. High per-token acceptance is where the geometric series stops decaying.
- **Long context is cheap for this drafter.** Its attention is a 4,096-token sliding window with append-only draft KV, so its cost stays flat while the context grows to 50K. LightSeek's long-context table shows 3.08x at 32K and 3.42x at 250K (**reported**), and time to first token at 250K is 5.83 s with EAGLE-4 against 5.79 s without.
- **An agent is often one user.** A developer driving a coding agent is batch 1 on the decode side most of the time, which is where verification is nearly free.

How much acceptance does 5.21x need? If the round costs the same 1.25 plain steps as on SPEED-Bench, 5.21x needs about **6.5** accepted tokens per round, and about 7 if the drafter goes eight deep and the round gets longer (**reasoned**; the calculator's agentic preset is that second case). LightSeek does not publish the agentic acceptance length, so I cannot check it. On the reply asking whether acceptance drops on tool output: nobody has published a per-turn breakdown, but a 5.21x average is hard to reach if tool-call turns accept badly.

The caveats are all in LightSeek's own text. 5.21x is **greedy** and **one user**. With sampling at temperature 1 the same workload gives **3.04x**. At 16 concurrent greedy requests it is 1.94x per user, and EAGLE-4 produces 4.1x to 1.9x as many tokens per GPU across that range (**reported**). For smaller setups the paper reports 3.49x on one AMD MI350X and 2.71x averaged over four benchmarks on a 4-vCPU c5.xlarge with 4-bit Gemma-4-E2B-it (**reported**); neither is agentic coding.

## Throughput at high concurrency

At 256 to 512 requests the question becomes how few tokens to waste. TokenSpeed's answer is `--speculative-chain-width-table`: 4 verified rows from 64 requests and 3 from 384, the `'64:4,384:3'` example in the flag's help text (**measured**, `server_args.py`). The same text says a batch the table covers drafts "without the confidence rule", and the default is `off`. At high load on TokenSpeed, width is a table calibrated per workload, not the harness deciding each round. LightSeek says width selection plus the table took 256 requests from 7.2K to 15.6K tok/s, and that its final stage reached 31.9K tok/s at 512 (**reported**).

## What it costs

- **Licence.** The `eagle-harness` code and both drafters are under the *Eagle 4 Source-Available License*, "not an OSI-approved open-source license", with a separate commercial agreement. TokenSpeed is MIT; the DFlash 2 and DSpark checkpoints are Apache-2.0.
- **Memory.** The Qwen drafter is a 6.18 GB BF16 file (**measured**), mostly embedding and head.
- **Bit-exactness.** "Lossless" holds in exact arithmetic. The support matrix's greedy check gives "5 exact / 15 near tie" on twenty prompts for SGLang, a near tie being a log-probability gap of at most 0.13 at the first divergence. Expect the same distribution, not the same bytes.
- **Profiling.** Costs are measured once per hardware setup. The support matrix shows a start with an empty cost cache at 2.53x against 2.99x warm on SGLang at 32K (**reported**).

## The take

EAGLE-4's idea is not a better drafter. The drafter is EAGLE-3.1 with a sliding window and a smarter head. The idea is that $A$ was always only half of the ratio. The harness prices each draft step and each verified node against a measured table, so one trained drafter builds a 12-deep tree for a single user and a two-token chain for a busy server. The Efficient LM Head makes each step cheap enough that the tree is worth growing. The paper's ratios, its parameter counts and LightSeek's round timeline all check out.

The 5.21x is real and narrow: one greedy user on Qwen3.8-27B, four B300s, a SWE-smith replay. For planning, use the support matrix: about 3x for one request on a tuned engine, about 2x at eight, about 1.4x at 64.

<RepoCard repo="eagle-hq/EAGLE-4" />

<RepoCard repo="lightseekorg/tokenspeed" />

<ModelCard repo="EAGLE-Corp/Qwen3.8-27B-EAGLE4" />

*Sources: the [EAGLE-4 paper](https://github.com/eagle-hq/EAGLE-4/blob/main/EAGLE4_Paper.pdf) and repository at commit `c10363b`, [Eagle's launch post](https://blog.eagle.ai/blog/introducing-eagle-4), [LightSeek's post](https://lightseek.org/blog/eagle-4), [TokenSpeed](https://github.com/lightseekorg/tokenspeed) at `12b7fa1`, and the drafter checkpoints' `config.json` and safetensors headers. Related on this site: [Kimi K3 on GB300](/articles/kimi-k3-gb300-megakernel), where a forced acceptance length decides the headline, and [Step 5 Preview](/articles/step-5-preview), whose sliding windows are the same trick at target scale. Figures are the papers' and posts'; the two interactives are mine.*