[{"title":"Carbon-X: teaching Qwen3-14B to read DNA in 6-base words","description":"Hugging Face's Carbon-X is Qwen3-14B mid-trained on 302B tokens of DNA and text, then fine-tuned and RL-trained to answer questions about raw sequences in English. It is one decoder with 5,515 new DNA tokens, not an encoder bolted onto a chat model. I ran the released tokenizer on CPU, read the weights' headers and the GGUF metadata by range request, and rechecked the card's arithmetic. The 6-mer vocabulary uses a third of the tokens Qwen's BPE spends on DNA. It also fixes the reading frame, so a one-base insertion rewrites every token after it. The 66.7 average that sits next to GPT-6.1 Sol's 67.5 is real on the card's terms, but it rests on two benchmarks Hugging Face built and has not published, and it is inside one standard error of a single sampled run. The 'full recipe' is a hyperparameter list for now; the linked code repository predates Carbon-X.","date":"2026-10-23","tags":["pretraining","tokenization","reinforcement-learning","grpo","benchmarks","qwen","biology"],"draft":false,"featured":false,"rating":{"novelty":2,"verification":2,"runnable":2,"explains":2,"takeaway":2,"durability":2,"reach":1,"unique":2,"why":"Runs Carbon-X's DNA tokenizer, reads the weights and GGUF headers, and separates what the 66.7 average does and does not show."},"topic":"training-rl","level":"practitioner","runsOn":"consumer-gpu","licence":"Apache-2.0","articleKind":"model","kind":"articles","slug":"carbon-x","body":"On 22 October Leandro von Werra posted the headline: \"you can mid-train\nyourself to the frontier! At a fraction of the cost! Without distillation!\" The\nmodel is [Carbon-X](https://huggingface.co/HuggingFaceBio/Carbon-X), from\nHugging Face's bio team with Zhongguancun Academy and TIGEM / University of\nNaples. You paste DNA into a chat message and it answers in English: benign or\npathogenic, promoter or enhancer, which organism group, what the next bases\nare.\n\nThe post said it started from \"Qwen3.5 14B\". A reply pointed out that no such\nmodel exists, and he corrected it to Qwen3-14B, which is what `config.json` and\nthe card's `base_model` say.\n\nWhat shipped on 21-22 October: the weights (Apache-2.0), a GGUF repository, a\nrelease blog on a Space, and a model card that lists the training recipe as\nhyperparameters. What did not ship yet is the training code. More on that at\nthe end.\n\nI could not run a 29.6 GB model on a shared 4-core VM. What I could run is the\npart that makes this model different: the tokenizer. I also read the weights'\nsafetensors headers and the GGUF metadata by HTTP range request, and rechecked\nevery average and every piece of arithmetic in the card. Every number below is\nlabelled **measured** (I computed it from a file or a run), **reported**\n(Hugging Face's figure, not re-run) or **reasoned** (my arithmetic on the other\ntwo).\n\n\n\n## One decoder, two alphabets\n\nThere are two existing ways to get a model that talks about DNA. Genomic\nfoundation models such as Hugging Face's own Carbon-3B read raw sequence and\nscore it, but they cannot explain a prediction. Chat models know biology from\npapers but have barely seen a genome. The usual bridge, used by BioReason, is a\nseparate DNA encoder whose embeddings are projected into a language model.\n\nCarbon-X skips the encoder. DNA is just more tokens in the same 40-layer\ndecoder:\n\n\n\nThe weights confirm the shape. I read both safetensors headers (**measured**):\n443 tensors, all bfloat16, **14,822,046,720 parameters**, 40 layers, hidden\n5,120, FFN 17,408, `q_proj` 5,120 x 5,120 and `k_proj`/`v_proj` 1,024 x 5,120.\nThat is 40 query heads and 8 KV heads of dimension 128, with `q_norm` and\n`k_norm` present. The embedding and LM head are separate tensors (untied), each\n157,184 x 5,120. The architecture string is plain `Qwen3ForCausalLM`; nothing\nabout the model body is custom.\n\nAt bfloat16 that is 29.6 GB of weights (**measured**, the two shard sizes sum to\n29,644,145,152 bytes). The KV cache costs 40 layers x 2 x 1,024 x 2 bytes =\n160 KiB per token (**reasoned**), which agrees with the GGUF card's \"about\n0.16 MB of KV cache per token\". A full 40,960-token context is about 6.3 GiB\nper sequence on top of the weights. \"Fits on a single 80 GB GPU\" holds with\nroom to spare.\n\n## The tokenizer, from the code\n\nEverything new lives in `tokenizer.py`, a `HybridDNATokenizer` that wraps Qwen's\nBPE. I ran it on CPU with `transformers` 5.20.2 (the run exception for this\narticle; peak RSS 210 MB). It is short enough to explain fully.\n\n**The vocabulary.** The code builds every 6-letter string over A, C, G, T:\n$4^6 = 4096$ k-mers. It adds one *tail* token for every prefix of length 1-5,\npadded with N (`CT` becomes `CTNNNN`): $4 + 16 + 64 + 256 + 1024 = 1364$. Three\nmarkers, ``, `` and ``, make 5,463. The total is then padded to\na multiple of 128 with 52 `` tokens. So the card's \"5,515 DNA tokens\"\nis 5,463 real ones plus 52 padding (**measured**: those are the counts the loaded\ntokenizer reports). The BPE part is Qwen's 151,669 ids, so the vocabulary is\n157,184, matching the embedding tensor.\n\n**Where the new rows came from.** Qwen3-14B's own `config.json` has\n`vocab_size` 151,936: Qwen pads its 151,669 real tokens with 267 spare rows.\nCarbon-X starts its DNA ids at 151,669, so the first 267 DNA tokens land on\nQwen's spare rows and 5,248 rows are appended. That is\n$2 \\times 5248 \\times 5120 = 53{,}739{,}520$ new parameters across the\nembedding and LM head (**reasoned** from the two configs and the measured\nshapes), about 0.36% of the model. The card says all DNA embeddings start\nrandom and are trained for about 10B tokens with the body frozen before\nanything else moves.\n\n**Compression.** I tokenized the same DNA both ways (**measured**):\n\n| Sequence | Bases | Carbon-X tokens | Qwen3 BPE tokens | Ratio |\n|---|---:|---:|---:|---:|\n| HBB fragment from the card | 55 | 12 | 30 | 0.40 |\n| Random, uniform ACGT | 600 | 102 | 308 | 0.33 |\n| Random, uniform ACGT | 6,000 | 1,002 | 3,093 | 0.32 |\n| CAG repeat | 600 | 102 | 400 | 0.26 |\n| Poly-A | 600 | 102 | 75 | 1.36 |\n\nCarbon-X counts include the two tags. The blog's \"roughly a third as many\ntokens as Qwen3's tokenizer\" holds on non-repetitive sequence. The one case BPE\nwins is a homopolymer, where it has merged long A-runs into single tokens. The\n6-mer count is fixed by length: $\\lceil n/6 \\rceil + 2$, whatever the content.\nThat predictability is the point. An 8,192-token training sequence holds at\nmost 49,152 bases (**reasoned**; the card says \"≈ 49 kbp\"), and tokenizing\n49,152 random bases gave 8,194 tokens in 13 ms (**measured**).\n\n**Auto-tagging.** You do not have to write `` yourself. Any run of six or\nmore A/C/G/T letters, either case, is wrapped; runs of twelve or more with up to\n20% IUPAC ambiguity codes are wrapped too; space-separated codon blocks are\njoined first. I tried edge cases (**measured**). `ACG TAC GAT TCA GGA CTT` is\njoined into one sequence. `Sequence ACGTA` stays BPE, five letters being too\nshort. And `GATTACA is a 1997 film`, in either case, comes out as\n` GATTAC ANNNNN is a 1997 film`. The detector has no notion of a\nword boundary. For a DNA chat model that is the right default, but pass\n`auto_dna_tags=False` if you feed it general text.\n\n**N costs six bases.** Only canonical bases form k-mers. `ACGTNACGTACGT`\nbecomes ` CGTACG TNNNNN`: the one unknown base takes its whole 6-mer\nwith it (**measured**). The card says the same; filter or mask before you\ntokenize.\n\n### The frame is fixed, and indels move it\n\nThe tokenizer cuts from the first base of each tagged run, every six bases,\nwith no overlap. That has a consequence the card does not spell out. A\nsubstitution changes exactly one token. An insertion or deletion of anything\nother than a multiple of six shifts the frame. Every 6-mer downstream then\nbecomes a different token id. In a 600-base random sequence I made each edit at\nbase 301 and diffed the ids (**measured**). A substitution changed 1 of 102\ntokens, and so did an N. A one-base insertion changed 52, a one-base deletion\n50, and a three-base insertion 52. A six-base insertion added exactly one new\ntoken and left every downstream id intact, one slot later.\n\nTry it on the card's own example. The reference is the 55-base HBB fragment\nfrom the usage snippet, and the variant there is A to T at base 20:\n\n\n\nThe model can still learn that `CAGGAG` shifted by one base is related to\n`GCAGGA`; 302B tokens of training is a lot of shifted DNA. But the two strings\nshare nothing at the token level, so it has to learn that relation; it gets no\nhelp from the input. This is the obvious place to look when you notice that\nthe one benchmark where Carbon-X trails the frontier by a wide margin is the\nnon-SNV one: insertions, deletions and other non-substitution changes, 85.1\nagainst Opus 5.5's 94.8 (**reported**). I cannot attribute the gap to\ntokenization from the outside. That would take an ablation Hugging Face has not\npublished. It is a hypothesis, **reasoned**, not a finding.\n\n## The recipe\n\nThe card describes four stages. None of the code to run them is public yet; the\nnumbers below are the card's (**reported**) unless marked.\n\n1. **Embedding warm-up.** About 10B tokens with the transformer frozen, so only\n the new DNA rows learn.\n2. **Mid-training.** 302B tokens at sequence length 8,192, global batch 512,\n 72,000 steps, cosine learning rate from 5e-5 to 5e-6 after 2,000 warmup steps.\n The arithmetic closes: $512 \\times 8192 = 4{,}194{,}304$ tokens a step, times\n 72,000 steps is 301.99B (**reasoned**).\n3. **SFT.** Three epochs over 20 dataset subsets, cosine 1e-5 to 1e-6, batch 32,\n examples up to 9,984 tokens, loss on assistant tokens only. About 30% is\n variant-effect reasoning, about 10% procedurally generated DNA skills\n (counting, searching, editing).\n4. **RL.** Asynchronous GRPO in TRL with verifiable rewards, thinking off, in\n two rounds: 98,433 prompts for 150 steps, then 58,249 prompts re-selected\n from the first round's rollouts. 16 samples per prompt, completions up to\n 4,096 tokens, learning rate 1e-6, asymmetric clipping 0.2 / 0.28, gradient\n norm capped at 0.2.\n\nThe mid-training mixture is 55% raw DNA, 20% English, 15% biology text and 10%\ninstruction data:\n\n| Component | Share | Source |\n|---|---:|---|\n| Eukaryotic genomes | 38.5% | GENERator corpus |\n| mRNA | 8.8% | OpenGenome2 |\n| Prokaryotic genomes | 5.5% | OpenGenome2 |\n| mRNA with splice sites and promoters | 2.2% | OpenGenome2 |\n| FineWeb-Edu / DCLM / FinePDFs-Edu | 8% / 8% / 4% | web and PDFs |\n| Biology documents with long DNA | 7% | FinePDFs, PubMed, bioRxiv, FineWeb, Biostars |\n| Other biology documents | 7% | same sources |\n| Synthetic RefSeq descriptions | 1% | written by Gemini |\n| Instruction following and chat | 10% | Nemotron post-training data |\n\nThe blog says eukaryote-infecting viral genomes were excluded from the DNA.\nThe English slice is the same kind of filtered web data I looked at in\n[Ultra-FineWeb: what an education filter costs you, measured](/articles/ultra-fineweb).\n\n### The learning-rate trap\n\nThe useful part of the blog is the account of what went wrong first. On\nQwen3-4B, the learning rate that taught DNA fastest destroyed English, and the\none that kept English barely taught DNA. The sweep's own plotted data (I read\nthe arrays out of the blog's figure; **reported**): at 2e-4 the English score\nfell 22.6 points after about 10B English tokens while the DNA score rose from\n0.33 to 0.45. At 2e-5 English fell only 2.3 points and DNA reached 0.36.\n\nThey also tried better embedding initialization, separate learning rates for\nembeddings and body, freezing layers whose DNA and English gradients conflict,\nmerging DNA and English specialists, and more English replay. The blog says\nnone of them recovered both. What worked was boring. A lower rate, a model\n3.5x larger, a run long enough for slow DNA progress to add up, and cosine\ndecay instead of warmup-stable-decay, which kept the rate high long enough to\nworsen forgetting.\n\n\n\nFrom the same figure's data (**reported**): HellaSwag ended at 0.792 against\nthe starting model's 0.790. MMLU-Pro biology ended at 0.725 against 0.721.\nOpenBookQA ended at 0.858 against 0.882, a 2.4-point loss and the one real\nregression. This is the same trade I measured for a language adaptation in\n[Jais 3: adapting K2 Horizon to Arabic without touching the tokenizer](/articles/jais-3);\nCarbon-X has it harder, because it had to change the tokenizer.\n\n### Why mid-train at all\n\nCould RL alone have done it? The blog ran the test. GRPO on FrontierVEP straight\nfrom Qwen3-14B never rose above the 50% chance line. From the mid-trained\ncheckpoint, the same RL kept climbing:\n\n\n\nThe base run's *training* reward does climb, to about 0.53, while its\nevaluations stay flat. That is the classic signature of RL finding format and\nprior-label shortcuts rather than reading the sequence. Mid-training is what\ngives RL something to sharpen. If you have read\n[Scaling agentic RL: 365,000 environments behind one contract](/articles/scaling-agentic-rl),\nthis is the same lesson from the other side: the environment and the verifier\nmatter, but RL cannot create a capability the policy never had.\n\n### \"Without distillation\"\n\nA reply under the post put it well: \"'without distillation' is doing a lot of\nheavy lifting — the supervision setup is the claim.\" Here is what the setup is.\nThe blog says frontier models refused or hallucinated too often to be teachers,\nand self-distilling from their own RL rollouts carried unsupported claims into\nSFT. So they built deterministic verifiers with domain experts that check each\nclaim in a training answer against the input sequence, and assembled SFT\nanswers that pass. RL rewards are code-checked too, with a penalty on\nvariant-effect answers whose rationale is not grounded.\n\nThat is a real distinction, and it is the right one. It is also narrower than\nthe slogan. The mid-training mixture includes 1% Gemini-written descriptions of\nRefSeq sequences and 10% Nemotron post-training data, which NVIDIA built\nlargely from other models' outputs (**reasoned** from the card's table). No\nteacher model wrote Carbon-X's DNA answers. Other models' text is still in the\ndiet.\n\n## What the benchmarks show\n\nFive DNA benchmarks, Carbon-X with thinking off and one sample per problem;\neach comparison model at whichever reasoning effort gave its best average:\n\n\n\nThe same table as text (**reported**; I recomputed every row's average from\nits five scores and all seven match, **reasoned**):\n\n| Benchmark | Carbon-X | GPT-6.1 Sol max | Opus 5.5 max | Gemini 4 Argon low | Kimi K3 low | DeepSeek V4.1 Flash low | GLM-5.3 low |\n|---|---:|---:|---:|---:|---:|---:|---:|\n| BioReason Coding (1,018) | **74.9** | 71.0 | 70.3 | 6.5 | 53.9 | 51.4 | 52.1 |\n| BioReason Non-SNV (873) | 85.1 | 91.7 | **94.8** | 90.3 | 65.3 | 55.9 | 60.1 |\n| GenomeQA (2,600) | 63.6 | **67.2** | 62.2 | 63.1 | 52.0 | 49.7 | 44.6 |\n| NextBase (600) | **39.4** | 32.9 | 1.1 | 31.2 | 30.6 | 27.2 | 9.7 |\n| FrontierVEP (1,000) | 70.4 | 74.9 | **76.2** | 58.0 | 50.1 | 51.3 | 32.7 |\n| **Average** | 66.7 | **67.5** | 60.9 | 49.8 | 50.4 | 47.1 | 39.8 |\n\nThe headline figure plots average score against cost per problem:\n\n\n\nFour things to hold next to that chart.\n\n**The two benchmarks Carbon-X leads by the most are the two Hugging Face\nbuilt.** FrontierVEP and NextBase are in-house. On 23 October the\nHuggingFaceBio organization lists nine datasets, none of them FrontierVEP or\nNextBase (**measured**, from the Hub API). So\nnobody outside can rerun them yet. The data was decontaminated against all\nfive with DNA overlap, text n-gram and homology checks (**reported**).\n\n**NextBase mostly measures refusals for the closed models.** The blog says\nOpus 5.5 scored 17.7% on NextBase at low effort and 1.1% at max, with refusal\nrates rising with effort. Gemini 4 Argon fell from 58% to 38% on FrontierVEP\nas effort rose. Opus's 1.1 alone pulls its average about 15 points below the mean of its other four.\nDrop NextBase and the four-benchmark averages are GPT-6.1 Sol 76.2, Opus 5.5\n75.9, Carbon-X 73.5 (**reasoned**). Carbon-X is third, and the gap is the\nnon-SNV row.\n\n**66.7 against 67.5 is a tie.** Carbon-X ran one sampled answer per problem at\ntemperature 0.6. The GGUF card estimates the standard error of a single run at\nabout 1.5 points on FrontierVEP and about 1 on NextBase (**reported**). A 0.8\ngap in a five-way average is well inside that noise. \"Close to GPT-6.1 Sol\" is\nthe honest phrasing, and it is the card's.\n\n**Where it is weak, the card says so.** GenomeQA's six tasks average 63.6\n(**reasoned**, from the per-task table: 70.0, 29.8, 93.2, 34.6, 64.0 and 90.0).\nSplice sites at 29.8 and histone marks at 34.6 are weak.\n\nTwo more reported results. IFEval prompt-level strict accuracy is 79.3%, with\nno number for Qwen3-14B beside it, so the size of any regression is not given.\nAnd used as a sub-agent that [Kimi K3](/articles/kimi-k3) was required to\nconsult, Carbon-X raised Kimi's five-benchmark index from 49.4% to 60.5%, on\n100 questions per benchmark with Python available in both arms.\n\n## The cost claim\n\nThe blog projects one pass over ClinVar's 2.4 million variants of uncertain\nsignificance. It extrapolates from each model's cost per completed FrontierVEP\nproblem, refusals included. Its figure gives the exact numbers (**reported**):\nCarbon-X \\$271, GPT-6.1 Sol \\$32,380 at low effort and \\$658,632 at max,\nOpus 5.5 \\$110,724 at low and \\$4,027,185 at max. The card rounds these to\n\"about \\$300\" against \"\\$32,000 to \\$4 million\".\n\nThe comparison is self-hosted against list price. Carbon-X is priced at \\$5\nper H200-hour from measured throughput, so \\$271 is about 54 H200-hours, or\nroughly 44,000 variants an hour (**reasoned**). The closed models are priced\nat standard API rates, and at max effort they think for thousands of tokens\nwhile Carbon-X answers directly. That is a real advantage of a direct-answer\nspecialist, and it is also why the ratio moves so much with effort: 120x\nagainst GPT-6.1 Sol low, 2,430x against it at max (**reasoned**). The blog's\nopening says \"1,000x lower cost\"; the card says \"about 1/1,800 of the inference\ncost per problem\". Both are defensible readings of the same chart.\n\nWhat the cost line leaves out is training. No GPU-hours are disclosed, and a\nreply asking for them went unanswered. A floor from the standard $6ND$\nestimate: $6 \\times 14.8\\text{B} \\times 302\\text{B} \\approx 2.7 \\times 10^{22}$\nFLOPs for mid-training alone. At an assumed 40% utilization of an H200's\nroughly 989 dense BF16 TFLOP/s, that is about 19,000 H200-hours, or about\n\\$94,000 at the same \\$5 rate (**reasoned**, with my assumptions, before SFT,\nRL and every ablation). \"A fraction of the cost\" of a frontier pretraining run,\ncertainly. Not free.\n\n## Running it\n\nThe card says Carbon-X needs Transformers v5 with `trust_remote_code=True`\nfor the tokenizer, or vLLM on one GPU. It also says \"llama.cpp and other GGUF\nruntimes are not supported\". That sentence is stale. The blog says the\nopposite, and a\n[Carbon-X-GGUF](https://huggingface.co/HuggingFaceBio/Carbon-X-GGUF) repository\npublished the same day ships BF16, Q8_0, Q6_K, Q5_K_M and Q4_K_M files. It needs\nllama.cpp b11742 or newer, which added the tokenizer.\n\nI read the Q4_K_M file's metadata with a 24 MB range request (**measured**).\n`tokenizer.ggml.model` is `hybriddna`, `tokenizer.ggml.dna_auto_tags` is true,\nand there are 157,184 tokens. Each DNA k-mer is stored with a private-use\ncharacter (U+E000) appended, so `AAAAAA` the DNA token cannot collide with a\nBPE string of the same spelling. The Python code flags that collision for\n`CCCCCC`, which exists in both vocabularies. The architecture block repeats the\nsafetensors shapes exactly. The GGUF card reports the llama.cpp tokenizer\nmatches Python's on 1,000 benchmark prompts (1.13M tokens) and 36 edge cases,\nand that Q8_0 through Q5_K_M score within about a standard error of the\noriginal on FrontierVEP and NextBase. Q4_K_M is a 9.6 GB file that fits a\n16 GB consumer GPU, at 67.8 on FrontierVEP against 70.4 (**reported**).\n\nI did not run the model itself: even Q4_K_M is over this article's 4 GB\ndownload cap.\n\nThe card's agent advice is worth following: pass sequences by name, because\nlanguage models miscopy long DNA, and leave counting and coordinates to\nPython. The bundled `carbon-dna` skill also ranks variants by summed\nlog-likelihood from vLLM's `prompt_logprobs`.\n\n## What is open\n\n- **Weights:** yes, Apache-2.0, safetensors and GGUF.\n- **Recipe:** as hyperparameters and data shares in the card, enough to plan a\n reproduction, not to run one.\n- **Training code:** not yet. The card's \"Code\" link points to\n `huggingface/carbon`, whose `main` is commit `10bbc4b` from 6 June 2026\n (**measured**, shallow clone). It holds Carbon-3B-era evaluation and\n fine-tuning scripts and a tech report; no text file in it mentions Carbon-X or Qwen. In\n the replies von Werra says the team is \"preparing the full release. It will\n be fully open and reproducible.\"\n- **Benchmarks:** FrontierVEP and NextBase are not published.\n- **Pretraining DNA:** the Carbon pretraining corpus from the Carbon-3B release\n is on the Hub. The 302B-token mid-training mix is not, as one artifact.\n\n\n\nCarbon-X has not been clinically validated; the card and the blog both say so. Its\nother stated limits are miscounted bases and coordinates in long sequences,\ninvented identities when context is missing, degraded long generations, and\nno evaluation of inputs much longer than training saw. The blog's opening\nline says you can \"generate new DNA\". The card says it \"was not trained to\ndesign new ones\". Believe the card.\n\n## The take\n\nThe idea is simple, and I think it is right. Give the decoder a vocabulary that\nfits DNA, then spend 302B tokens at a learning rate low enough to keep English.\nThat puts a 14B open model within noise of the best closed model on these five\nbenchmarks, cheaply enough to run all of ClinVar's uncertain variants. The\ntokenizer is the whole architectural change, and it checks out: a third of\nBPE's tokens, exact round trips, and a fixed frame that turns indels into a\ndifferent sequence downstream.\n\nWhat I would want before calling it frontier: FrontierVEP and NextBase\npublished, more than one sample per problem, and the training code the post\npromised. Until then the 66.7 is Hugging Face's number on Hugging Face's terms,\nand it is a credible one.\n","readingTimeMins":19,"url":"https://ai.thesatyajit.com/articles/carbon-x","lastUpdated":"2026-10-23","facts":{"words":3896,"readingTimeMins":19,"figures":4,"interactives":2,"film":true,"citations":2,"cards":2,"measured":16,"inbound":3,"lastUpdated":"2026-10-23"},"score":67.7,"tier":{"id":"high","label":"High","level":4},"rank":159,"lenses":{"must-read":67.7,"run-it":9,"learn":7.5,"new":7,"deep":10},"highlights":["Original analysis","Runs on a consumer GPU","A lasting reference"],"signal":{"interest":null,"helpful":null,"score":null,"level":4,"label":"High","source":"tier"}},{"title":"d3-turbo: the state stops looking at the question, so a photo is read once for 255 questions","description":"vLLM Semantic Router's d3-turbo is d3, the 27B Qwen3.8 decision model, trained further with one change to its attention: in the 16 full-attention layers the state block no longer reads the question. That makes the state's hidden states identical for every question, so the runtime computes them once and runs each question on top, which is where the 15.5x over d3-flash at 255 questions about one photo comes from. I read the shipped runtime, the configs, the weights by HTTP range request and the Decision Index's own data files. The mechanism is exactly as described, and the latency table fits about 129 ms per question for d3-flash against a fixed 0.33 s plus about 7 ms per question for d3-turbo. Box answers are four 128-bin edge distributions whose confidence is the share of 512 sampled boxes that overlap the answer at IoU 0.5, passed through a calibration table; I ran that decoder. The '#1 on both boards' was the team's own estimate against a 16 October snapshot. On the official boards since, d3-turbo is tenth on text, level with d3 inside the tie band, and second on vision, where it was first for about 29 hours. Almost all of its gain over d3 is on the public benchmarks.","date":"2026-10-23","tags":["calibration","benchmarks","multimodal","qwen","linear-attention","inference"],"draft":false,"featured":false,"rating":{"novelty":1,"verification":3,"runnable":1,"explains":2,"takeaway":2,"durability":1,"reach":1,"unique":2,"why":"Reads the state-once mask and the Box decoder from d3-turbo's own runtime, and replays the Decision Index history behind '#1 on both boards'."},"topic":"llm-architecture","level":"practitioner","runsOn":"workstation","licence":"Apache-2.0","articleKind":"model","kind":"articles","slug":"d3-turbo","body":"[Xunzhuo Liu's launch post](https://x.com/XunzhuoLiu/status/2112343603209261092) for d3-turbo makes three kinds of claim. It is \"#1 on both Jev Decision Index boards\". It \"points(box), follows and sees images in your questions\". And \"StateOnce brings 15x faster than d3 flash\". The thread adds the numbers: 255 questions about one photo in 2.12 s against 32.94 s, a Box accuracy of 0.90, a track presence F1 of 0.97, +6.2 over d3 on the public suite, Apache-2.0 weights.\n\nThe speed claim is the interesting one, because it is an architecture claim. A decision model reads a piece of state, an email or a photo, and answers typed questions about it with a probability per option, without generating text. If it answers 255 questions, the cheap way is to read the state once and hang every question off it. Whether a model is allowed to do that is decided by its attention mask, and the last two releases in this family went in opposite directions on it.\n\nSo I went to the files: the [model repository](https://huggingface.co/vllm-sr/d3-turbo) at commit `a0f3e9e` (its runtime, configs and card), the weights of d3-turbo, d3 and Qwen3.8-27B read by HTTP range request, and the Decision Index Space's own data files at seven commits. There is no paper. The model card is the only write-up, so every architecture statement below comes from code.\n\nThe short answer: the mechanism is real and small. One mask changed, and the speed follows from it exactly. The leaderboard claim was an estimate, and the official boards disagree with it on text.\n\n\n\n## What a decision model computes\n\nEvery model in this wave shares one output contract, the `/v1/systemone` shape that [Jev](/articles/where-to-use-jev) made common. The prompt (`d3_format.py`) is a system turn, then `State:` and the state, then `Question:`, the instructions, and the options listed under one-token codes: `A`, `B`, …, `Z`, `AA`, … up to 255 of them. The model does not decode. It takes the final hidden state at the last prompt position, $h \\in \\mathbb{R}^{5120}$, and multiplies it by a 255-row readout $W$:\n\n$$\np = \\operatorname{softmax}\\!\\left(\\frac{W_{1:m}\\, h}{T}\\right), \\qquad W \\in \\mathbb{R}^{255 \\times 5120},\\; T = 1\n$$\n\nwhere $m$ is the number of options and the codes past $m$ are masked to $-\\infty$. That readout is 1,305,600 parameters (measured: $255 \\times 5{,}120$, and the manifest says the same). A yes/no question is a two-option choice. A score is a choice over levels. None of that changed from d3. What changed is how $h$ is computed when there are many questions.\n\nThe backbone is the Qwen3.5 hybrid that [Qwen3.8-27B](/articles/qwen3-8-open-weights) uses: 64 layers, every fourth a full softmax-attention layer and the other 48 [Gated DeltaNet](/articles/linear-attention-state-roundup) layers, a linear recurrence whose memory is a fixed-size matrix per head (measured from `config.json`'s `layer_types`). That split matters, because the two kinds of layer can be made to share work in different ways.\n\n## The one change: `prefix_bidir`\n\nd3's `decision_config.json` sets `attention_mode` to `noncausal_full_attention` (measured). In the 16 full-attention layers every token sees every other token, before and after it. That is what [pplx-decider v1.1](/articles/pplx-decider-v1-1) introduced, and for the same reason: a classifier wants the state read with the question in view. The cost is that the state's hidden states now depend on the question. Change the question and the state's representation changes too. Nothing can be cached, so every question is a full pass over the state. d3-flash, the 9B sibling, uses the same mode (measured).\n\nd3-turbo's config says `prefix_bidir` (measured). The runtime's docstring describes it, and `StateBlocks` and `block_attention` in `d3_runtime.py` implement it. Split each prompt at the end of the state text into a state block $\\mathcal{S}$ (system turn, images, videos, `State:` and the state) and a question block $\\mathcal{Q}$. In a full-attention layer, query $i$ may read key $j$ when\n\n$$\nM_{ij} = \\begin{cases} 1 & i \\in \\mathcal{S},\\; j \\in \\mathcal{S} \\\\ 1 & i \\in \\mathcal{Q},\\; j \\in \\mathcal{S} \\cup \\mathcal{Q} \\\\ 0 & i \\in \\mathcal{S},\\; j \\in \\mathcal{Q} \\end{cases}\n$$\n\nThe state is bidirectional within itself, the question is bidirectional within itself and reads the whole state, and the state never reads the question. The implementation does not even build the mask. It makes two unmasked SDPA calls per row, one with the state's queries against the state's keys and one with the question's queries against all keys, which together equal the dense masked attention.\n\nThe Gated DeltaNet layers stay causal over the whole prompt, as they are in d3 (measured). A causal scan is already shareable: its recurrent state after the last state token is the same whatever follows.\n\nPut those together and the state's hidden states, in every layer, are a function of the state alone. That is the whole trick, and `StateOnce` is the code that cashes it in. A state pass runs the state block once as one row and keeps, per layer, either the full-attention keys and values (RoPE applied) or the Gated DeltaNet's final recurrent state plus the last three inputs of its width-4 short convolution. A question pass then runs the question blocks in batches of up to 64 rows, right padded. Each full-attention query reads the cached state keys and values plus its own block. Each DeltaNet layer continues its scan from the cached state. `usage.input_tokens` counts the state once plus each question block. With `D3_STATE_ONCE=0` the runtime falls back to one full row per question, the reference path the cached one is meant to equal.\n\nToggle the two modes and the two kinds of layer below. The counter is the number of token positions pushed through the trunk for $k$ questions: $k(s + q)$ when every question re-reads a state of $s$ tokens, $s + kq$ when the state is read once.\n\n\n\nSome numbers make the gap concrete. An image is resized to at most 1,638,400 pixels and each 32 x 32 patch becomes one token (measured from the runtime docstring), so a large photo is up to 1,600 tokens of state. Questions are tens of tokens. With a 1,600-token state, 40-token questions and 64 of them, re-reading costs 104,960 positions and state-once 4,160, 25 times fewer (reasoned).\n\nThis is the same idea as the tree that [Agr](/articles/agr-decision-model) and [Liquid's d1-3B](/articles/liquid-open-d1) use to read the state once. The difference is that d3-turbo keeps the bidirectional view inside each block, where the earlier tree designs are causal. Perplexity's v1.2 reached a related layout from the other direction, putting every question in one sequence.\n\n### Is it a new model or a new mask?\n\nThe checkpoint is the same shape as d3, down to the tensor names: 1,183 tensors in both indexes, 26,085,330,160 backbone parameters (measured from the safetensors index and the Hub's count). I read slices of the same tensors from d3-turbo, d3 and Qwen3.8-27B by HTTP range request (the first 8 MB of each tensor) and compared them:\n\n| Tensor (slice) | Qwen → d3 | Qwen → d3-turbo | d3 → d3-turbo |\n| --- | ---: | ---: | ---: |\n| layer 0 `mlp.up_proj` | 0.687% | 0.944% | 0.617% |\n| layer 3 `self_attn.q_proj` | 0.457% | 0.564% | 0.314% |\n| layer 31 `self_attn.q_proj` | 0.526% | 0.701% | 0.434% |\n| layer 31 `mlp.down_proj` | 0.606% | 0.861% | 0.565% |\n| layer 63 `self_attn.k_proj` | 0.674% | 0.952% | 0.709% |\n| layer 0 `linear_attn.in_proj_qkv` | 0.465% | 0.612% | 0.383% |\n| vision block 0 `attn.qkv` | 0 | 0 | 0 |\n\nRelative Frobenius distance, measured. The readout moved 1.22% from d3's (measured). The vision tower blocks I sampled are bit-identical to Qwen's in all three.\n\nThe pattern fits d3-turbo being d3 trained further. The distance from Qwen to d3-turbo is close to what you get by adding the other two as independent steps: for layer 0's `up_proj`, $\\sqrt{0.687^2 + 0.617^2} = 0.923$ against a measured 0.944 (reasoned). Two independent fine-tunes of Qwen would sit about equally far from it, and further from each other. The card does not describe training, so this is inference from weights, not a documented recipe. Training under the new mask is necessary, though: a model trained to read the question from inside the state cannot be switched to a mask that hides it without relearning.\n\n## The speed claim, against its own table\n\nThe card times one request at a time on one AMD Instinct MI325X through `AutoModel`, 1 to 255 yes/no and choice questions about one 1,280 x 1,600 JPEG, median of 15 timed requests after 3 warm-ups (reported). On ROCm `gfx942` the state-once path runs through fused kernels in `d3_fast.py` (measured from the code path, not run).\n\n\n\n| Questions | d3-turbo p50 | d3-flash p50 | flash ÷ turbo |\n| ---: | ---: | ---: | ---: |\n| 1 | 302 ms | 154 ms | 0.5× |\n| 4 | 328 ms | 541 ms | 1.6× |\n| 16 | 426 ms | 2.07 s | 4.9× |\n| 64 | 867 ms | 8.26 s | 9.5× |\n| 128 | 1.29 s | 16.50 s | 12.8× |\n| 255 | 2.12 s | 32.94 s | 15.5× |\n\nThe rows are reported; the ratios recompute (reasoned). A least-squares line through each column says what the two designs cost (reasoned). d3-flash is close to a constant 129 ms per question, from 135 ms at 4 to 129 ms at 255, which is what reading the photo once per question looks like. d3-turbo is a fixed 0.33 s plus about 7.2 ms per added question. The fixed part is the state pass: the vision tower plus 64 layers over about 1,600 image tokens. The marginal part is a question block on a cached state.\n\nSo the ratio is not a constant property of the model. It grows with the question count toward roughly $129 / 7.2 \\approx 18$ and is below 1 for a single question, where the 9B model is twice as fast as the 27B one (reasoned). The card says both of those things plainly. Two things the headline leaves out: the comparison is against d3-flash, a third of d3-turbo's size, not against d3 at the same size, which the card does not time; and the free API takes 64 questions per request, so 9.5× is the most an API user can see (reported limits, reasoned ratio).\n\nOne more thing the code shows. Box, track and image-in-question requests do not take the cached path in the shipped `AutoModel` runtime. `_system_one_composed` runs \"every question as a row of its own under the block mask\", so each Box question recomputes the state block, image included (measured from `d3_runtime.py`). Its `usage` still counts the state once. The card says the vllm-sr Engine \"answers Box and track questions too\" from one read and that 64 questions about one photo take 0.50 s there. I did not read or run the Engine, so that part is unverified.\n\n## Box, track and images in questions\n\nThese are the new question types, and they reuse the same readout rather than adding a detection head. A Box question's prompt ends with four lines, `top:`, `bottom:`, `left:`, `right:`, and the instruction to give each edge \"as a code on a 128-step scale\" (`d3_compose.py`). The runtime reads the readout at each colon. Each edge becomes a distribution over the first 128 codes, and the edge is its expectation:\n\n$$\n\\hat e = \\sum_{k=0}^{127} p_k \\,\\frac{k + 0.5}{128}\n$$\n\nIf the expected top lands below the expected bottom, `order_edges` collapses the pair to a one-bin sliver around their midpoint. There is always exactly one box, even when the object is absent, which is why the card says to pair a Box question with a yes/no one.\n\nThe confidence is not a probability the model outputs. It is computed from the four distributions. The runtime draws 512 boxes, one per point of a quasi-random R4 sequence, by inverting each edge's cumulative distribution. It counts the share $c$ whose IoU with the reported box is at least 0.5, and maps $c$ through a 38-point piecewise-linear table in `decision_config.json`:\n\n$$\n\\text{confidence} = g\\!\\left(\\frac{1}{512}\\sum_{i=1}^{512} \\mathbf{1}\\!\\left[\\operatorname{IoU}(b_i, \\hat b) \\ge 0.5\\right]\\right)\n$$\n\nThe table lifts the middle of the range: a raw 0.53 becomes 0.67 and 0.35 becomes 0.50 (measured from the table). Where the calibration data came from is not published.\n\nThis is the one piece of the release I could run on a CPU without the weights, so I did, under the run exception for this piece. I loaded the repository's `d3_compose.py` (numpy 2.5.3, on a shared 4-core cloud VM) and fed it synthetic Gaussian edge distributions (measured):\n\n| Synthetic case | Raw share | Calibrated |\n| --- | ---: | ---: |\n| 40% box, edge spread 0.4% of the image | 1.0000 | 1.0000 |\n| 40% box, spread 5% | 0.9551 | 0.9898 |\n| 40% box, spread 10% | 0.4453 | 0.6341 |\n| 10% box, spread 2% | 0.6699 | 0.8245 |\n| top and bottom crossed | 0.0000 | 0.0000 |\n\nThe fourth row is the one to remember. The same 2% edge uncertainty that gives a 40% box a confidence of 1.00 gives a 10% box 0.82, because IoU punishes a fixed error more on a small box. Confidence here is \"how likely is this box to overlap the truth at IoU 0.5\", which is the right question for a detector, and it means small objects read as less certain for the same localisation error. The widget below is a TypeScript port of the same functions; it matches the Python to four decimals on all five cases (measured).\n\n\n\nA track is the same machinery over time. The clip is read in windows of up to 32 frames at twice the requested rate, so 16-second windows at 1 update per second and 8-second windows at 2 (measured from `d3_compose.py`). Each window is a state of its own, and each moment is a Box question with an extra `visible:` line read over two codes.\n\nImages inside questions and options work because the composed path places an image's tokens in that question's block. Only that question sees it, and under `prefix_bidir` the state never does.\n\n\n\nThe Box and tracking scores (0.90 accuracy at IoU 0.5, confidence AUROC 0.96, success AUC 0.79 and 0.80, presence F1 0.97, and 0.96 accuracy for images in options) are on \"our Box test set\" and \"our tracking test clips\" (reported). Neither set is published, so I cannot check them or say how hard they are.\n\n## \"#1 on both boards\"\n\nThe tweet does not say \"estimate\". The card does: \"70.2 on the Jev Decision Index, +4.0 over d3 (66.3). Our estimate, not an official board result\", and its rank figure is labelled a 0.3.2 snapshot of 16 October.\n\n\n\nThe estimate itself holds up against the snapshot it names. On the Space's `data/v03.json` at the 16 October commit, the text board is led by StartLux-Decision v1.1 at 69.12 with d3 second at 66.26, and the vision board by Amos 1 Large at 72.4 (measured). 70.2 and 72.9 would have been first on both.\n\nBut the board did not stay on 0.3.2. The post went out at 00:42 UTC on 20 October (from the post id). At 13:53 UTC the same day the Space published Decision Index 0.4, which scores the full index from the private parts only, 60% same skills and 40% new domains, with a 0.9-point tie band (measured from `formula` in the data file). d3-turbo was scored officially from then on (measured, `data/v03.json` and `data/vision.json` at each commit):\n\n| Board snapshot | Text: d3-turbo | Text: d3 | Vision: d3-turbo | Vision: d3 |\n| --- | ---: | ---: | ---: | ---: |\n| 0.4, 20 Oct | 69.04, 6th | 69.06, 4th | 72.35, 2nd | 71.26, 5th |\n| 0.4.1, 21 Oct | 67.31, 9th | 67.15, 10th | 72.46, **1st** | 71.26, 7th |\n| 0.4.1, 22 Oct (current) | 67.31, 10th of 229 | 67.15, 11th | 72.46, 2nd of 55 | 71.26, 9th |\n\nOn the current text board, d3-turbo is 10th of 229 by score, tied with d3. Ranks are by score; v1.0 was scored on 20 October and v1.1, the weights released before the post, from 21 October. On text, d3-turbo and d3 have shared a tie group in every snapshot, 0.02 and 0.16 points apart against a 0.9 band. It was never first. On vision it was first from 21 October 15:13 UTC until RBDash-Decision v0.2 joined at 73.3 on 22 October 20:12 UTC, about 29 hours (measured from the commit times).\n\nThe components say where the gain is (measured, current snapshot):\n\n| Part | d3-turbo v1.1 | d3 | Difference |\n| --- | ---: | ---: | ---: |\n| Text, public benchmarks | 71.62 | 65.45 | +6.17 |\n| Text, private same skills | 64.29 | 62.83 | +1.46 |\n| Text, private new domains | 56.41 | 57.86 | −1.45 |\n| Vision, public | 76.84 | 74.08 | +2.76 |\n| Vision, private | 68.08 | 68.45 | −0.37 |\n\nThe thread's \"+6.2 on the public suite\" is real: the board's own public score agrees to 0.03 (reasoned from the two). That gain is mostly on the public benchmarks, the part of the index a team can measure for itself and that 0.4 stopped counting. On the private tests nobody can train on, d3-turbo is a wash against d3 on text and a third of a point behind on vision.\n\nA third-party run says the same. SaylorTwift's [Typed Decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) evaluation, linked from the card's own eval results, puts d3-turbo at 0.736 accuracy and ECE 0.069 over 400 cases and 2,000 decisions, behind d3 at 0.749 and 0.049 and d3-mini at 0.743 (reported, from the bucket's `leaderboard.md`). d3-turbo ran through `transformers` and the others through vLLM, which \"does not yet support\" the new attention mode, so the serving paths differ. The 0.013 accuracy gap is about 26 decisions, a little over one standard error at that sample size (reasoned). It does not show d3-turbo worse. It does show it is not better.\n\n## Smaller things I found\n\n- **The long-context number.** The card says `long_context=True` allows 131,072 tokens per question. The shipped runtime's `question_cap` returns `LONG_CONTEXT_TOKENS = 262_144` when the package declares no `max_length`, and d3-turbo's declares none (measured from `d3_runtime.py` and `decision_config.json`). Its docstring calls that mode \"slower and less accurate, as the model card says\"; the card says neither. The default cap is the `validated_length` of 32,768 tokens, and a question over a cap is refused, never truncated (measured).\n- **Size.** 26.09B parameters is 52.17 GB of BF16 shards (measured from the Hub listing). It needs one large accelerator, not a laptop, which is why I ran only the decoder.\n- **Licence.** Apache-2.0 for the weights, readout and runtime, per `MODEL_MANIFEST.json`, with a one-line NOTICE (measured). The example media keep their own CC licences.\n\n## What to take from it\n\nEncoders and decoders have long been one mask apart. Decision models have now been through three masks in a month. Causal reads the state once but sees the question only at the end. Fully noncausal sees everything and pays for the state on every question. `prefix_bidir` gives the state a bidirectional view of itself, gives the question a bidirectional view of itself and the state, and keeps the one property that makes caching legal: the state never looks forward at the question.\n\nThat is a good design for exactly the workload these models are sold for, many cheap questions about one expensive input. The 15.5× is real for that workload, and smaller everywhere else. The accuracy claim is weaker than the post makes it: d3-turbo is d3's accuracy on held-out tests, with d3's latency problem fixed.\n\n*Sources, read on 2026-10-23: the [launch post](https://x.com/XunzhuoLiu/status/2112343603209261092) and its thread; [vllm-sr/d3-turbo](https://huggingface.co/vllm-sr/d3-turbo) at `a0f3e9e` (`README.md`, `config.json`, `decision_config.json`, `MODEL_MANIFEST.json`, `d3_runtime.py`, `d3_compose.py`, `d3_format.py`, `.eval_results/`); [vllm-sr/d3](https://huggingface.co/vllm-sr/d3) and [vllm-sr/d3-flash](https://huggingface.co/vllm-sr/d3-flash) configs; [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) weights by range request; the [Decision Index Space](https://huggingface.co/spaces/multimodalart/jev-decision-index) data files at commits `0ddf6758`, `de1f4ea8`, `16f312e2`, `76fbe376` and `0b5ecaf2`; and the [Typed Decisions eval bucket](https://huggingface.co/buckets/SaylorTwift/typed-decisions-evals). I ran only the repository's Box decoder, on synthetic inputs; no weights were loaded. Figures are reproduced for commentary from the Apache-2.0 model card; the two interactives are original.*\n","readingTimeMins":18,"url":"https://ai.thesatyajit.com/articles/d3-turbo","lastUpdated":"2026-10-23","facts":{"words":3674,"readingTimeMins":18,"figures":3,"interactives":2,"film":true,"citations":7,"cards":1,"measured":26,"inbound":5,"lastUpdated":"2026-10-23"},"score":61.4,"tier":{"id":"notable","label":"Notable","level":3},"rank":304,"lenses":{"must-read":61.4,"run-it":8,"learn":6,"new":5.5,"deep":12},"highlights":["Checked against the source","Concrete numbers to act on","Explained from first principles"],"signal":{"interest":null,"helpful":null,"score":null,"level":3,"label":"Notable","source":"tier"}},{"title":"DeepInfer: DeepSeek's serving engine, and a throughput chart that multiplies by 3.25","description":"DeepSeek open-sourced DeepInfer, the engine behind DeepSeek-V4.1-Flash, plus deepseek-recipe, its request encoder. I read the code and ran what runs on a CPU. The release is the engine, not the service: KV storage, prefill-to-decode transfer and load balancing are described in a Q&A but left out. The scheduler is C++, the kernels are TileLang plus DeepSeek's own libraries, and DSpark's verify length is chosen by a greedy against a profiled step-time curve. The headline 26,270 tokens/s per GPU is offline decode with an assumed acceptance length of 3.25, and the viral chart that put it on SemiAnalysis's InferenceMAX plot mixed a mean with a P90 and per-decode-GPU with per-all-GPU. Normalised the same way, the gap to vLLM's best GB300 run is about 1.5x, not 4x.","date":"2026-10-23","tags":["inference","deepseek","kv-cache","speculative-decoding","mixture-of-experts","benchmarks","vllm"],"draft":false,"featured":false,"rating":{"novelty":1,"verification":3,"runnable":1,"explains":2,"takeaway":2,"durability":2,"reach":1,"unique":2,"why":"DeepInfer read from the code: what shipped vs what's described, the DSpark verify greedy rebuilt, and the viral chart's numbers traced back to their sources."},"topic":"inference-serving","level":"practitioner","runsOn":"datacenter","licence":"Apache-2.0","articleKind":"tool","kind":"articles","slug":"deepinfer","body":"On 20 October DeepSeek posted a four-part thread: \"Introducing DeepInfer, our in-house inference\nframework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low\nlatency.\" The second post lists \"High-performance kernels & optimized inference engine\", \"Distributed\ninference & speculative decoding\" and \"SWA Bounded Replay & structured outputs\". The third adds\n**deepseek-recipe**, \"our production token encoding & decoding component\".\n\nAn hour later a community chart went round. It took SemiAnalysis's InferenceMAX plot for\nDeepSeek-V4.1-Flash, where the best configuration tops out near 6,000 output tokens a second per chip,\nand drew an arrow off the top of the page to \"25600 TPS/GPU\" for DeepInfer. One reply said the real gain\nwas 1.3x to 1.5x. Another asked to see the maths.\n\nSo I read the code, ran what runs on a CPU, and traced every number in that chart to its source. Labels\nthroughout: **measured** means I computed it from a file or a run, **reported** is DeepSeek's or\nSemiAnalysis's figure taken as given, **reasoned** is my arithmetic on the other two.\n\n\n\n## What shipped, and what didn't\n\nThe README is unusually direct about this. \"This open-source release provides offline inference with the\ncore features needed for serving at scale.\" Then, under \"Required for a complete inference service (not\nincluded in this release)\": a KV cache system, \"VE/Prefill/Decode disaggregation (most required engine\nfunctionality is already implemented)\", and load balancing across data-parallel replicas.\n\nThe code agrees. `LLMEngine`'s docstring says the `LLM` class wraps it \"for offline batched inference.\nThis release does not include an online-serving wrapper.\" The hooks that would save KV to the store or\nhand it to a decode machine (`init_pd_split`, `pe_prepare_save_kvcache`, `pe_do_save_kvcache`) sit under a\ncomment that translates as \"from here on these are all virtual functions\", and each body is a docstring.\nWhat does ship is the engine that runs on either side of that split: `EngineType` has `NORMAL`,\n`PROMPT` (prefill only) and `COMPLETION` (decode only), and the scheduler handles all three (measured).\n\nThe repository is 348 tracked files: 270 Python files holding 54,929 lines, and 5,908 lines of C++ in 15\n`.cpp` and 18 `.hpp` files. There are no `.cu` files at all (measured). Twenty-three Python files open with\n`# Derived from vLLM; modified by DeepSeek.`, and the acknowledgements say it \"was originally built\nprimarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM\" (measured, reported). The\nlicence is Apache-2.0; deepseek-recipe is MIT (measured).\n\nWhat fills the gap is `doc/inference_deployment_qa.md`, a long Q&A describing the whole service. Almost\neverything below about storage and balancing comes from it, so it is reported.\n\n## The system the engine was cut from\n\n
\n\nThe shape is the one DeepSeek described for V3: separate prefill and decode, attention in data parallel,\nthe MoE in expert parallel with as many EP ranks as DP ranks. What changed is the numbers (reported):\n\n- **Prefill** runs EP2 to EP8, \"the smallest EP/DP that HBM allows\". **Decode** runs EP64 to EP128,\n \"the largest EP/DP that the available GPUs allow\". No tensor parallelism.\n- Prefill and decode GPUs come in roughly a 1:1 ratio, so \"the smallest efficient deployment unit is 64\n GPUs of prefill plus 64 GPUs of decode\".\n- The KV cache lives on a dedicated 3FS cluster of about 100 PiB, kept 3 days for agent traffic and 7 for\n chatbot traffic, and decode writes its KV back too. Any prefill machine can hit any prefix, so the\n prefill balancer does no prefix-affinity routing.\n- \"A single cluster currently serves 155 trillion tokens per day, with overall revenue of US\\$38 per\n GPU-hour (at valley pricing).\" Neither number can be checked from the repository.\n\nHere is one request through it, with what each stage holds. Solid boxes are code you can read today;\ndashed ones are prose.\n\n\n\nIn words: deepseek-recipe turns the API request into tokens. The entrypoint matches the prefix against\nstore metadata and computes the hit before routing. The prefill balancer buckets requests into four\ncontext tiers (under 16K, 16K to 64K, 64K to 256K, 256K and up). A prefill engine swaps the cached KV in\nlayer by layer over GPUDirect RDMA while its forward runs, and computes the rest. The KV and the\nsliding-window state move to a decode machine. The decode balancer places the request to even out both\nrequest count and total KV length. The decode engine drafts, verifies and streams tokens back.\n\n## Inside the engine: C++ schedules, Python launches\n\nThe control loop is Python (`LLMEngine.step`), but the scheduler, sequence state, block manager,\ninput-tensor preparation and output processing are C++ behind pybind11. `Scheduler::_schedule` releases\nthe GIL on entry. Three design choices stand out against [vLLM](/articles/vllm) and\n[SGLang](/articles/sglang).\n\n**One step type per DP group.** vLLM's V1 scheduler mixes prefill chunks and decode tokens in one batch.\nDeepInfer's `NORMAL` engine never does: each step the whole DP group votes, through an all-reduce, on\nwhether to run prefill or decode. The comment says only that a DP group must run one step type; the\nMoE all-to-all, which every rank joins, is the likely reason (reasoned). Every `prefill_attempt_interval` decode steps it asks whether *all*\nranks have prefill work; every `prefill_any_attempt_interval` steps it asks whether *any* do, so a lone\nrank's prompt waits a bounded number of steps (measured, from `scheduler.hpp`).\n\n**Prefix caching is not the engine's job.** vLLM hashes blocks inside the engine; SGLang keeps a radix\ntree. DeepInfer's `BlockManager` is a reference-counted free list with no hashing at all: the prefix\nindex lives in the entrypoint and the bytes in 3FS. Decode admits sequences shortest-output-first and,\nwhen blocks run out, preempts from the longest-output end. The header comment says it preempts the\nshortest; the code sorts by output length descending and evicts `running[l]`, the longest (measured,\n`_schedule_decode`).\n\n**Async scheduling.** It follows SGLang and NanoFlow, the Q&A says: while the GPU runs `forward(N)` and\n`sample(N)`, the CPU runs `schedule(N+1)`, launches it, and only then post-processes step N−1. That is why\nthe C++ keeps two counters per sequence (`schedule_cnt`, `process_cnt`) and accepts being one step behind\non acceptance rates.\n\n## A KV cache for three kinds of attention memory\n\nDeepSeek-V4.1-Flash has 40 layers, a 128-token sliding window, compressed attention with a sparse\nindexer, and two Engram n-gram tables at layers 1 and 14 (measured, from the Hugging Face `config.json`).\nThe [V4.1-Flash-1017 article](/articles/deepseek-v4-1-flash-1017) covers the weights; here is what the\nserving side keeps per request.\n\n`cache_engine.py` allocates two pools. The **paged pool** holds the compressed main KV and the indexer\nkeys in fixed blocks, stored in FP4: \"per-16, e4m3\" for main KV and MXFP4 for the indexer (reported).\nThe **state pool** gives each sequence one fixed `StateBlock`: an SWA ring per layer plus the compression\naccumulators. The ring is the 128-token window plus `DEEPINFER_MAX_BACKTRACKING_TOKENS`, which defaults to\ndraft length plus one, so 134 tokens at 528 bytes each (measured from `environ.py` and `cache_engine.py`).\n\nThe layout is ordered by what must persist. Only the first 20 layers' SWA goes to disk, and the Q&A\ngives the size: \"20 layers × 134 tokens × 528 B = 1,415,040 B per sequence\". The arithmetic holds\n(reasoned). The other 20 layers are the decoder half of V4.1's encoder-decoder split: their block KV is\nprojected from the last encoder layer, so after a cache hit only their SWA needs rebuilding.\n\nThat is what **SWA bounded replay** buys. Rebuilding a sliding-window cache exactly means running the\nlast 2,560 tokens through 20 decoder layers. DeepInfer recomputes only the last 128, truncating each\nquery's window to the replayed span, which \"yields an approximate SWA KV cache\". The README's default\nkeeps 140 tokens. The Q&A says this nearly halves total prefill compute (reported). It is a lossy\nshortcut, and the release says so: approximate, used for decode only, never written to the prefix\ncache.\n\nThe **Engram tables** are handled the opposite way from the community recipes. `engram.py` shards them\nacross an engram-parallel group into a DeepEP `EngramBuffer` and fetches rows by n-gram hash across\nranks (measured). SemiAnalysis's vLLM recipes for the same model offload the tables \"to host DRAM through\nUVA while the KV cache stays GPU-resident\" (reported, InferenceX changelog), because those recipes run\non four GPUs. Spread over 64 decode GPUs, the 202.76 GB of tables is about 3.2 GB each (reasoned).\n\n## Kernels: mostly reused, some new\n\nEvery GPU kernel in the tree is TileLang: 93 files and 23,331 lines under `tilelang_ops/`, with CUDA and\nAscend variants (measured). They cover the glue: Engram gathers, KV stores and loads, the DSpark\nverify-length kernel, sampling, structured-output masks. The heavy lifting is DeepSeek's existing\nlibraries, pinned as git submodules: [DeepGEMM](/articles/deepgemm), FlashMLA, DeepEP and DeepSelect,\nplus `tile-kernels` from PyPI and FlashAttention-4 for the vision encoder only (measured). TileLang\nitself is covered [here](/articles/tilelang).\n\nOn whether the public kernels are the production ones: the internal versions are \"broadly the same\";\npublic FlashMLA \"lacks one hand-tuned SASS optimization, which makes the decoding kernel 5%–10% slower\",\nand end-to-end performance is \"at most 5% apart\" (reported).\n\n## Expert parallelism and EPLB\n\nWith 384 routed experts and 64 redundant slots at EP64, each GPU holds 448 / 64 = 7, matching the Q&A's\n\"7 experts per GPU\" (reasoned). The policy in `eplb/policy.py` is pure PyTorch, so I ran it on a CPU. On\nsynthetic Zipf-skewed loads over 384 experts, a plain six-per-GPU placement gave a max-over-mean GPU load\nof 3.58 at skew 0.6 and 10.50 at skew 1.0; after the policy placed 448 copies it gave 1.004 and 1.009\n(measured, 2 threads on a shared 4-core VM). Those numbers are in-sample, planned and scored on the same\ncounts. The Q&A's production figure, re-planned every 10,000 steps on a 1,000-step window, is\n`moe_lb` = 1.090 (reported).\n\n## DSpark: how many drafts to verify\n\n[DSpark](/articles/deepseek-dspark) drafts five tokens and a confidence head estimates each one's chance\nof acceptance. The DSpark paper describes a load-aware verifier; DeepInfer's version is\n`MTPVarlen::schedule` in `csrc/engine/mtp_varlen.hpp`, and it is not a threshold. It is a greedy:\n\n1. For every draft of every request, take the cumulative acceptance rate: the chance it *and every draft\n before it* survive. Sort all of them, across the batch, highest first.\n2. Start with one query token per request. Add drafts down the sorted list one at a time, tracking\n expected tokens and the step time from a profiled curve of step time against query tokens.\n3. Keep the size where expected tokens per millisecond peaks; then give each request one extra query\n token per draft of its own that made the cut.\n\nBecause cumulative rates fall within a request, the cut always keeps a prefix of each request's drafts.\nUnder asynchronous scheduling the CPU fixes only the total, and a TileLang kernel redistributes it on the\nGPU with the freshest confidences.\n\nTo see the trade, I rebuilt the greedy in the widget below. The step-time curve is DeepInfer's own\ndecode chart read back as milliseconds per step: 9.99 ms at 32 query tokens per rank, 13.20 at 256, 18.21\nat 512 and 31.53 at 1,024 (reasoned, from the chart in the next section).\n\n\n\nBelow a few hundred query tokens, a step costs about the same however many tokens it carries, so\nverifying all five drafts is free and the greedy keeps them. Past that, each extra draft costs time and\nthe low-probability tail is cut first. With mean acceptance 0.75 and 60 requests per rank, my\nreconstruction verifies 3.88 drafts per request and gains 2.8% over verifying all five; at 96 requests,\n3.42 drafts and 8.2% (reasoned). The Q&A's own table, from EP64 benchmark data: 3.58 drafts and +5.4% at\n60 requests per rank, 3.24 and +11.0% at 96 (reported). Its fixed-five throughput, 13,730 tokens/s per\nrank at 60, sits within 4% of my curve's 13,271 (reasoned), which says the curve I read back is about\nright.\n\n## What the benchmark measures\n\n
\n\nThe README says these were \"Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one\nscale-up domain\". The script that makes the plot is `examples/sweep/run_sweep.sh`, and the axes come\nfrom `plot_pareto.py`:\n\n```python\n# examples/sweep/plot_pareto.py (abridged)\ndef decode_tau(points, draft_acceptance):\n \"\"\"τ = 1 + nextn × a\"\"\"\n assert not points[0][\"mtp_varlen\"]\n return 1 + points[0][\"nextn\"] * draft_acceptance\n\nxs = [p[\"decode_avg_sps\"] * tau for p in points]\nys = [p[\"decode_avg_sps\"] * p[\"decode_num_requests\"] * tau / p[\"cp\"] for p in points]\n```\n\n`decode_avg_sps` is steps per second, measured over 60 steps after 50 of warm-up, with fixed-length\nverification; a comment in `benchmark.py` translates as \"each step always commits 1+nextn tokens, so it\ndoes not depend on the acceptance rate\". Then both axes are multiplied by τ = 1 + 3 × 0.75 = 3.25, which\nis an input to the plotting script, not a measurement. Every request runs `ignore_eos` at 131,072\ntokens of context with 4,096 not cached (measured, from the scripts).\n\nReading the six points off the PNG's pixels against its gridlines gives 26,270 tokens/s per GPU at 103.1\ntokens/s per user, 22,806 at 178.5, 15,746 at 246.2, down to 2,619 at 325.2. Divide the first axis by\nthe second and you get the batch label every time, 254.9 for 256 and 64.0 for 64, because the y-axis is\nthe x-axis times requests per rank (measured). Per-user speed over 3.25 gives the step time: 31.53 ms at\n256 requests and 9.99 ms at 8 (reasoned).\n\nHow fair is 3.25? Two things say it is optimistic. Speculative acceptance is a chain: a draft only counts\nif every draft before it was accepted. The Q&A reports a fixed-five accepted length of 3.29 at the\nproduction operating point; a chain with a constant 0.75 per position reproduces that (1 + 0.75 + … +\n0.75⁵ = 3.29), and the same chain cut at three drafts gives 2.73, not 3.25 (reasoned). Independently,\nSemiAnalysis measured V4.1-Flash's DSpark acceptance on SPEED-Bench coding prompts: 3.27 at three drafts\nwith thinking off, 2.90 with thinking on, and 3.51 at five drafts with thinking on (reported). DeepSeek's\nown 3.29 at five drafts sits below even that. Scale 3.25 down to about 2.7 and the decode chart drops by\nroughly a sixth (reasoned).\n\n
\n\nThe Wide-EP chart makes the case for 64-GPU decode. At 64 requests per rank, EP8 reaches 12,646 tokens/s\nper GPU at 197.7 per user; EP64 reaches 15,746 at 246.2 (measured from pixels). Fewer experts per GPU, fewer weight bytes read per step.\n\n
\n\nPrefill on four GPUs runs 60,051 tokens/s per GPU at 4,096 tokens a step in 68.3 ms, and 89,474 at 65,536\ntokens in 731.7 ms (measured from pixels). This curve has no assumed factor in it.\n\n## The chart that ran off the page\n\n
\n\nThe red text reads: \"Output 25600 TPS/GPU, 100 TPS/User P90 output speed\"; \"23500 … 175\"; \"15000 … 250\";\n\"All data in the figure is B300 DeepInfer on 64 cards. Parallelism: DP64 EP64.\" The post says \"Data comes\nfrom SemiAnalysis's InferenceMAX\". Four things about that:\n\n1. **The DeepInfer numbers are not InferenceMAX's.** They are DeepInfer's README chart, read by eye:\n 25,600 at 100 against 26,270 at 103.1, and so on. SemiAnalysis's benchmark repository contains no\n DeepInfer configuration at all (measured, by search of a fresh clone).\n2. **\"P90\" is wrong.** InferenceMAX's x-axis is P90 interactivity across a trace replay. DeepInfer's\n is the mean step rate of a fixed batch, times 3.25.\n3. **\"B300\" is unverified.** The README says SM100 and \"one scale-up domain\" of 64 GPUs. An eight-GPU\n B300 box is not a 64-GPU NVLink domain; that needs a rack-scale system such as GB200 or GB300\n NVL72 (reasoned). DeepSeek doesn't say which.\n4. **Per GPU means different GPUs.** For a disaggregated row, InferenceMAX's API divides output by\n prefill plus decode GPUs: GB300 Dynamo vLLM at concurrency 7,480 reports 250,443 tokens/s on 18 + 16\n GPUs, 7,366 per GPU (measured, API rows fetched 23 October). DeepInfer divides by decode GPUs only,\n and its Q&A says prefill and decode come 1:1.\n\nPut both on one denominator (decode GPUs) at matched interactivity, and interpolate DeepInfer's curve:\n\n| Run | Per-user speed | Output tok/s per decode GPU | DeepInfer at that speed | Ratio |\n|---|---:|---:|---:|---:|\n| GB300 Dynamo vLLM, DEP16 decode, c7,480 | 145.6 (P90) | 15,653 | 24,317 | 1.55x |\n| GB300 Dynamo vLLM, DEP32 decode, c10,888 | 196.5 (P90) | 13,732 | 20,929 | 1.52x |\n| GB300 Dynamo SGLang, DEP16 decode, c4,096 | 146.8 (P90) | 9,456 | 24,262 | 2.57x |\n| GB300 Dynamo SGLang, DEP16 decode, c2,884 | 196.5 (P90) | 6,978 | 20,929 | 3.00x |\n\nThe InferenceMAX columns are measured from its API; DeepInfer's column and the ratios are reasoned.\nAgainst the best vLLM run the gap is about 1.5x, which is what the \"1.3x-1.5x\" reply guessed, and before\ntaking 3.25 down toward 2.7. It is still not like for like. DeepInfer's side is offline decode with no\narrivals, no prefill interference and a mean; InferenceMAX's is an agentic trace replay at P90 with\nmean inputs near 140K tokens, on EP16 or EP32 rather than EP64, and with its own simulated acceptance.\nThe honest summary is that DeepInfer's decode looks clearly ahead of the open engines' best published\nV4.1 runs, by less than the chart drew. Nobody has yet run DeepInfer under InferenceMAX's harness. That\nneeds 64 GPUs and the store, router and transfer code DeepSeek kept back.\n\nFor a single-user number on similar hardware, see [Kimi K3 on GB300](/articles/kimi-k3-gb300-megakernel);\nfor why a Pareto chart is not a deployment, see [Qwen3.8 on vLLM](/articles/qwen38-pd-serving).\n\n## What I ran\n\nIn a Python 3.12 venv with CPU PyTorch 2.14.1, on a shared 4-core cloud VM, no GPU:\n\n- `tests/models/test_eplb_policy.py`: 26 passed, 6 skipped (CUDA cases), with an empty `deep_ep`\n stand-in so the package's imports resolve.\n- `tests/test_cuda_sync_guards.py`: 6 passed, 1 skipped. `tests/distributed/test_shm_broadcast.py`: 3\n passed.\n- The EPLB probe above.\n\nWhat did not run: the C++ extension `deepinfer._C`. `setup.py` builds it as a `CUDAExtension` linked\nagainst `libcuda`, so the scheduler, block manager and DSpark greedy tests (`test_sequence.py`,\n`test_flops_estimator.py`) cannot load without a CUDA toolkit. Hence the widget re-implements the greedy\nrather than calling it. I did not build or run deepseek-recipe's Rust crates.\n\n## Where this leaves things\n\n- **The release is the engine, not the service** (measured). KV storage, P/D transfer and the four load\n balancers are described, not shipped, and the Q&A says they are \"tightly coupled to our internal\n infrastructure\".\n- **The scheduler is C++**, with no in-engine prefix cache and one step type per DP group (measured).\n- **DSpark's verify length is a greedy over a profiled step-time curve**, worth a few percent at\n production concurrency and more under load (reported, and reproduced in shape, reasoned).\n- **The headline decode chart multiplies a measured step rate by an assumed 3.25**, which DeepSeek's own\n acceptance numbers suggest is high by about a sixth (reasoned).\n- **The viral comparison mixed sources and denominators**; on a common denominator DeepInfer is about\n 1.5x vLLM's best GB300 V4.1 run (reasoned).\n- **Not verified:** the GPU model, the 155-trillion-tokens-a-day and \\$38-per-GPU-hour figures, and\n anything about the unreleased store and balancers.\n\nFor the request side, deepseek-recipe is the more immediately usable piece: an MIT Rust library that\nrenders V4 and V4.1 prompts and parses DSML tool calls the way the official API does. Combine it with\n[DeepSeek Harness](/articles/deepseek-harness) and the client end of DeepSeek's stack is now open. The\nserving end is open as far as one engine, which is further than [most labs go](/articles/glm-built-its-own-inference-infra).\nIf [prefill and decode](/articles/how-llm-inference-works) are new to you, start there.\n\n\n","readingTimeMins":19,"url":"https://ai.thesatyajit.com/articles/deepinfer","lastUpdated":"2026-10-23","facts":{"words":3747,"readingTimeMins":19,"figures":5,"interactives":2,"film":true,"citations":0,"cards":2,"measured":26,"inbound":5,"lastUpdated":"2026-10-23"},"score":65.2,"tier":{"id":"notable","label":"Notable","level":3},"rank":213,"lenses":{"must-read":65.2,"run-it":8,"learn":7.5,"new":5.5,"deep":12},"highlights":["Checked against the source","A lasting reference","Concrete numbers to act on"],"signal":{"interest":null,"helpful":null,"score":null,"level":3,"label":"Notable","source":"tier"}},{"title":"EAGLE-4: speculation that prices itself, and where the 5.21x comes from","description":"EAGLE-4 keeps the one-layer drafter of EAGLE-3 and adds two things: a harness that stops drafting and trims verification when the next token is not worth its measured cost, and a rank-128 head that shortlists 128 tokens before scoring them exactly. I read the paper, both launch posts, the TokenSpeed and eagle-harness source and the drafter's safetensors header. The parameter counts hold (0.435B layer parameters against 1.665B for DFlash 2), the paper's 3.59x and LightSeek's 3.56x are different engines with different baselines, the shipped package measures 3.13x on TokenSpeed, and the 5.21x on agentic coding is one greedy user replaying SWE-smith sessions; at 16 users it is 1.94x.","date":"2026-10-23","tags":["speculative-decoding","inference","agentic-coding","benchmarks"],"draft":false,"featured":false,"rating":{"novelty":2,"verification":2,"runnable":1,"explains":2,"takeaway":2,"durability":1,"reach":1,"unique":2,"why":"How EAGLE-4 prices each draft step, its speedups reconciled across paper, LightSeek and the support matrix, and a calculator on TokenSpeed timings."},"topic":"inference-serving","level":"practitioner","runsOn":"datacenter","licence":"source-available","articleKind":"paper","kind":"articles","slug":"eagle-4","body":"On 20 October LightSeek, a day-0 engine partner, posted: *\"EAGLE-4 is here—a game changer for agentic coding, with massive decoding speedups.\"* Its pitch: *\"Train with TorchSpec. Serve with TokenSpeed.\"* The replies quoted **5.21x** back, and asked the right questions: is that only Qwen on B300s, and does acceptance hold when the drafter sees tool output?\n\nThree primary sources, and they do not report the same numbers. The [EAGLE-4 paper](https://github.com/eagle-hq/EAGLE-4/blob/main/EAGLE4_Paper.pdf) and the [`eagle-hq/EAGLE-4`](https://github.com/eagle-hq/EAGLE-4) repository come from Eagle AI (`@EagleCorp` on X; the paper's contact is EAGLE co-author Hongyang Zhang). The [Eagle launch post](https://blog.eagle.ai/blog/introducing-eagle-4) summarises the paper. [LightSeek's post](https://lightseek.org/blog/eagle-4) covers training in TorchSpec and serving in [TokenSpeed](https://github.com/lightseekorg/tokenspeed). I read all three, the harness and TokenSpeed source, and the drafter checkpoints' safetensors headers (range requests, no weights downloaded).\n\nEvery number below is labelled: **measured** (I computed it from a file or from code), **reported** (the publisher's figure, not re-run), or **reasoned** (my arithmetic on the other two). I ran no benchmark: the Qwen drafter alone is 6.18 GB and its target is a 27B model, so nothing on a CPU would say anything about B300 speed.\n\n## Speculation in one paragraph\n\nA decode step of a 27B model at batch 1 is memory-bound: the GPU streams every weight to produce one token. Speculative decoding has a small **drafter** guess several tokens ahead. The **target** then checks all of them in one forward pass and keeps the longest prefix it agrees with, plus one bonus token of its own. Verification is lossless by construction. For greedy decoding the kept prefix is exactly what the target would have produced. For sampling, rejection sampling (accept $\\hat x_i$ with probability $\\min(1, p_i(\\hat x_i)/q_i(\\hat x_i))$, resample from the residual on rejection) preserves the target's distribution. The [EAGLE-3 article](/articles/eagle-3-speculative-decoding) walks through it; [how LLM inference works](/articles/how-llm-inference-works) covers the memory-bound step.\n\nSo the speedup is a ratio of two things:\n\n$$\nS = \\frac{A}{t_\\text{verify} + k\\,t_\\text{draft} + t_\\text{other}} \\cdot t_\\text{plain}\n$$\n\n$A$ is the **acceptance length**: tokens advanced per round, bonus token included. The denominator is what a round costs: one verify forward, $k$ draft steps, and bookkeeping. $t_\\text{plain}$ is one ordinary decode step. If each drafted token survives with probability $a$, independently, along a chain of $k$, then $A = (1 - a^{k+1}) / (1 - a)$. That is a geometric series: token $i$ counts only if all tokens before it were accepted. Two consequences follow, and EAGLE-4 is built around both.\n\n1. **Acceptance length alone is not speed.** A deeper draft always raises $A$, but each extra step costs $t_\\text{draft}$ and adds less than the one before.\n2. **The cost side moves with load.** At batch 1, verifying 16 tokens costs about what decoding 1 does. At batch 512, verify is compute-bound, and every speculative token competes with real work.\n\n## The EAGLE lineage\n\nEach EAGLE generation raised the numerator.\n\n- **EAGLE-1** ([arXiv 2401.15077](https://arxiv.org/abs/2401.15077), January 2024) drafts at the **feature level**. A one-layer drafter reads the target's second-to-top hidden state and predicts the next feature, which the target's own LM head turns into a token. It also reads the embedding of the token one position ahead, so it knows which branch it is on. The tree was fixed.\n- **EAGLE-2** ([arXiv 2406.16858](https://arxiv.org/abs/2406.16858), June 2024) made the tree **dynamic**. A node's score is the product of the draft probabilities along its path. That cheaply estimates the chance the target accepts it.\n- **EAGLE-3** ([arXiv 2503.01840](https://arxiv.org/abs/2503.01840), March 2025) dropped feature regression for direct token prediction. It fuses low, middle and high target layers, and it trains with **training-time test**: the drafter learns from its own multi-step outputs, so deep drafts look at training time the way they will at inference. Draft quality began to scale with data. EAGLE-3.1 (Eldenk et al., [arXiv 2605.09992](https://arxiv.org/abs/2605.09992)) added post-normalisation and per-hidden-state RMSNorm against what it calls attention drift.\n\nNone of them priced the denominator. EAGLE-2's tree is sized by confidence, not by what the hardware charges for it. That is the gap EAGLE-4 fills.\n\n## What EAGLE-4 changes\n\n