Right formula, wrong variable: reading Modular's LLM Inference Handbook

handbook.modular.com is 57 pages and 70,493 words of practitioner documentation on LLM inference, Apache-2.0, with the Markdown source and every interactive widget in one repository. It covers metrics, batching, PagedAttention, prefix caching, KV offloading, prefill/decode disaggregation, speculative decoding, parallelism, quantization, FlashAttention and GPU architecture, with four calculators and eighteen simulators and visualizers built in.

It has been in my to-read pile since it appeared and I never wrote about it, because a summary of a handbook is worth nothing. What is worth something is checking it. This site has a pile of inference arithmetic already on the shelf — a decode ceiling worked against an RTX 5090 results file, a speculative-decoding sweep on MI300X read out of an appendix, a hybrid-attention KV budget recomputed from published geometry — and a handbook is exactly the kind of document you can hold those against.

So I did, formula by formula. The result is narrower and more useful than "vendor content is biased," which turns out not to be the finding at all.

Five of the handbook's rules are correct rules attached to the wrong variable. Almost none of them is wrong as mathematics. Each one takes a quantity whose name matches something the reader has to hand, and that something is a different number:

1. Leviathan's acceptance-length formula, meeting the acceptance rate your server logs.
2. The KV-cache shape, meeting num_attention_heads.
3. The serving-memory overhead, meeting weight precision — a variable the KV cache is independent of.
4. The weight footprint P × (Q / 8), meeting a nominal bit width that no real checkpoint has.
5. The tensor-parallel sharding limit, meeting attention heads instead of KV heads.

And they all land in the same place: the calculator, the code block, the sizing rule — the parts a reader copies, rather than the prose, which is mostly careful.

Whose handbook is it, and does that show

Start with the question everyone asks about vendor documentation, because the answer is not what I expected and it is checkable in about a minute.

Modular sells MAX and Mojo. Count the framework mentions across docs/ with word-boundary regexes, so --max-num-batched-tokens does not score as a mention of MAX, and at HEAD you get vLLM 117, SGLang 104, MAX 70. The vendor's own product is third in the vendor's own handbook.

The git history explains why, and it is the more interesting fact. The repository was opened on 2025-07-07, and every pull request merged into it came from a bentoml/… branch for the first thirteen months — up to #190 on 2026-07-29. From #192, four days later, the branches are modular/…. The same handful of authors carried straight through, and the two benchmarks the speculative-decoding page leans on are BentoML's own.

This handbook was BentoML's before it was Modular's. However that transfer happened, the even-handedness is inherited rather than authored, which is a much better explanation for it than corporate restraint.

Which lets you measure the push exactly, because the PR titles do not hide it: docs/add-max-mentions, docs/more-max-mentions, docs/add-max-command, docs/update-max-messaging.

MAX's mention density more than doubled in the eight weeks after the handover — the biggest relative move of the three, and unambiguously deliberate. It started so far behind that doubling left it third. Nine of the ten framework tab groups now list MAX first, and since none of them sets defaultValue, the command a reader sees by default on every page that shows one is max serve. Across 70,000 words there are 15 UTM-tagged links to modular.com, five of them request-demo.

That is the whole of the commercial surface, and it is in placement rather than in claims. The one head-to-head number — "Modular reported serving google/gemma-4-31B-it with MAX on an NVIDIA B200 at 2.5× faster P99 TTFT and 1.5× the throughput of vLLM" — is followed immediately by "Numbers like these are workload-specific, so treat them as a reason to benchmark your own traffic rather than as a ranking that holds everywhere." One page carries an unhedged marketing claim — "MAX serves diffusion models like FLUX up to 4x faster than native PyTorch", linked to a solutions page. The PagedAttention page credits vLLM as the originator in its first sentence about implementations.

And the single place in the handbook where Modular cites its own unpublished measurements is this, on prefill/decode disaggregation — a feature MAX ships:

A vendor using its private benchmark data to tell you when its own feature makes things worse is the opposite of the failure mode I went looking for. So the interesting criticism is not about who wrote it. It is about arithmetic.

The acceptance rate is not the acceptance rate

The speculative-decoding page gives three metrics and one formula. The metric:

Acceptance rate (α): The probability of accepting draft tokens by the target model.

The formula, correctly attributed to Leviathan et al.:

Both are right. They are about different quantities.

The handbook's own diagram pins down which semantics are in play, and it is the clearest statement of the problem in the whole document:

expand

Leviathan's $\alpha$ is the per-position conditional probability that the target accepts a draft token. What vLLM, SGLang and every engine I have read report as "acceptance rate" is accepted tokens divided by proposed tokens — which counts all $\gamma$ proposals in the denominator, including the ones after the first rejection that the round never reached. As $\gamma$ grows, more of the denominator is positions that were never in play, so the reported number falls even when the drafter has not changed at all.

Under that definition there is a trivial identity: if every round proposes $\gamma$ tokens and the target always supplies one, then

exactly, by construction. It is not a model. It is the definition rearranged.

Both formulas are in circulation on this site already, in different articles, and until now I had not noticed they were the same statement under two definitions: the EAGLE-3 piece reasons about geometric decay along the chain, and the Cinference read uses $1 + a \cdot k$ against a repository's published tables. Both are right. Which one you need depends entirely on which number you have.

Checking it against a real sweep

AMD and Embedded LLM's vLLM study publishes the thing that settles it: a proposal-length sweep with all three columns — $\gamma$, the mean accepted length, and the acceptance rate — for Qwen3-8B with an EAGLE-3 drafter on MATH500, MI300X.

Seven rows, seven exact hits for the identity, to the precision the study prints. And the handbook's formula, fed the acceptance rate the handbook's own metric bullet describes, is 0% off at $\gamma = 1$ and 55% low at $\gamma = 7$.

Worse than low — it points the wrong way.

The dashed curve peaks at $\gamma = 3$ and then falls. Read literally, it says that proposing more tokens makes rounds shorter. The measurement rises monotonically across the whole sweep. That matters because it is precisely the question the page's guidance answers: "Increasing γ helps only when τ is high; otherwise, performance may be negatively affected." True advice, and you cannot evaluate it off this curve.

Now recover the variable the formula actually wants. Invert $\tau = (1-\alpha^{\gamma+1})/(1-\alpha)$ at each row and you get the per-position acceptance that would have produced that mean accepted length:

The reported rate collapses 41% across the sweep. The recovered one moves 6%. The drafter is not degrading; the metric is averaging in positions the round never reached. And a single $\alpha = 0.85$ pushed through the handbook's formula reproduces all seven measured accepted lengths to within 3.6%.

That is the shape of the whole article in one result. The formula is right. The variable is wrong. And the two are printed on the same page, eighteen lines apart.

An independent confirmation, from a completely different engine: the Cinference tables published alongside its 450 tok/s claim give six matched (acceptance, tokens-per-round) pairs at $\gamma = 3$. All six fit $1 + a \cdot \gamma$ to the printed digits; the handbook's formula fed the same $a$ runs 6.4% to 26.1% low, worst on the story-writing workload where acceptance is 37.4%. Two engines, two vendors, two architectures, thirteen data points, thirteen exact fits for the identity.

The KV cache does not know how many query heads you have

The handbook gives the KV-cache shape in three places, and all three use the wrong head count.

The calculator on the KV-offloading page:

KV Cache Size (GB) = 2 × B × S × L × H × D × (Q / 8) / (1024^3)

with H labelled "Attention Heads (H)", described as "Number of attention heads per layer," and an info box pointing the reader at config.json for "the number of layers, hidden size, number of attention heads." The simplified mode substitutes the model's hidden dimension for H × D. And the PagedAttention page writes the allocation as 2 × num_layers × num_heads × head_dim × max_seq_len.

Grouped-query attention appears exactly once in the 70,000 words, in a bullet about L2 cache reuse. num_key_value_heads appears nowhere.

The handbook knows the right answer. It is in the prose, on a different page:

For Llama 3 8B in FP16, for example, a single 8K-token sequence holds about 1 GB of KV cache (learn more about the calculation). The weights take roughly 16 GB. On an 80 GB GPU, this leaves at most around 64 GB for the KV cache and other runtime data, giving a theoretical upper bound of roughly 60 such sequences.

Check it: Llama-3-8B is 32 layers, 8 KV heads, head dimension 128, so 2 × 32 × 8 × 128 × 2 bytes is 131,072 bytes per token, and 8,192 tokens is exactly 1.000 GiB. The paragraph is right, and 64 GB divided by that is 59.6 sequences, which is "roughly 60."

Now follow the link it gives you. Put the same model into the calculator, filling H with the attention-head count the calculator asks for, and it returns 4.00 GB. Fifteen sequences instead of sixty. The sentence and the tool it cites as its own working differ by a factor of four.

The factor is the GQA group size, and it is not small on anything modern. The KV calculator has no model presets, so here is the list the handbook's two sibling calculators offer — its own choice of exemplar models — at fp16:

Every number in that table comes from the model's own config.json. Six presets, six wrong answers, and the error grows with model size because the frontier has been widening the GQA ratio for years — the thing the formula divides by is the thing it does not have.

DeepSeek V3 is not a scaling error at all; it is the wrong shape. Multi-head latent attention caches one compressed latent per token per layer — kv_lora_rank 512 plus qk_rope_head_dim 64, 576 values — not 2 × H × D. vLLM says so in three lines (vllm/config/model.py):

if self.use_mla:
    # When using MLA during decode it becomes MQA
    return 1

At a 128K context the calculator claims 488 GiB where the model holds 8.6.

The stress test: a hybrid model

Grouped-query attention is one multiplier. Hybrid attention is a second, and it multiplies with the first. Qwen3-Next-80B-A3B publishes everything you need: 48 layers, full_attention_interval: 4, 16 attention heads, 2 KV heads, head dimension 256.

192 GiB against 6 GiB, and the 32× decomposes exactly: 8× because 16 query heads share 2 cached KV heads, 4× because only 12 of the 48 layers keep a cache at all. One of those numbers does not fit on an H200. The other fits on a laptop.

This is not an exotic case any more. Both large hybrids this site has read recently have the same shape. The Cinference write-up found 48 of Qwen3.8-27B's 64 layers holding a fixed 144 MiB recurrent state instead of a KV cache, which is most of why 256K context fits on a 32 GB consumer card. The vLLM PD-serving numbers are for a 92-layer model with 69 gated DeltaNet layers, where I recomputed a request's KV budget at 759 MiB and found 37.5% of it to be recurrent state that a 200-token request pays in full.

Which makes the flat claim on the KV-offloading page — "the KV cache size grows linearly with sequence length" — false for a growing share of what people are actually deploying. On Qwen3-Next at a 2,048-token request, 61% of the KV allocation does not grow with the sequence at all.

The KV cache is not a percentage of the weights

The GPU-memory page carries the sizing rule most readers will take away:

Memory (GB) = P * (Q / 8) * (1 + Overhead)

with overhead described as "Additional serving memory beyond weights, such as KV cache, activation buffers, workspace memory," typical range 10–30%. The calculator implements it verbatim: params * (precision / 8) * (1 + overhead / 100).

The page is honest about it being rough — "The percentage-based overhead is a quick sizing shortcut, not an exact capacity model" — and that is the right disclaimer for a number that is imprecise. This one is not imprecise. It is monotone in the wrong direction, because the overhead is multiplicative on the weight bytes, and the KV cache is not a function of weight precision.

Work it on Llama-3.1-8B, whose cache is 128 KiB per token whatever you do to the weights:

• bf16, 20% overhead. 8 × 2 × 1.2 = 19.2 GB required, of which 3.2 GB is the allowance for everything that is not weights. At 128 KiB/token that is 24,414 tokens of KV.
• 4-bit, 20% overhead. 8 × 0.5 × 1.2 = 4.8 GB required, allowance 0.8 GB. 6,103 tokens.

Quantizing the weights divided the cache budget by four. The cache did not move. And on real hardware the true capacity goes the other way: on a 24 GB card at 0.9 utilization, bf16 weights leave 5.54 GB for cache — about 42,000 tokens — while 4-bit weights leave 15.86 GB, about 121,000. The formula's allowance falls 4× where the actual headroom rises 2.9×.

The handbook states the correct version of this, once, in a sentence on the quantization page:

A smaller weight footprint also leaves more GPU memory available for the KV cache, larger batches, and more concurrent requests. Weight quantization does not reduce the KV cache size per token by itself. It requires quantizing the KV cache separately.

And then contradicts it, ninety lines later on the same page, in the list of reasons to quantize:

You want to support higher concurrency. Quantization reduces KV cache size per token, allowing more tokens (and therefore more parallel requests) to fit within the same GPU memory.

Same page, same document, opposite claims. The first is correct. The second is the assumption the calculator is built on.

Measured: what a 4-bit checkpoint actually weighs

The other half of that formula is the weight term, P × (Q / 8), and it is worth checking because it is the part everyone trusts. I measured it rather than argued about it: for five real checkpoints I pulled every .safetensors header over HTTP range requests — eight bytes of length prefix, then the JSON header — and summed data_offsets for the exact byte count.

Exact at bf16, 13.5% low at FP8, 43.4% low at 4-bit on the 8B. The remainder is the embedding table, the LM head, the norms and one scale per quantization group, none of which shrink with the matrices; a "4-bit" 8B checkpoint measures 5.71 bits per parameter. And the error is worst where the formula's users are: the fixed high-precision remainder is a larger share of a small model, so the 8B is off by more than the 70B on the same quantization scheme.

Put the two halves together and the handbook's memory calculator, asked about a 4-bit 8B model at 20% overhead, returns 4.8 GB for a deployment whose weights alone are 5.74 GB.

One more place the same variable is wrong

The parallelism page, on how far you can shard:

The number of GPUs you shard across is also constrained; it generally has to divide the number of attention heads evenly, so you can't scale to arbitrary GPU counts.

The binding constraint is the KV heads, and vLLM says so in one line with four of comment (vllm/config/model.py, 72675c4):

# If tensor parallelism is used, we divide the number of KV heads by
# the tensor parallel size. We will replicate the KV heads in the
# case where the number of KV heads is smaller than the tensor
# parallel size so each GPU has at least one KV head.
return max(1, total_num_kv_heads // parallel_config.tensor_parallel_size)

The operational consequence is not that a setting is rejected. It is that past tp_size = num_kv_heads the cache stops sharding and starts replicating. Llama-3.1-8B has 8 KV heads: at TP=8 each GPU holds one, and at TP=16 each GPU still holds one, so the cluster now stores twice as much KV cache for the same sequence. Sixteen GPUs give you the same KV capacity per sequence per GPU as eight. A reader following the attention-head rule would size TP=16 or TP=32 on this model and expect the cache to keep dividing.

There is a whole separate flag for the case, which is the strongest evidence I can offer that the KV heads are the real boundary. Decode context parallelism shards the cache along the sequence instead, and vLLM will only let you enable it above that line: "Decode context parallelism for GQA/MQA requires --tensor-parallel-size to be greater than the model's total number of KV heads." A knob that exists exclusively for tp_size > num_kv_heads is not a knob you need if num_attention_heads is the constraint.

What it gets right, which is most of it

None of the above should be read as a verdict on the document, because on the things the field most reliably gets wrong, this handbook is better than the field.

The metrics chapter makes a distinction almost nobody makes. It separates average TPOT from average ITL and explains that they differ in how you average — TPOT is request-weighted, ITL is token-weighted, so a long response moves one and not the other — and then says the sentence that should be the first line of every benchmark post: "always check whether TPS metrics refer to input, output, or a combined view." That is the same question the Cinference read had to ask before anything else: 450 tokens a second for whom.

expand

The benchmark template demands every denominator. Model, framework version, quantization, GPU and count, tensor and data parallelism, max batch size, request rate, concurrency, input length, output length — all of it before a single result row. That is the discipline that makes a throughput number mean something, and most published benchmarks supply about a third of it. The one slip is that the template's own Results block lists Throughput: 98 tokens/sec without saying input, output or combined, one chapter after telling you always to check.

Its one piece of bandwidth arithmetic is right, and I can check it against something I have. The GPU page gives the bandwidth-bound decode ceiling:

maximum decode tokens/sec ≈ memory bandwidth / bytes read per token

and works it: 140 GB of fp16 70B weights over an H100 SXM's 3.35 TB/s is about 24 tokens/s per sequence. (3,350 ÷ 140 = 23.9.)

This site has exactly one first-hand measurement of that ceiling, and it comes from the other end of the market: Cinference's non-speculative decode of a 27B NVFP4 model on an RTX 5090, streaming 19.729 GiB of weights per step at 1,792 GB/s of peak bandwidth.

Consumer Blackwell instead of Hopper, 4-bit weights instead of fp16, a hybrid attention stack instead of a dense one, a C++ engine instead of vLLM — and the rule lands within 16% at both ends of a 32× context sweep, with the residual almost constant. A constant residual is the sign that a model is capturing the whole structure and the gap is fixed overhead. The second row required adding the 6.69 GB KV term the handbook's worked example explicitly sets aside; put it in and the rule keeps working.

It publishes counter-evidence. The 20–30% disaggregation regression. The note that Hugging Face TGI is in maintenance mode and you should plan an exit. The observation that Ollama does not do concurrency, which makes half the optimizations in the handbook inapplicable to it. The repeated instruction to benchmark your own traffic rather than trust any of the numbers on the page, including theirs.

What is missing, and it is one thing

The words "compute-bound" and "memory-bound" appear twelve times across eight pages. The ratio that decides which side you are on appears nowhere. "Arithmetic intensity" occurs once in 70,000 words, in a passing clause about what tensor cores are for. There is no roofline, no FLOPs-per-byte, no ridge point.

What that costs the reader is the ability to check the dichotomy against their own request shape, and the dichotomy is not reliable. Prefill is not compute-bound because it is prefill — a 128-token prefill is memory-bound, because it has not accumulated enough work to amortize one pass over the weights. Decode is not memory-bound because it is decode — batch it hard enough and its intensity climbs toward the ridge, which is the actual reason continuous batching multiplies throughput while leaving per-token latency alone. This site's prefill-and-decode explainer plots both phases against an A100's ridge of 153 FLOP/byte, and the DGX Spark piece is what the batched end of that axis looks like in practice. A reader who has the ratio can derive the handbook's own decode-ceiling rule, and can tell when it stops applying. A reader who has the dichotomy has a mnemonic.

What I would take from it

• Read the prose, rebuild the calculators. The handbook's paragraphs get Llama-3-8B's KV cache right and its calculator gets it wrong by 4×, on the same model, with a link between them. That inversion — careful text, loose tool — is worth watching for in any technical document, because the tool is the part that gets copied.
• Never put a logged number into a textbook formula without checking the definition. "Acceptance rate" names two quantities that differ by 41% on a real sweep, and the formula that takes one of them is printed next to the definition of the other.
• The KV cache is decoupled from almost everything people couple it to. Not the query-head count, not the weight precision, and on a hybrid model not even the sequence length for a large fraction of the bytes. Price it from num_key_value_heads, the layers that actually have a cache, and the cache dtype — nothing else.
• Vendor authorship is the wrong thing to screen for. This handbook is more careful than most of the neutral writing on the same subjects, and where it is careless the carelessness is not commercial — it is the ordinary kind, in a calculator nobody went back and re-derived. Screen for whether the arithmetic can be checked. Here it can, which is the best thing about it.

Related reading: prefill, decode, and where the time goes; Cinference's 450 tokens a second; a Pareto frontier is not a deployment; speculative decoding on AMD GPUs; EAGLE-3 and the draft-model scaling law; vLLM read from the source; SGLang's radix tree; continuous batching on a small box; and VRAM is a policy, not a property, which is the same argument about a different resource.

Read at modular/llm-inference-handbook commit 0d64912 (2026-09-21, Apache-2.0), 57 pages and 70,493 words of docs/, plus the four calculator components under src/components/Calculator/. Model geometry from each model's own config.json on Hugging Face. Weight footprints measured from safetensors headers over HTTP range requests. vLLM source at 72675c4. The speculative-decoding sweep is AMD and Embedded LLM's, as read here earlier; the decode-ceiling measurement is Cinference's, as read here earlier. I ran no model and own none of the hardware in this article. Both figures are the handbook's own diagrams, rendered from the committed SVG and flattened onto white.