# Right formula, wrong variable: reading Modular's LLM Inference Handbook
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/modular-llm-inference-handbook
> date: 2026-09-22
> tags: inference, inference-optimization, kv-cache, speculative-decoding, quantization, benchmarks, systems, llm, explainer
[handbook.modular.com](https://handbook.modular.com/) is 57 pages and 70,493 words
of practitioner documentation on LLM inference, Apache-2.0, with the Markdown
source and every interactive widget in one repository. It covers metrics,
batching, PagedAttention, prefix caching, KV offloading, prefill/decode
disaggregation, speculative decoding, parallelism, quantization, FlashAttention
and GPU architecture, with four calculators and eighteen simulators and
visualizers built in.
It has been in my to-read pile since it appeared and I never wrote about it,
because a summary of a handbook is worth nothing. What is worth something is
checking it. This site has a pile of inference arithmetic already on the shelf
— a decode ceiling worked against an RTX 5090 results file, a
speculative-decoding sweep on MI300X read out of an appendix, a hybrid-attention
KV budget recomputed from published geometry — and a handbook is exactly the
kind of document you can hold those against.
So I did, formula by formula. The result is narrower and more useful than
"vendor content is biased," which turns out not to be the finding at all.
**Five of the handbook's rules are correct rules attached to the wrong
variable.** Almost none of them is wrong as mathematics. Each one takes a
quantity whose name matches something the reader has to hand, and that something
is a different number:
1. Leviathan's acceptance-length formula, meeting the acceptance rate your
server logs.
2. The KV-cache shape, meeting `num_attention_heads`.
3. The serving-memory overhead, meeting weight precision — a variable the KV
cache is independent of.
4. The weight footprint `P × (Q / 8)`, meeting a nominal bit width that no real
checkpoint has.
5. The tensor-parallel sharding limit, meeting attention heads instead of KV
heads.
And they all land in the same place: the calculator, the code block, the sizing
rule — the parts a reader copies, rather than the prose, which is mostly
careful.
## Whose handbook is it, and does that show
Start with the question everyone asks about vendor documentation, because the
answer is not what I expected and it is checkable in about a minute.
Modular sells MAX and Mojo. Count the framework mentions across `docs/` with
word-boundary regexes, so `--max-num-batched-tokens` does not score as a mention
of MAX, and at HEAD you get **vLLM 117, SGLang 104, MAX 70.** The vendor's own
product is third in the vendor's own handbook.
The git history explains why, and it is the more interesting fact. The
repository was opened on 2025-07-07, and every pull request merged into it came
from a `bentoml/…` branch for the first thirteen months — up to #190 on
2026-07-29. From #192, four days later, the branches are `modular/…`. The same
handful of authors carried straight through, and the two benchmarks the
speculative-decoding page leans on are BentoML's own.
**This handbook was BentoML's before it was Modular's.** However that transfer
happened, the even-handedness is inherited rather than authored, which is a much
better explanation for it than corporate restraint.
Which lets you measure the push exactly, because the PR titles do not hide it:
`docs/add-max-mentions`, `docs/more-max-mentions`, `docs/add-max-command`,
`docs/update-max-messaging`.
MAX's mention density more than doubled in the eight weeks after the handover —
the biggest relative move of the three, and unambiguously deliberate. It started
so far behind that doubling left it third. Nine of the ten framework tab groups
now list MAX first, and since none of them sets `defaultValue`, the command a
reader sees by default on every page that shows one is `max serve`. Across
70,000 words there are 15 UTM-tagged links to modular.com, five of them
`request-demo`.
That is the whole of the commercial surface, and it is in placement rather than
in claims. The one head-to-head number — "Modular reported serving
`google/gemma-4-31B-it` with MAX on an NVIDIA B200 at 2.5× faster P99 TTFT and
1.5× the throughput of vLLM" — is followed immediately by *"Numbers like these
are workload-specific, so treat them as a reason to benchmark your own traffic
rather than as a ranking that holds everywhere."* One page carries an unhedged
marketing claim — *"MAX serves diffusion models like FLUX up to 4x faster than
native PyTorch"*, linked to a solutions page. The PagedAttention page credits
vLLM as the originator in its first sentence about implementations.
And the single place in the handbook where Modular cites its own unpublished
measurements is this, on prefill/decode disaggregation — a feature MAX ships:
*"Thresholds matter: if your workload is too small, or your GPU setup isn't
tuned for this approach, performance can drop (by 20-30% in our tests)."*
A vendor using its private benchmark data to tell you when its own feature makes
things worse is the opposite of the failure mode I went looking for. So the
interesting criticism is not about who wrote it. It is about arithmetic.
## The acceptance rate is not the acceptance rate
The speculative-decoding page gives three metrics and one formula. The metric:
> **Acceptance rate (α)**: The probability of accepting draft tokens by the
> target model.
The formula, correctly attributed to [Leviathan et
al.](https://arxiv.org/abs/2211.17192):
$$
\tau = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}
$$
Both are right. They are about different quantities.
The handbook's own diagram pins down which semantics are in play, and it is the
clearest statement of the problem in the whole document:
Leviathan's $\alpha$ is the **per-position conditional** probability that the
target accepts a draft token. What vLLM, SGLang and every engine I have read
report as "acceptance rate" is **accepted tokens divided by proposed tokens** —
which counts all $\gamma$ proposals in the denominator, including the ones after
the first rejection that the round never reached. As $\gamma$ grows, more of the
denominator is positions that were never in play, so the reported number falls
even when the drafter has not changed at all.
Under that definition there is a trivial identity: if every round proposes
$\gamma$ tokens and the target always supplies one, then
$$
\tau = 1 + \text{AR} \cdot \gamma
$$
exactly, by construction. It is not a model. It is the definition rearranged.
Both formulas are in circulation on this site already, in different articles,
and until now I had not noticed they were the same statement under two
definitions: the [EAGLE-3 piece](/articles/eagle-3-speculative-decoding) reasons
about geometric decay along the chain, and the [Cinference
read](/articles/cinference) uses $1 + a \cdot k$ against a repository's published
tables. Both are right. Which one you need depends entirely on which number you
have.
### Checking it against a real sweep
[AMD and Embedded LLM's vLLM study](/articles/vllm-speculative-decoding-amd)
publishes the thing that settles it: a proposal-length sweep with **all three
columns** — $\gamma$, the mean accepted length, and the acceptance rate — for
Qwen3-8B with an EAGLE-3 drafter on MATH500, MI300X.
| γ | Mean accepted length | Acceptance rate | $1 + \text{AR}\cdot\gamma$ | Handbook τ, fed AR |
|---:|---:|---:|---:|---:|
| 1 | 1.89 | 89.0% | 1.890 | 1.890 |
| 2 | 2.64 | 82.2% | 2.644 | 2.498 |
| 3 | 3.27 | 75.6% | 3.268 | 2.760 |
| 4 | 3.75 | 68.7% | 3.748 | 2.706 |
| 5 | 4.14 | 62.8% | 4.140 | 2.523 |
| 6 | 4.43 | 57.2% | 4.432 | 2.290 |
| 7 | 4.68 | 52.5% | 4.675 | 2.093 |
Seven rows, seven exact hits for the identity, to the precision the study
prints. And the handbook's formula, fed the acceptance rate the handbook's own
metric bullet describes, is 0% off at $\gamma = 1$ and **55% low at $\gamma = 7$**.
Worse than low — it points the wrong way.
The dashed curve peaks at $\gamma = 3$ and then falls. Read literally, it says
that proposing more tokens makes rounds *shorter*. The measurement rises
monotonically across the whole sweep. That matters because it is precisely the
question the page's guidance answers: *"Increasing γ helps only when τ is
high; otherwise, performance may be negatively affected."* True advice, and you
cannot evaluate it off this curve.
Now recover the variable the formula actually wants. Invert
$\tau = (1-\alpha^{\gamma+1})/(1-\alpha)$ at each row and you get the
per-position acceptance that would have produced that mean accepted length:
| γ | Reported acceptance rate | Recovered per-position α |
|---:|---:|---:|
| 1 | 0.890 | 0.890 |
| 2 | 0.822 | 0.875 |
| 3 | 0.756 | 0.867 |
| 4 | 0.687 | 0.856 |
| 5 | 0.628 | 0.849 |
| 6 | 0.572 | 0.842 |
| 7 | 0.525 | 0.839 |
**The reported rate collapses 41% across the sweep. The recovered one moves 6%.**
The drafter is not degrading; the metric is averaging in positions the round
never reached. And a single $\alpha = 0.85$ pushed through the handbook's
formula reproduces all seven measured accepted lengths to within 3.6%.
That is the shape of the whole article in one result. The formula is right. The
variable is wrong. And the two are printed on the same page, eighteen lines
apart.
An independent confirmation, from a completely different engine: the Cinference
tables published alongside its 450 tok/s claim give six matched
(acceptance, tokens-per-round) pairs at $\gamma = 3$. All six fit
$1 + a \cdot \gamma$ to the printed digits; the handbook's formula fed the same
$a$ runs 6.4% to 26.1% low, worst on the story-writing workload where acceptance
is 37.4%. Two engines, two vendors, two architectures, thirteen data points,
thirteen exact fits for the identity.
## The KV cache does not know how many query heads you have
The handbook gives the KV-cache shape in three places, and all three use the
wrong head count.
The calculator on the KV-offloading page:
```text
KV Cache Size (GB) = 2 × B × S × L × H × D × (Q / 8) / (1024^3)
```
with `H` labelled **"Attention Heads (H)"**, described as "Number of attention
heads per layer," and an info box pointing the reader at `config.json` for "the
number of layers, hidden size, number of attention heads." The simplified mode
substitutes the model's hidden dimension for `H × D`. And the PagedAttention page
writes the allocation as `2 × num_layers × num_heads × head_dim × max_seq_len`.
Grouped-query attention appears exactly once in the 70,000 words, in a bullet
about L2 cache reuse. `num_key_value_heads` appears nowhere.
The handbook knows the right answer. It is in the prose, on a different page:
> For Llama 3 8B in FP16, for example, a single 8K-token sequence holds about 1
> GB of KV cache (learn more about the calculation). The weights take roughly 16
> GB. On an 80 GB GPU, this leaves at most around 64 GB for the KV cache and
> other runtime data, giving a theoretical upper bound of roughly 60 such
> sequences.
Check it: Llama-3-8B is 32 layers, 8 KV heads, head dimension 128, so
`2 × 32 × 8 × 128 × 2 bytes` is 131,072 bytes per token, and 8,192 tokens is
exactly 1.000 GiB. The paragraph is right, and 64 GB divided by that is 59.6
sequences, which is "roughly 60."
Now follow the link it gives you. Put the same model into the calculator, filling
`H` with the attention-head count the calculator asks for, and it returns **4.00
GB**. Fifteen sequences instead of sixty. The sentence and the tool it cites as
its own working differ by a factor of four.
The factor is the GQA group size, and it is not small on anything modern. The KV
calculator has no model presets, so here is the list the handbook's two sibling
calculators offer — its own choice of exemplar models — at fp16:
| Preset model | L | attn heads | KV heads | Calculator, B/token | Actual, B/token | Overstated |
|---|---:|---:|---:|---:|---:|---:|
| Llama 3.2 3B | 28 | 24 | 8 | 344,064 | 114,688 | **3×** |
| Llama 3.1 8B | 32 | 32 | 8 | 524,288 | 131,072 | **4×** |
| Qwen2.5 32B | 64 | 40 | 8 | 1,310,720 | 262,144 | **5×** |
| Llama 3.3 70B | 80 | 64 | 8 | 2,621,440 | 327,680 | **8×** |
| Llama 3.1 405B | 126 | 128 | 8 | 8,257,536 | 516,096 | **16×** |
| DeepSeek V3 671B | 61 | 128 | *MLA* | 3,997,696 | 70,272 | **57×** |
Every number in that table comes from the model's own `config.json`. Six presets,
six wrong answers, and the error grows with model size because the frontier has
been widening the GQA ratio for years — the thing the formula divides by is the
thing it does not have.
DeepSeek V3 is not a scaling error at all; it is the wrong shape. Multi-head
latent attention caches one compressed latent per token per layer —
`kv_lora_rank` 512 plus `qk_rope_head_dim` 64, 576 values — not `2 × H × D`.
vLLM says so in three lines (`vllm/config/model.py`):
```python
if self.use_mla:
# When using MLA during decode it becomes MQA
return 1
```
At a 128K context the calculator claims 488 GiB where the model holds 8.6.
### The stress test: a hybrid model
Grouped-query attention is one multiplier. Hybrid attention is a second, and it
multiplies with the first. Qwen3-Next-80B-A3B publishes everything you need:
48 layers, `full_attention_interval: 4`, 16 attention heads, 2 KV heads, head
dimension 256.
192 GiB against 6 GiB, and the 32× decomposes exactly: 8× because 16 query heads
share 2 cached KV heads, 4× because only 12 of the 48 layers keep a cache at all.
One of those numbers does not fit on an H200. The other fits on a laptop.
This is not an exotic case any more. Both large hybrids this site has read
recently have the same shape. The [Cinference
write-up](/articles/cinference) found 48 of Qwen3.8-27B's 64 layers holding a
fixed 144 MiB recurrent state instead of a KV cache, which is most of why 256K
context fits on a 32 GB consumer card. The [vLLM PD-serving
numbers](/articles/qwen38-pd-serving) are for a 92-layer model with 69 gated
DeltaNet layers, where I recomputed a request's KV budget at 759 MiB and found
**37.5% of it to be recurrent state that a 200-token request pays in full**.
Which makes the flat claim on the KV-offloading page —
*"the KV cache size grows linearly with sequence length"* — false for a growing
share of what people are actually deploying. On Qwen3-Next at a 2,048-token
request, 61% of the KV allocation does not grow with the sequence at all.
## The KV cache is not a percentage of the weights
The GPU-memory page carries the sizing rule most readers will take away:
```text
Memory (GB) = P * (Q / 8) * (1 + Overhead)
```
with overhead described as "Additional serving memory beyond weights, such as KV
cache, activation buffers, workspace memory," typical range 10–30%. The
calculator implements it verbatim: `params * (precision / 8) * (1 + overhead /
100)`.
The page is honest about it being rough — *"The percentage-based overhead is a
quick sizing shortcut, not an exact capacity model"* — and that is the right
disclaimer for a number that is imprecise. This one is not imprecise. It is
monotone in the wrong direction, because the overhead is **multiplicative on the
weight bytes**, and the KV cache is not a function of weight precision.
Work it on Llama-3.1-8B, whose cache is 128 KiB per token whatever you do to the
weights:
- **bf16, 20% overhead.** 8 × 2 × 1.2 = 19.2 GB required, of which 3.2 GB is the
allowance for everything that is not weights. At 128 KiB/token that is 24,414
tokens of KV.
- **4-bit, 20% overhead.** 8 × 0.5 × 1.2 = 4.8 GB required, allowance 0.8 GB.
**6,103 tokens.**
Quantizing the weights divided the cache budget by four. The cache did not move.
And on real hardware the true capacity goes the other way: on a 24 GB card at 0.9
utilization, bf16 weights leave 5.54 GB for cache — about 42,000 tokens — while
4-bit weights leave 15.86 GB, about 121,000. The formula's allowance falls 4×
where the actual headroom rises 2.9×.
The handbook states the correct version of this, once, in a sentence on the
quantization page:
> A smaller weight footprint also leaves more GPU memory available for the KV
> cache, larger batches, and more concurrent requests. **Weight quantization does
> not reduce the KV cache size per token by itself.** It requires quantizing the
> KV cache separately.
And then contradicts it, ninety lines later on the same page, in the list of
reasons to quantize:
> You want to support higher concurrency. **Quantization reduces KV cache size
> per token**, allowing more tokens (and therefore more parallel requests) to fit
> within the same GPU memory.
Same page, same document, opposite claims. The first is correct. The second is
the assumption the calculator is built on.
### Measured: what a 4-bit checkpoint actually weighs
The other half of that formula is the weight term, `P × (Q / 8)`, and it is worth
checking because it is the part everyone trusts. I measured it rather than
argued about it: for five real checkpoints I pulled every `.safetensors` header
over HTTP range requests — eight bytes of length prefix, then the JSON header —
and summed `data_offsets` for the exact byte count.
Exact at bf16, 13.5% low at FP8, **43.4% low at 4-bit on the 8B**. The remainder
is the embedding table, the LM head, the norms and one scale per quantization
group, none of which shrink with the matrices; a "4-bit" 8B checkpoint measures
5.71 bits per parameter. And the error is worst where the formula's users are:
the fixed high-precision remainder is a larger share of a small model, so the 8B
is off by more than the 70B on the same quantization scheme.
Put the two halves together and the handbook's memory calculator, asked about
a 4-bit 8B model at 20% overhead, returns 4.8 GB for a deployment whose weights
alone are 5.74 GB.
## One more place the same variable is wrong
The parallelism page, on how far you can shard:
> The number of GPUs you shard across is also constrained; it generally has to
> divide the number of attention heads evenly, so you can't scale to arbitrary
> GPU counts.
The binding constraint is the KV heads, and vLLM says so in one line with four
of comment (`vllm/config/model.py`, `72675c4`):
```python
# If tensor parallelism is used, we divide the number of KV heads by
# the tensor parallel size. We will replicate the KV heads in the
# case where the number of KV heads is smaller than the tensor
# parallel size so each GPU has at least one KV head.
return max(1, total_num_kv_heads // parallel_config.tensor_parallel_size)
```
The operational consequence is not that a setting is rejected. It is that past
`tp_size = num_kv_heads` the cache stops sharding and starts **replicating**.
Llama-3.1-8B has 8 KV heads: at TP=8 each GPU holds one, and at TP=16 each GPU
still holds one, so the cluster now stores twice as much KV cache for the same
sequence. Sixteen GPUs give you the same KV capacity per sequence per GPU as
eight. A reader following the attention-head rule would size TP=16 or TP=32 on
this model and expect the cache to keep dividing.
There is a whole separate flag for the case, which is the strongest evidence I
can offer that the KV heads are the real boundary. Decode context parallelism
shards the cache along the sequence instead, and vLLM will only let you enable
it *above* that line: *"Decode context parallelism for GQA/MQA requires
`--tensor-parallel-size` to be greater than the model's total number of KV
heads."* A knob that exists exclusively for `tp_size > num_kv_heads` is not a
knob you need if `num_attention_heads` is the constraint.
## What it gets right, which is most of it
None of the above should be read as a verdict on the document, because on the
things the field most reliably gets wrong, this handbook is better than the
field.
**The metrics chapter makes a distinction almost nobody makes.** It separates
average TPOT from average ITL and explains that they differ in how you average —
TPOT is request-weighted, ITL is token-weighted, so a long response moves one and
not the other — and then says the sentence that should be the first line of every
benchmark post: *"always check whether TPS metrics refer to input, output, or a
combined view."* That is the same question the Cinference read had to ask before
anything else: 450 tokens a second **for whom**.
**The benchmark template demands every denominator.** Model, framework version,
quantization, GPU and count, tensor and data parallelism, max batch size, request
rate, concurrency, input length, output length — all of it before a single result
row. That is the discipline that makes a throughput number mean something, and
most published benchmarks supply about a third of it. The one slip is that the
template's own Results block lists `Throughput: 98 tokens/sec` without saying
input, output or combined, one chapter after telling you always to check.
**Its one piece of bandwidth arithmetic is right, and I can check it against
something I have.** The GPU page gives the bandwidth-bound decode ceiling:
```text
maximum decode tokens/sec ≈ memory bandwidth / bytes read per token
```
and works it: 140 GB of fp16 70B weights over an H100 SXM's 3.35 TB/s is about 24
tokens/s per sequence. (3,350 ÷ 140 = 23.9.)
This site has exactly one first-hand measurement of that ceiling, and it comes
from the other end of the market: Cinference's non-speculative decode of a 27B
NVFP4 model on an RTX 5090, streaming 19.729 GiB of weights per step at 1,792
GB/s of peak bandwidth.
| | Handbook ceiling | Measured | Fraction of ceiling |
|---|---:|---:|---:|
| 8K context | 84.6 tok/s | 71.2 | **84.2%** |
| 260K context, cache included | 64.3 tok/s | 52.9 | **82.3%** |
Consumer Blackwell instead of Hopper, 4-bit weights instead of fp16, a hybrid
attention stack instead of a dense one, a C++ engine instead of vLLM — and the
rule lands within 16% at both ends of a 32× context sweep, with the residual
almost constant. A constant residual is the sign that a model is capturing the
whole structure and the gap is fixed overhead. The second row required adding the
6.69 GB KV term the handbook's worked example explicitly sets aside; put it in and
the rule keeps working.
**It publishes counter-evidence.** The 20–30% disaggregation regression. The note
that Hugging Face TGI is in maintenance mode and you should plan an exit. The
observation that Ollama does not do concurrency, which makes half the
optimizations in the handbook inapplicable to it. The repeated instruction to
benchmark your own traffic rather than trust any of the numbers on the page,
including theirs.
## What is missing, and it is one thing
The words "compute-bound" and "memory-bound" appear twelve times across eight
pages. The ratio that decides which side you are on appears nowhere.
**"Arithmetic intensity" occurs once in 70,000 words**, in a passing clause
about what tensor cores are for. There is no roofline, no FLOPs-per-byte, no
ridge point.
What that costs the reader is the ability to check the dichotomy against their
own request shape, and the dichotomy is not reliable. Prefill is not compute-bound
because it is prefill — a 128-token prefill is memory-bound, because it has not
accumulated enough work to amortize one pass over the weights. Decode is not
memory-bound because it is decode — batch it hard enough and its intensity climbs
toward the ridge, which is the actual reason continuous batching multiplies
throughput while leaving per-token latency alone. This site's
[prefill-and-decode explainer](/articles/how-llm-inference-works) plots both
phases against an A100's ridge of 153 FLOP/byte, and the [DGX Spark
piece](/articles/dgx-spark-batching) is what the batched end of that axis looks
like in practice. A reader who has the ratio can derive the handbook's own
decode-ceiling rule, and can tell when it stops applying. A reader who has the
dichotomy has a mnemonic.
## What I would take from it
- **Read the prose, rebuild the calculators.** The handbook's paragraphs get
Llama-3-8B's KV cache right and its calculator gets it wrong by 4×, on the same
model, with a link between them. That inversion — careful text, loose tool — is
worth watching for in any technical document, because the tool is the part that
gets copied.
- **Never put a logged number into a textbook formula without checking the
definition.** "Acceptance rate" names two quantities that differ by 41% on a
real sweep, and the formula that takes one of them is printed next to the
definition of the other.
- **The KV cache is decoupled from almost everything people couple it to.** Not
the query-head count, not the weight precision, and on a hybrid model not even
the sequence length for a large fraction of the bytes. Price it from
`num_key_value_heads`, the layers that actually have a cache, and the cache
dtype — nothing else.
- **Vendor authorship is the wrong thing to screen for.** This handbook is more
careful than most of the neutral writing on the same subjects, and where it is
careless the carelessness is not commercial — it is the ordinary kind, in a
calculator nobody went back and re-derived. Screen for whether the arithmetic
can be checked. Here it can, which is the best thing about it.
Related reading: [prefill, decode, and where the time
goes](/articles/how-llm-inference-works); [Cinference's 450 tokens a
second](/articles/cinference); [a Pareto frontier is not a
deployment](/articles/qwen38-pd-serving); [speculative decoding on AMD
GPUs](/articles/vllm-speculative-decoding-amd); [EAGLE-3 and the
draft-model scaling law](/articles/eagle-3-speculative-decoding);
[vLLM read from the source](/articles/vllm); [SGLang's radix
tree](/articles/sglang); [continuous batching on a small
box](/articles/dgx-spark-batching); and [VRAM is a policy, not a
property](/articles/vram-is-a-policy), which is the same argument about a
different resource.
**Measured** — computed from each model's published `config.json`
(`num_hidden_layers`, `num_attention_heads`, `num_key_value_heads`, `head_dim`)
against the calculator's `2 × B × S × L × H × D × (Q/8)` with `H` filled in as
the field the calculator's label names. If the calculator's `H` is meant as KV
heads and the label is the only problem, the arithmetic is unchanged but the
finding becomes a documentation bug rather than a numerical one — except that
the simplified mode explicitly says to substitute the hidden dimension for
`H × D`, which is unambiguously the query-side width, and the PagedAttention page
writes `num_heads` in the same shape. Three independent statements would all have
to be read charitably at once.
**Measured** — the arithmetic is mine, run on AMD/Embedded LLM's published
γ/MAL/AR sweep for Qwen3-8B + EAGLE-3 on MATH500 (**Reported**). The whole
finding rests on AR being accepted-over-proposed rather than per-position
conditional. The identity `MAL = 1 + AR·γ` holding exactly on all seven rows is
strong evidence for that reading, but it is inference from the numbers, not from
the code. Point me at the vLLM metric definition showing `num_draft_tokens`
counting only positions actually evaluated — that is, excluding proposals after
the first rejection — and AR becomes the conditional rate, the handbook's formula
is the right one to use, and the seven exact fits become a coincidence I would
need to explain.
**Measured** — safetensors headers pulled over HTTP range requests, exact
`data_offsets` sums, five checkpoints. The denominators are the bf16 parameter
counts of the same models, which is the choice that makes scales and packing
metadata count against the budget. If you instead divide by the number of
*quantized* values only, the FP8 and W4A16 numbers drop toward their nominal
widths and the finding shrinks to "quantized checkpoints keep some tensors in
high precision," which everyone knows. I chose the denominator that answers the
question the calculator asks — how much VRAM does this file need — and a
different question deserves a different denominator.
**Reasoned**, from evidence that is strong but circumstantial: every merged PR
came from a `bentoml/…` branch until 2026-07-29 and from `modular/…` from
2026-08-03, the single most prolific committer (nine times the next) authored
commits on both sides of that line, and the two benchmarks the
speculative-decoding page leans on are BentoML's. I did not look for a corporate
announcement and I make no claim about what kind of transfer it was. What I am
asserting causally — that the even-handedness is inherited — is weaker still: a
handbook written inside Modular from scratch might have come out the same way.
The git history makes that explanation unnecessary, not impossible.
**Measured** on my side (the ceiling arithmetic), **Reported** on the
measurement side (Cinference's own MTP0 results file). The weak link is the
19.729 GiB of per-step weight traffic, which I took from that repository's
published figure rather than measuring — and it is smaller than the 21.49 GB
checkpoint, so something is not being streamed every step. If the real per-step
traffic is the full checkpoint, the ceiling drops to 83.4 tok/s and the fractions
move a couple of points; if it is materially lower than 19.729 GiB, the measured
rate exceeds the ceiling and the rule is broken rather than confirmed. Profile
one decode step's HBM read volume and this becomes a real test instead of a
consistency check.
The counts are **Measured** — word-boundary greps over `docs/` at `0d64912`. The
judgement is **Reasoned** and it is the arguable half: a handbook aimed at people
choosing a framework may not owe anyone a roofline, and the dichotomy is a fine
first approximation for the request shapes most readers have. I would also have
liked to blame the percentage-overhead model on the missing ridge point, and I
cannot — it is entirely possible to reason correctly about arithmetic intensity
and still ship a calculator that multiplies the cache allowance by the weight
precision, in a different chapter written months apart.
---
Read at `modular/llm-inference-handbook` commit `0d64912` (2026-09-21,
Apache-2.0), 57 pages and 70,493 words of `docs/`, plus the four calculator
components under `src/components/Calculator/`. Model geometry from each model's
own `config.json` on Hugging Face. Weight footprints measured from safetensors
headers over HTTP range requests. vLLM source at `72675c4`. The
speculative-decoding sweep is AMD and Embedded LLM's, as
[read here earlier](/articles/vllm-speculative-decoding-amd); the decode-ceiling
measurement is Cinference's, as [read here
earlier](/articles/cinference). I ran no model and own none of the hardware in
this article. Both figures are the handbook's own diagrams, rendered from the
committed SVG and flattened onto white.