Qwen3.8-Flash-Next: four changes and an honest report

Most model releases give you a benchmark table and an architecture diagram, and the diagram is a picture of a decision whose justification you have to take on faith. Qwen3.8-Flash-Next comes with a 28-page technical report whose ablation tables are properly controlled, and which repeatedly reports that the thing they optimised and the thing they wanted did not move together.

That is the interesting part. The architecture is four changes — attention, residual, embedding, optimiser — and each one comes with the experiment that chose it, including the ones that came out ambiguous.

expand

1 — Attention: remember cheaply, retrieve precisely

Three of every four layers are Gated DeltaNet, which compresses the prefix into a fixed-size recurrent state rather than a growing cache. The fourth is global attention, because a finite state cannot reproduce exact token-level retrieval no matter how well it is gated.

The delta rule is what makes GDN more than decayed averaging. At each step the layer estimates what value is already associated with the incoming key and writes back only the residual error, scaled by a write gate, after decaying the whole state. Repeated or similar keys therefore update an existing association instead of piling up outer products — an erase-and-write, not an append.

Both comparisons in that control are worth having. Against a full-attention Transformer the hybrid wins eight of nine, which mostly tells you the hybrid is not a compromise. Against a sliding-window hybrid — the cheap way to get the same asymptotics — it wins seven of nine, and the split is diagnostic: MATH +8.50, MultiPL-E +5.55, MMMLU +3.50, while EvalPlus goes the other way by 2.41.

A window forgets by position. A gated delta rule forgets by relevance. The benchmarks where that distinction pays are exactly the ones needing something from far back that a fixed 128-token window has already dropped.

Two implementation notes from the report that are easy to skip and shouldn't be. GDN here uses a bounded sigmoid output gate rather than the SiLU of the original formulation, which they say improved things consistently — the same preference for bounded gates recurs in the residual work below. And on positional encoding: RoPE and NoPE looked equivalent during pre-training, but the NoPE variant showed "a substantially higher rate of endless generation after post-training." A choice that looks free on the pre-training curve and costs you termination behaviour later is the kind of finding that only comes from having shipped.

QSA: the selector is the cost

expand

The global-attention layers use Qwen Sparse Attention. The observation behind it: sparse attention fixes the number of positions read, so the attention itself goes flat in context — but the indexer that chooses those positions still scores the query against everything, and at a million tokens the choosing costs more than the reading.

QSA aggregates the sequence into micro-blocks first, estimates importance per block, then selects regions. indexer_compress_ratio: 4 in the config, with a budget of 2,048. Both the scan and the indexer's own cache shrink fourfold.

One design decision deserves the emphasis the report gives it. Some approaches share indices across layers to amortise the selector. Qwen deliberately do not, and the reason is specific to this architecture: in a hybrid where GDN and attention layers alternate, there is much less cross-layer attention similarity to exploit, so a shared index would be built on an assumption the stack violates.

The measured quality is more interesting than the speedup. On 8-needle MRCR, QSA is behind full attention at 128K and 256K — by 1.16 and 1.20 — and then ahead by 9.87 at 512K and 5.73 at 1M. A compressed index does lose information, and at moderate lengths that shows up as a small loss. What it buys is that the surviving signal keeps working at lengths where full attention's own scores go soft.

That crossover is worth stating plainly because the announcement doesn't: if your workload lives at 128K, QSA is a small accuracy cost for a large speed gain, not a free win. Past 256K it is both.

expand

2 — Residual: four lanes instead of one

In a standard Transformer every layer reads from and writes to the same residual stream, so early features get progressively diluted by everything written after them. Gated Residual widens that stream into four parallel branches and gates the reads and writes.

The ladder in that control is the report at its best, and its own summary of it is the line I would keep from the whole document: this is one of several places in this report where loss and downstream accuracy do not move together.

By training loss, widening the stream is essentially the entire result — 0.021 of 0.027 — and making the gates data-dependent adds 0.002, which reads as noise. By benchmark average the ordering inverts: the data-dependent step is worth 1.98 points against 1.58 for the widening. A team watching the loss curve during the run would have concluded the second change did nothing.

The empirical note attached to it is lovely: one of the four branches reliably becomes a long-range pathway connecting the first attention layer to most of the middle and later layers. Nobody designed that; the gates found it.

And then the disagreement. GR is hyper-connections with the branch-mixing operator deleted — the ablation says removing it costs nothing while removing memory traffic and a source of instability. That operator is precisely what GLM-5.3-Flash keeps, constrained to a doubly-stochastic manifold with twenty Sinkhorn iterations, in a model that shipped the same week. Same idea, four branches each, opposite conclusions about the one operator, and no head-to-head anywhere. Worth knowing that the question is open rather than settled.

Two practical consequences of the gating that the report flags: it suppresses activation outliers, and the residual state can be held in FP8, which matters when you are carrying four of them.

3 — Embedding: 51B of parameters that cost almost no compute

N-gram embedding looks up a table using the current token and the two before it — ngram_size: 3 — rather than the current token alone. The appeal is arithmetic: lookups are deterministic and known in advance, so the table can live in host memory and be prefetched in parallel with computation, adding enormous capacity for almost no per-token FLOPs. Qwen add 51B of it and place it at layer 2, so the prefetch overlaps layer 1.

Now the honest part, which the report does not bury. Scaled with additional parameters, loss falls monotonically from 1.585 to 1.526 across the whole range from none to 200×. Downstream does not follow: MATH peaks at 20× and gives back two of its five points; GSM8K the same; BBH peaks at 50×; MMLU-Pro and MMMLU at 100×. The only group that tracks the loss curve to the end is Chinese — C-Eval 66.91 → 74.94, CMMLU 68.10 → 73.24 — which is a coherent result rather than a fluke, since a trigram table is a memory for frequent local character patterns and should help most where the tokenizer is under most pressure.

There is a second study, under a fixed parameter budget with experts traded away to pay for the embeddings, and it is even more candid: loss is best at 10×, that optimum "is not evident in other evaluations", out-of-domain perplexity barely moves, and downstream shows "no clear improvement over the MoE-only baseline." Their conclusion is that N-gram embeddings and MoE experts play distinct roles — which is a fair reading, and also an admission that swapping one for the other did not pay. The paragraph after it lists parameter-efficiency tricks they tried (token normalisation, non-uniform allocation across N-gram orders, frequency-based partitioning) and reports no consistent gains from any of them.

So: 51B is defensible on loss and on Chinese. That it beats spending the same parameters on more experts is, by their own evidence, not demonstrated.

4 — Optimisation: Muon, split carefully

The model is trained with Muon, with three refinements the report is specific about. Muon is applied to parameters that genuinely act as two-dimensional linear maps — attention, GDN and expert weights — while embeddings, the MoE router and GR's low-rank parameters stay on AdamW. Fused matrices are split back into their independent linear transformations before orthogonalisation: QKV, SwiGLU and the GDN projections are stored fused for speed, and orthogonalising the fused block would be orthogonalising a matrix that does not correspond to any single map.

With the scaling law refitted for the new architecture and optimiser, the model tolerates larger learning rates and batch sizes. The finding I liked most is a negative one: batch-size warmup turned out to be unnecessary, and ramping up from a small batch cost 18.8% more optimizer steps for no better final result. That is a widely-followed practice reported as pure overhead, with a number attached.

Where it lands

The headline claim is the cost ratio: Qwen3.8-Flash-Next trains for about one ninth of what Qwen3.7-Plus did — a 397B model with 17B active — and beats it nearly everywhere. On the published language table it leads DeepSWE 1.1 (58.7), SWE-bench Pro (62.5), SWE-bench Multilingual (81.0), CoWorkBench (73.9), JobBench (55.7 against Qwen3.7-Plus's 27.6), Toolathlon (73.5), IFBench (81.3), GPQA Diamond (91.7) and LiveCodeBench v6 (91.9).

Two rows go the other way and both are worth noting. NL2Repo-Bench is 48.1 against DeepSeek-V4-Flash's 54.2 — repo-scale generation again, the same benchmark GLM-5.3-Flash loses badly. And HLE is 35.9 against Claude Opus 4.6's 40.0, the one place the frontier model is clearly still ahead.

On vision the story is cleaner: it leads its own 27B sibling and Qwen3.7-Plus on every listed row, and beats Opus 4.6 comfortably on AndroidWorld (84.5 vs 62.0), ERQA (72.3 vs 40.8), LVBench (76.6 vs 63.0) and RealWorldQA (88.5 vs 73.9). OSWorld 2.0 binary at 19.4 is a reminder of where computer-use actually is: partial credit 52.3, full task completion under one in five.

The base-model table is the quiet confirmation. At 6B active against Qwen3.7-Plus's 17B, it wins MMLU-Pro (73.23 vs 70.90), SuperGPQA (51.36 vs 48.42) and BBH (90.87 vs 89.41), and loses MMLU, MMLU-Redux, GPQA and MATH by small margins. Competitive at roughly a third of the activated compute is the architecture's actual claim, and the base numbers support it better than the instruct ones do.

Running it locally

This is where the architecture produces a genuinely unusual deployment story. Because so much of the model is either sparsely-activated experts or a deterministically-addressed lookup table, it runs on system RAM far better than a dense model would — Unsloth's guidance is that CPU-with-RAM versus GPU-with-VRAM "may make relatively little difference", which makes large-unified-memory machines (Macs, DGX Spark) unusually good targets.

The GGUF quants span 75 GB to 355 GB, and the small end has a wrinkle worth understanding: the 1-bit build is 75 GB, which is large for 1-bit, because the N-gram and per-layer embedding tables are not quantized below 4-bit. Their access pattern is random, and quantizing them hard damages the model disproportionately. The result is a quant that is less aggressive than its name suggests — 79% smaller than BF16 while retaining a reported 80% top-1 accuracy. You can also push the N-gram table to SSD and mmap it, which is exactly the property the architecture was designed around.

unsloth run --model unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

Sampling differs by mode and it matters: thinking mode wants temperature=1.0, top_p=0.95, top_k=20, presence_penalty=0.0; instruct mode wants temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5. Reasoning effort is set through the chat template — --chat-template-kwargs '{"reasoning_effort":"medium"}'. At time of writing it needs a specific llama.cpp PR rather than mainline.

Day zero in SGLang, where the claims get tested

The architecture section above is a set of promises about cost. LMSYS shipped day-0 support in SGLang and, in doing so, published the numbers that say whether those promises hold when someone else has to implement them. It is the best companion piece to the technical report, because a serving team has no incentive to flatter an architecture they now have to make fast.

The N-gram claim survives contact, decisively. The whole argument for a 51B lookup table was that its access pattern is sparse and deterministic, so it need not live on the accelerator. SGLang tested exactly that: each token touches only 16 rows, so they keep each rank's vocabulary-parallel shard in pinned host memory and gather the selected rows into a small BF16 GPU buffer with a Triton UVA kernel, on a dedicated CUDA stream that overlaps the first decoder block. On H200 at TP4:

Twenty-three gigabytes off the GPU, seventy-eight percent more KV budget, and throughput unchanged to within a rounding error. They also verified that four fixed prompts produced exactly matching output IDs, and that the chosen-token logprob trace matched exactly — so this is a pure storage relocation, not an approximation. That is the single most convincing piece of evidence in either document that the N-gram design is sound, and it comes from the people who had to make it work rather than the people who proposed it.

The Gated Residual costs real kernel engineering. Four residual streams mean every block does a Mix to read and a Combine to write, and those are new operators nobody had tuned. Built with NVIDIA and shipped through FlashInfer, the fused single-GEMM path takes Mix from 12.36 µs to 6.03 µs at M = 4 on B300 — a 2.05× kernel speedup worth +7.6% end to end — and Combine from 4.17 µs to 2.13 µs, 1.96×, worth +5.49%. At large M the fused Combine is up to 2.54× faster than the cuBLAS baseline at 6,144 GB/s effective bandwidth.

Read that as a cost, not just a win. A widened residual stream is nearly free in FLOPs and distinctly not free in memory traffic and kernel work, and the model only performs as advertised once someone has written the fused path. The report says the branch-mixing operator was dropped partly to reduce memory access; these numbers are what that sentence looks like in practice.

QSA's indexer gets special handling in speculative decoding. SGLang found that the draft model was re-running the indexer on every MTP step, and eliminated it: each iteration opens with a draft-extend over the tokens the target just accepted, that pass runs the indexer anyway, and its last accepted row is captured and reused across the whole draft loop, with N + 1 extra columns filled in at lookup so the draft still sees its own in-flight tokens. Because the query has moved by at most N positions out of L, the reused ranking is essentially what a recomputation would have produced — and they report accept length unchanged. Draft indexer work per iteration drops from N invocations to one.

The headline serving number: at TP4 on B200, the NVFP4 checkpoint decodes at 540 tok/s at batch size 1 with MTP, at an accept length of 3.3 including the bonus token.

The family around it

Flash-Next is a preview of the Qwen4 architecture, not the whole 3.8 line, and the dense siblings are where most derivative work is landing. One worth noting for how carefully it is documented: Jiunsong/SuperQwen3.8-27b-abliterated, built on Qwen3.8-27B — the dense model, not this one — with a rank-4 refusal-subspace edit applied to 100 tensors (output projections in layers 15–63, plus embeddings and lm_head).

Abliteration releases are usually a set of adjectives and a download link. This one ships receipts: the parent revision pinned to a commit hash, refusal measured at 30/32 → 0/32 with zero empty outputs, a paired capability floor of 7/8, tool use and vision both passing, and — the part that matters technically — 333 vision tensors and 15 MTP tensors verified byte-exact, with 100 declared tensors changed and zero unexpected ones. It also fixes something upstream: the chat template defaulted unspecified reasoning effort to xhigh, and this release defaults to medium and adds a stop condition, tested across 36 effort/task combinations.

Two honest notes. It is a 52 GB full-BF16 checkpoint measured at 4.34 tok/s decode on a single DGX Spark — a useful reminder that dense 27B on that class of hardware is a very different proposition from the RAM-friendly sparse model this article is about. And a verified 262,043-token retrieval is an acceptance test, which the card says plainly: "not a claim of perfect recall on every task."

A second derivative worth checking rather than just repeating: orcarouter/Qwen3.8-27B-Uncensored-NVFP4, also abliterated, also built on Qwen3.8-27B dense, announced with "uncensored Qwen3.8-27B can now fit on a 12GB RTX 5070." It's a real two-tier quant — compressed-tensors' NVFP4 format for the bulk of the model, FP8 for a smaller sensitive subset, and the linear-attention projections, vision tower, and lm_head left in BF16 — and naming a specific RTX 50-series card rather than "any NVIDIA GPU" makes sense, since NVFP4 only gets hardware-accelerated dequantization on Blackwell silicon. The "12GB" part doesn't hold up against the repo's own file manifest, though: six safetensors shards, usedStorage 24.7 GB in the Hub API — essentially double a 12 GB card's VRAM before a single token of KV cache is allocated. The README that presumably explains the gap was gated behind a Hub login at the time of writing, so this is checked against the repo's published file sizes and safetensors metadata directly, not against whatever the write-up itself says. (Searching Hugging Face for "Qwen3.8-27B" now returns well over a hundred derivative repos — abliterated, re-quantized, re-abliterated-then-requantized — which is its own small data point about how fast a dense, well-documented open model turns into an ecosystem.)

Flash-Next itself — the sparse model, not a dense derivative — got dedicated day-zero support in ik_llama.cpp (PR 2365), a llama.cpp fork whose own README leads with exactly the four pieces this article is about: MLA, Gated Delta Net (its name for GDN linear attention), MTP, and DFlash. That's a plausible reason a single RTX 3090 or a CPU-only box can run a 125B-total model at all: the fork's --cpu-moe/--n-cpu-moe flags offload routed experts to system RAM while keeping attention, the N-gram embedding, and the MTP head resident on the GPU — the standard llama.cpp-family trick for sparse MoEs, and a good match for a model whose own author split it exactly that way (51B of embedding, cheap to keep local; 125B of experts, expensive to keep loaded). The PR itself wasn't reachable through this session's GitHub access to pull a specific tok/s number, so take "runs on a 3090" as plausible and architecturally consistent with the offload mechanism, not independently benchmarked here — though a second fork, covered next, supplies a concrete flag set and a concrete number on exactly that class of card, which resolves part of that hedge without closing it.

A more surprising angle on the same problem, and one that was directly checkable: oMLX, a Mac-native serving app, shipped "Soft-REAP" expert streaming in PR #3260 — fetched via its refs/pull/3260/head git ref directly, since the PR page itself sits behind this session's GitHub access restrictions — extending the hot/cold tiering the app already does for KV cache to the MoE experts themselves. Its own docs/soft_reap.md, added in that PR, benchmarks the identical checkpoint this article covers (a 98.995 GiB Qwen3.8-Flash-Next build) in "analytical cache only" mode: no REAP manifest, no expert permanently pruned or pinned — every one of the 512 routed experts per layer stays reachable, evicted and reloaded from SSD purely by how often the router actually selects it. Measured active memory at one hot-cache setting: 39.684 GiB, close enough to a claim of "running on just 37GB of memory" that it reads like the same measurement at a slightly smaller cache size — and "FULL experts (no prune)" matches this mode's design exactly, in contrast to the REAP-pinned sibling mode the same doc measures at 72.6–75.3 GiB. What the doc doesn't hand over is a sustained decode tok/s to check against a stated "40 tok/s": its own numbers are almost entirely load-time and I/O-engineering microbenchmarks (a two-token warm-SSD smoke test at 14.7s, isolated 512×128/1024×256 synthetic fixtures) rather than an end-to-end chat-throughput figure, and the "60% of experts on disk" ratio depends on a hot-cache setting the doc treats as user-configurable rather than fixed — so the memory claim is well corroborated by real, dated engineering work; the specific throughput number isn't confirmable from the same source.

A third fork closes more of the gap than either of the two above, because it was built for this model specifically rather than adapted to it: cafe-llama.cpp, and its source backs up most of what its README claims. -cmoe/-ncmoe — the same mechanism the ik_llama.cpp paragraph above describes — route expert tensors through ggml_backend_cpu_buffer_type(), ordinary pageable memory. This fork adds -hmoe/-nhmoe, which route the identical ffn_*_exps tensors through a new helper, common_host_buffer_type(), that asks each backend device for its pinned-memory allocator (ggml_backend_dev_host_buffer_type) and only falls back to plain CPU memory if none exists. That is a real, checkable difference, not a rename: pinned host memory lets cudaMemcpyAsync run as a genuine DMA transfer instead of first being staged through the driver's own internal bounce buffer. It's paired with real supporting code — a roughly 130-line device-side LRU cache (ggml_cuda_expert_lru_cache) that keeps recently-used experts resident on the GPU under an "elastic" budget polled from cudaMemGetInfo, plus a scheduler change that starts the next split's host-to-device copy while the current one is still computing. Where the README overstates it is the phrase "zero-copy async DMA": the code that actually moves the bytes is a plain cudaMemcpyAsync into a cached device buffer — a real, worthwhile copy elision relative to CPU-RAM offload, but not the CUDA-technical meaning of zero-copy, which would mean no copy at all.

The command in the report spends more than a flag on the harder half of the problem, though. -ctk q8_0 -ctv q8_0 -kvu -fa on -ngl 99 -nhmoe 36 --no-ngram -np 1 -b 1024 -ub 128 includes --no-ngram, and tracing it through the loader removes any ambiguity about what that does. qwen4exp.cpp's hparams loader only populates the PLE fields — n-gram size, head count, per-head vocabulary ranges — inside if (n_ple > 0 && ml.load_ngram); --no-ngram sets load_ngram false, so hparams.ple_n_heads stays at its initialised zero and the lookup table this article spends a whole section on is never configured. Tensor creation follows the same branch: instead of the normal create_tensor(..., TENSOR_READ_LAZY), the disabled path creates per_layer_token_embd with TENSOR_SKIP — and TENSOR_SKIP is not a placement hint, it's the model loader's early-return branch that logs the tensor as unused, subtracts its byte count from the load, and hands back a null pointer. Zero bytes, in RAM or VRAM, exactly as the README says. That is the ~51B-parameter table the rest of this article calls defensible on loss and on Chinese benchmarks — switched off entirely for this measurement.

Nothing in the fork measures what that costs. The README introduces the flag with a single line — "Disable Ngram if you don't have enough RAM/VRAM" — and there is no perplexity run, benchmark table, or even a code comment weighing the trade-off anywhere in the diff. The technical report this article is otherwise built on ran the adjacent experiment at a much smaller scale — trading N-gram parameters for MoE experts under a fixed budget — and found "no clear improvement over the MoE-only baseline"; it never tried removing PLE outright at the shipped 51B scale on the shipped model. So the reported tok/s figure is a real measurement of a real, working configuration, and it is not a measurement of the model the rest of this article describes: it's the same weights with roughly a fifth of the total parameter count not participating in the forward pass. Whether that costs the Chinese-benchmark gains the report attributes specifically to this table, or nothing anyone would notice in English, isn't addressed anywhere in the repository — and the single commit sitting at HEAD while this was checked is "Fix typo in README regarding Ngram RAM/VRAM," which is to say the memory framing for this exact flag was still being corrected the day before.

The rest of the command is ordinary llama.cpp, inherited rather than invented here: -ctk q8_0 -ctv q8_0 quantizes the KV cache to 8 bits per element on both sides, -kvu shares one unified cache across parallel slots instead of pre-allocating per slot, and -b 1024 -ub 128 runs a large logical batch against a small physical micro-batch, favouring steady decode over raw prefill speed. All memory-for-something trades, none of them new here. What is specific to this model is why disabling PLE needs its own flag family separate from offloading experts, and this article's own numbers already supply the arithmetic: at the 4-bit floor the "Running it locally" section above documents for the N-gram and per-layer embedding tables, 51B parameters is at least ≈25.5 GB for that one tensor alone — before a single expert or a byte of KV cache is loaded — and -hmoe/-cmoe's override regex matches only ffn_*_exps tensors, never per_layer_token_embd. Offloading experts with -nhmoe does nothing at all for that table; by the loader's ordinary per-layer offload logic it sits on the GPU regardless, unless --ngram-ssd (mmap it from disk on demand) or --no-ngram (drop it) moves or removes it. On a single 24 GB 3090 with 64 GB of system RAM, that table is plausibly the difference between a configuration that fits and one that doesn't — independent of whatever it costs to remove.

The fork also ships its own answer to the MTP question covered earlier: quimmedes/Qwen3.8-Flash-Next-MTP-GGUF, four standalone draft checkpoints verified against the Hub's own blob metadata — Q4_K_M at 2.59 GiB, Q6_K at 3.17 GiB, Q8_0 at 3.85 GiB, BF16 at 7.24 GiB, each a touch smaller than the README's own rounder figures — loaded with --spec-type draft-mtp --spec-draft-n-max 2. It's the same NextN/MTP mechanism this article and the SGLang section above both cover in depth; the fork's own contribution is a converter that exports the draft head as an independently loadable model rather than something bundled inside the main checkpoint.

Sizing the fork honestly: diffed against its own merge-base with ggml-org/llama.cpp master, it touches 44 files, +1,179/−170 lines — a real but narrow piece of work concentrated almost entirely where you'd expect, 384 added lines in the architecture file (src/models/qwen4exp.cpp) and 127 in the conversion script. The commit messages are not a reliable guide to that size, and one is worth flagging specifically. A commit titled "cross-backend double-buffered DMA streaming and semantic anchor state caching" changes 24 lines across two files: four more chat-template delimiter registrations, and one more condition under which an existing upstream scheduler flag gets set to true. Neither "double-buffered prefill" nor "semantic anchors" are generic terms here — they're the specific mechanisms of FreeToken, an unrelated published serving engine this site covered separately, and the diff underneath this commit does neither of the things those names describe. The next commit, titled only "FreeToken pinned host MoE offload and elastic LRU cache," undersells itself by comparison — that one really does add a working device-side cache with an eviction policy, the code the paragraphs above lean on. Both are true of the same day's work on a young, single-maintainer fork: real engineering, and borrowed vocabulary from somewhere else on this site.

Which leaves the number as reported: "200% speed boost... now 28-24 tok/sec" names no baseline — not the prior flag set, not the prior quant, not whether --no-ngram was in the "before" configuration or only the "after." Read at face value it's a genuine measurement — 24 to 28 tok/s decode on a 3090 plus 64 GB of DDR4, an IQ3_XXS-class quant, a fifth of the model's parameters switched off, most of the rest sitting in pinned host memory across PCIe — and between this fork and ik_llama.cpp above, a single-3090 configuration for this specific architecture is now considerably more credible than either report alone would make it. It's still one user's machine, once, against a number nobody wrote down.

The cafe-llama.cpp paragraphs above end on a single unanswered number: what disabling that ~51B-parameter table costs, nobody has measured. Three more reports, checked directly rather than taken at their word, don't answer that question either — but they show how differently the same tensor gets treated once other builders reach for it, and the first of the three takes the opposite side of --no-ngram outright.

vcruz305/Qwen3.8-Flash-Next-EXL3-DGX-Spark-recipe (cloned with GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1) serves turboderp's 3.05bpw_h5_ng5 EXL3 pack on a single NVIDIA DGX Spark — one box, 128 GB unified memory — through the author's own vllm-exl3 plugin for vLLM:

hf download turboderp/Qwen3.8-Flash-Next-exl3 --revision 3.05bpw_h5_ng5 \
  --local-dir ~/models/Qwen3.8-Flash-Next-EXL3
 
# no draft
MODEL_DIR=~/models/Qwen3.8-Flash-Next-EXL3 bash scripts/serve_one_spark_qwen.sh
 
# MTP -- the README labels this block "k=2" while the flag it shows is 1,
# with a parenthetical saying to use that same value "for k=1"; read
# literally the two don't agree, and no k=2 invocation actually appears
# in the "Serve" section
MODEL_DIR=~/models/Qwen3.8-Flash-Next-EXL3 \
  SPEC_CONFIG='{"method":"mtp","num_speculative_tokens":1}' \
  bash scripts/serve_one_spark_qwen.sh

That labeling slip is worth naming rather than silently correcting, in the same spirit as the cafe-llama.cpp HEAD commit above: scripts/serve_one_spark_qwen.sh just forwards SPEC_CONFIG to vllm serve --speculative-config unmodified (line 56), so whatever value actually produced the k=2 row in the table below did not come from the command the README shows for it. bench_v1.py's decode metric is at least unambiguous — decode_s = wall - ttft, tok/s computed only over that remainder — which is the same TTFT-excluded convention the "Headline" table below uses.

The memory section states the opposite of --no-ngram as a design choice. Model resident on device: 78.57 GiB, of which the n-gram table alone is 30.4 GiB — checked directly against the pack's own ngram_embedding.safetensors, 32,640,183,408 bytes on the Hub, 320,001,536 rows by the recipe's own count. Not offloaded, not dropped, not mmap'd on demand: quantized and held on the GPU for the life of the server. The pack's own quantization_config.json is explicit that most of the backbone runs through a generic per-layer scheme with a literal bits_per_weight field:

{
  "quant_method": "exl3",
  "bits": 3.05,
  "head_bits": 5,
  "vision_bits": 5,
  "mtp_bits": 3,
  "codebook": "mul1"
}

— 5 bits for the attention and GDN projections, 3 for the 512 MoE experts, blending to the revision name's own 3.05bpw backbone average. The n-gram table's 128 shards carry no such field at all: each is a raw [2500012, 51] int16 trellis array, a dedicated embedding-specific packing rather than the generic linear-layer path. Do the arithmetic anyway — 51 int16 words per 160-wide row is 5.1 bits per value — and it lines up with what the revision name's own ng5 implies, even without a metadata field to check it against directly. Where cafe-llama.cpp fits a 24 GB 3090 by making the table disappear, this recipe fits a 128 GB Spark by keeping the whole table, quantized rather than absent. Same tensor, opposite bet, and — unlike the fork above — a controlled sweep with a stated sample size sits behind the number, not one unlabeled run:

Reading the sweep against its own methodology matters as much as the numbers do. Four runs each for no draft, k=1 and k=2; seven for k=3 — the README lists every one, and the imbalance is never explained, which is worth naming before treating the four means as comparable. It also isn't, per the recipe's own torch-profiler breakdown, the number that would have changed the conclusion: decode here is bound by trellis dequantization rather than memory bandwidth (the EXL3 kernels run roughly 3–5× the time the weight bytes alone would need at the GB10's 273 GB/s), which is exactly why MTP helps — fewer dequant passes per emitted token — and exactly why k=3 stops helping: vLLM's Qwen MTP implementation replays the single draft layer and the 5-bit lm_head once per extra draft token, so a third guess buys more accepted tokens per step at a dequant cost that eats the gain. k=2 remains, by the recipe's own recommendation, the setting to use — not because the sample sizes were equal, but because the mechanism explains why it wins independent of them.

The separate 122,902-token prompt probe, covered in the same chart, is the more interesting result precisely because it complicates the story instead of confirming it:

MTP k=2 wins decode at long context by a wide margin and loses time-to-first-token by a small one. Framed the way the README frames it, that gap makes sense rather than reading as a contradiction: prefill doesn't route through the draft model at all, so TTFT was never where MTP was supposed to help, and the extra scheduling overhead of standing up the draft loop shows up exactly where the mechanism has nothing to offer. It's the kind of finding this article's own SGLang section would recognise — a real cost sitting next to a real win, in the same table, neither one hidden.

The author is careful about what the probe does and doesn't establish, and that scoping is worth relaying rather than compressing away. 262,144 was the configured context ceiling; the probe ran a real ~123K-token input, not a filled window. Greedy output under MTP matched the no-draft baseline exactly on only one of four fixed prompts at k=1 and k=2 alike (two of four at k=3), diverging "with coherent text" on the others — expected of greedy-consistent speculation, the README says, since the target either accepts or rejects each draft token and never emits one it wouldn't have chosen anyway, not evidence of end-to-end bit-exact equivalence. And the whole set of numbers is labelled "preliminary," with the further caveat that this is a performance measurement, not a claim about output quality. The trellis format, the codebook, and the pack itself are turboderp's; this recipe's own contribution is the serving integration on top of it — three vLLM patches that add quant-config plumbing for Qwen4ExpForConditionalGeneration, plus the pack-preparation scripts that regenerate a safetensors index the native pack doesn't ship with — and the testing that produced the numbers above.

A second recipe for the same hardware class takes a third position on the same table, and its configuration is worth reading past the headline. bilikaz/qwen38-flash-next-recipe serves a different checkpoint — myllmbox/Qwen3.8-Flash-Next-hibrid47 — on the same class of box:

git clone https://github.com/bilikaz/qwen38-flash-next-recipe.git
cd qwen38-flash-next-recipe
./run.sh        # downloads ~99G from HF on first run, serves OpenAI API on :8000

# recipe.yaml, the only file this kit reads
speculative-config: '{"method":"mtp","num_speculative_tokens":3}'   # K=3: acceptance ~3.5 code, ~2.9 reasoning
kv-cache-memory: "7000000000"     # 7G fp8 = 391,943 KV tokens (measured)
kv-cache-dtype: fp8               # drop the line for bf16 (217,808 tokens on the same pin)
max-model-len: 262144
max-num-seqs: 8

Its v2 README claims sustained throughput over peak as the thing that changed, and backs the claim with a stated measurement method — ten-second engine windows, "sustained" as the run average and "peak" as the best window, "all streams decoding, zero prefill in the window" — rather than only asserting it:

"The peaks moved little," the README summarises; "the floors moved — the average became the floor." Two things are worth flagging without either confirming or dismissing them: this kit runs MTP at k=3 by default, the exact depth the EXL3 sweep above found didn't pay for itself — different checkpoint, different quantization, different hardware config, so it isn't a refutation, but it is a second, independent report landing on the opposite side of the same knob; and both "sustained" and "peak" here are one user's own box, dated and methodologically described, not a benchmark run against anyone else's baseline.

What v2 does to the n-gram table itself is a fourth distinct answer, and it's independently checkable rather than only claimed. The Hub lists eight ple-nvfp4-*.safetensors shards for myllmbox/Qwen3.8-Flash-Next-hibrid47 totalling 28,800,141,252 bytes — 26.82 GiB, close to the README's own rounder "26.9 GiB" — the table re-quantized to NVFP4 rather than left at whatever precision the base checkpoint shipped. It isn't resident the way the EXL3 recipe's table is, either:

env:
  MBX_PLE_MMAP: "1"
  MBX_PLE_MMAP_MODE: "auto"       # direct NVMe reads at boot, memory-mapped gather from then on
  MBX_PLE_MMAP_PREWARM: "auto"    # populate the whole table right after boot

./ple.sh status      # rows resident / free / swap, right now
./ple.sh populate     # pull the whole 26.9 GiB table into memory now (repeatable)

The eight shard files are mapped and the GPU gathers rows straight out of the mapping over unified memory, MBX_PLE_MMAP_MODE=auto reads directly from NVMe during boot (so autotune gets the transient room it needs) and switches to the mapped path afterward, and a populate pass pulls the whole table into memory once the server is warm. Demand-paged and re-quantized, neither dropped nor eagerly resident from boot — a fourth point on the same axis this section has been plotting, and one that, unlike v1's int3 table sitting in a CPU worker, needs no separate process to serve it.

The most surprising entry on that axis doesn't come from a community recipe at all. NVIDIA's own nvidia/Qwen3.8-Flash-Next-NVFP4 — a different release from the community NVFP4 quant covered earlier in this section, which abliterated a dense 27B model that has no PLE table to speak of, and whose "12GB" claim didn't survive its own file manifest. This one is NVIDIA's official quant of the sparse model this whole article is about — 18,068 downloads and 144 likes at the time of writing. The Hub API's dtype breakdown answers the PLE question directly, without needing a README at all:

Of the repo's 132,724,334,216-byte (123.6 GiB) usedStorage, the n-gram table isn't touched by the NVFP4 job in the slightest. hf_quant_config.json lists 292 exclude_modules patterns:

{
  "quantization": {
    "quant_algo": "MIXED_PRECISION",
    "group_size": 16,
    "exclude_modules": [
      "lm_head",
      "model.language_model.embed_tokens",
      "model.language_model.hyper_connection_mixer*",
      "model.language_model.layers.0.linear_attn*",
      "model.language_model.layers.0.mlp.gate",
      "model.language_model.layers.0.mlp.shared_expert*",
      "model.language_model.layers.0.self_attn*"
      // ...292 patterns total, one set per layer -- none named "ple" or "ngram"
    ]
  }
}

— every linear_attn*, every hyper_connection_mixer*, the MoE router and shared-expert gates, lm_head, embed_tokens, the vision tower, layer by layer — and not one of them names a PLE or n-gram tensor, because the table was never part of that mixed-precision job to begin with. It ships in a separate file, model-fp8-mtp-ple.safetensors, entirely in FP8 (F8_E4M3): reading the safetensors header directly over an HTTP range request (rather than trusting the file listing) turns up 128 shards of shape [2,500,012, 160] plus a single BF16 scale tensor — 320,001,536 rows, matching the EXL3 pack's own count above independently — for 51,200,245,760 parameters, the same ~51B figure this article has used throughout, stored at one byte apiece: 47.68 GiB. NVIDIA's own official release is, in other words, the least compressed treatment of this tensor of anything in this section — more conservative than the Unsloth GGUF's 4-bit floor from this article's "Running it locally" section, more conservative than turboderp's own ~5-bit EXL3 packing above, more conservative than bilikaz's NVFP4 re-quant just above. FP8, left exactly where the FP8 baseline it's benchmarked against already had it.

That conservatism is legible in the storage math too. The PLE table's own share of the model's total parameters is about 28% (51B of roughly 180B). Its share of this release's download is 38.6% — because the backbone got quantized down to a quarter its size (each U8-packed byte holding two 4-bit values, which is also why the Hub's own reported "119.6B total parameters" for this repo undercounts the real figure: it's counting packed bytes as if each were one parameter) while the table didn't move at all. Compress the part that compresses easily and leave the part that doesn't, and the part that doesn't grows as a share of what actually ships — the same table this article's own "Running it locally" section flagged for exactly this reason, at a smaller scale, on the Unsloth GGUFs.

Where the community quant's headline claim didn't survive contact with its own file sizes, this one comes with something none of the derivatives in this section offer: a controlled quality comparison against a matched baseline, rather than a compression ratio asserted on its own. NVIDIA's model card benchmarks the NVFP4 checkpoint against Qwen/Qwen3.8-Flash-Next-FP8 at identical sampling settings:

Close and genuinely mixed rather than a clean win either way — four evals move up under NVFP4, five move down, none by much. That's evidence a "12GB" claim never had to offer, and — unlike the EXL3 recipe or either Spark kit above — it comes from the lab that owns the license to publish this checkpoint, not from a single tester's own box.

Which leaves this one tensor with a longer ledger than it had before this update:

Four builders, four bets, the same 51B parameters, inside a few weeks of each other — and the one with the most reputational exposure if it goes wrong made the least aggressive choice of any of them.

The ledger

Unusually well evidenced. Four architectural changes, each with a controlled ablation at 25B–35B scale, same pipeline, one variable at a time. A GDN-versus-SWA comparison that isolates content-dependent memory from windowing. QSA quality measured across four length bands, including the bands where it loses. A residual ladder that reports loss and benchmarks separately because they disagree. An N-gram scaling study whose conclusion is mostly negative. A kernel library, FlashQLA, released alongside. Open weights, GGUFs on day zero, and a config that matches the diagram field for field. And independent corroboration from SGLang, whose host-offload result is bit-exact and costs 0.07% throughput — the strongest evidence anywhere that the N-gram table belongs off the accelerator.

Load-bearing assumptions. The 8.6× prefill figure assumes a 90% prefix-cache hit rate — stated, but it is the difference between a headline and a benchmark. The 1/9 training cost is a ratio against their own previous model with no absolute figures. And the ablations run at 25B–35B while the shipped model is 125B+51B, so every architectural conclusion is an extrapolation across roughly a 4× scale gap.

Not shown. That 51B of N-gram embedding beats 51B of additional experts — their own fixed-budget study says it doesn't clearly. Whether GR's deleted branch-mixing operator is genuinely free, since the lab shipping the competing answer the same week disagrees. What any of this does to post-training behaviour beyond the endless-generation note, which is the one place they looked and found something. And, from the community section above, what disabling 51B of PLE parameters costs on the shipped model — nobody has measured it yet, fork included.

The thing worth taking from this release is not the model, good as it is. It is a technical report from a frontier lab that keeps saying the metric we could watch and the metric we wanted diverged here — three separate times, with tables — and then tells you which one it followed. That is rarer than a new attention variant, and considerably more useful.