Qwen3.8-27B variants: the bill is tokens times bytes

mdjsonmcpcopy↗ai

2026-09-26 · 20 min · qwen · quantization · open-weights · inference-optimization · reasoning · efficiency · benchmarks

Between 20 and 24 September three derivatives of Qwen/Qwen3.8-27B landed on Hugging Face. All three start from the same dense 64-block hybrid: 48 Gated DeltaNet blocks, 16 full-attention blocks, a multi-token-prediction (MTP) head and a vision tower. I read that architecture out of the files in the Qwen3.8 open-weights article. All three are pitched as making that model better to live with, and they pull different levers.

For one user decoding one answer, a dense model is bound by memory bandwidth. Every generated token streams every decoder weight through the chip once. To first order, the time to an answer is

where $N$ is tokens per answer (thinking plus reply), $B$ is bytes read per generated token, and $W$ is memory bandwidth in bytes per second.

• ThinkingCap-Qwen3.8-27B (BottleCap AI) is a fine-tune that thinks less. It cuts $N$.
• OrcaSAQ-2-27B (OrcaRouter) is a 3.21-bit quant. It cuts $B$.
• ThinkingCap's own FP8, NVFP4, MLX and GGUF builds cut both.
• Hemmingway-1 (Altworld) is a writing fine-tune in BF16. It cuts neither.

I read the Hub API, the configs and cards, and the safetensors headers by range request wherever a repository is not gated. Numbers are measured (computed from the files), reported (the publisher's) or reasoned (my arithmetic on someone else's numbers).

Bytes per token, from the headers

$B$ is not the file size. Three things in the file are not read per token:

1. The embedding table is a lookup. A token reads one row of it, 10 KB, not the 2.54 GB table.
2. The vision tower does nothing for text.
3. The MTP head is read only when you run speculative decoding.

What streams per decoded token is the linear layers of the 64 blocks, plus the LM head, plus a few tens of MB of norms and the narrow Gated DeltaNet gates. The decoder's linear layers are 24,326,963,200 weights in every build. The builds differ only in how many bits each weight gets. It is the same count-only-what-is-read rule that sets bits per weight in runtime dynamic compression.

"On disk" is not like for like either. The BF16, FP8, NVFP4 and MLX builds carry the 0.92 GB vision tower; OrcaSAQ-2 does not.

Two rows get their own sections below: the "4-bit" MLX build is six bits per weight where it matters, and OrcaSAQ-2 moves 4.75× fewer bytes per token than BF16.

Before trusting the formula, check it against a measurement. BottleCap publishes single-request decode on one RTX PRO 6000 Blackwell, whose spec bandwidth is 1,792 GB/s. BF16 does 26.2 tok/s against a roofline of 35.0. FP8 does 44.7 against 66.5. NVFP4 does 68.0 against 110.1. So real decode reaches 62–75% of the floor, and the NVFP4 speedup it measured (2.6×) is smaller than the bytes ratio predicts (3.15×). The formula ranks builds correctly. It does not predict ratios to the decimal.

Averaged over BottleCap's twelve benchmarks on an H200 (4,800 GB/s), the floor for the base in BF16 is 186.3 s per answer. ThinkingCap alone gets it to 148.4 s, ThinkingCap in NVFP4 to 47.1 s, and OrcaSAQ-2 to 39.2 s with no change to $N$, an assumption, since OrcaRouter publishes no token counts.

At batch 1, bits are the bigger lever. ThinkingCap cuts mean total tokens from 17,450 to 13,900, a factor of 0.797. NVFP4 cuts bytes by a factor of 0.318 and OrcaSAQ-2 by 0.211. The token lever never catches up: its best case is MMMLU, 1,660 → 575 total tokens, a factor of 0.346.

The formula stops working on a busy server. There, weights are read once per step for the whole batch, and the cost moves to KV-cache reads and compute. Fewer weight bits help there only indirectly, by freeing memory for more cache. Fewer tokens cut the cache and the number of steps directly. BottleCap measured both levers on one axis: 32 MMLU-Pro questions, one H200, 16 concurrent requests, vLLM 0.26. Reported:

At 16 concurrent, the tokens bought 2.4× (8.9 → 3.7 s) and NVFP4 bought 1.4× on top (3.7 → 2.6 s). That is the reverse of the batch-1 ranking. BottleCap's throughput post goes further: on one RTX PRO 6000 with 8 GB of KV cache and 64 concurrent copies of one AIME prompt, traces 2.3× shorter finished 4.7× more tasks in 20 minutes, 112 against 24, because more short sequences fit in the cache at once. One prompt is not a workload, but the mechanism is sound.

The formula also ignores prefill. With 32,768-token prompts, the weight-only NVFP4 build (NVFP4A16, through the Marlin kernel) is no faster than BF16, 18.9 tok/s against 18.6, and its median first token comes later, 13,670 ms against 9,678. The FP8 build, which quantizes activations too, does 27.0 tok/s and 7,668 ms. The W4A4 build, with FP4 activations, does 32.8 tok/s and 6,393 ms, and is 29% slower than NVFP4 for a single request. Weight bits buy decode; activation bits buy prefill and batching.

Fewer tokens: ThinkingCap

What shipped

The BF16 checkpoint's 18 shards each have the same byte size as Qwen's, which is what the same tensors in the same dtype would give. It is gated (auto-approved) and licensed PolyForm Small Business 1.0.0 plus a personal-use grant, a real step down from the base's Apache-2.0. Five quantized builds sit alongside it.

How it was made

For this release, BottleCap does not say. The Qwen3.8 blog post states the objective ("Less number of thinking tokens to get the same answer") and one fact about training: the model "is trained at xhigh". The earlier Qwen3.6 post says a little more:

we trained on a curated set of problems covering multiple domains and difficulty levels. The training objective rewarded efficient reasoning rather than simply rewarding correctness.

"Rewarded" suggests reinforcement learning with a length-aware reward, reasoned from one word. No algorithm, dataset or length penalty is published for either model. The only other trace is the source path in the W4A4 build's quant_manifest.json, /shared/caveman_models/soup-r2-pubsoup. A directory name is not a method.

What the evaluation shows

The evaluation is the best part of the release. Both models ran through one harness on one H200 (vLLM 0.29.0), at reasoning_effort=xhigh, with Qwen's sampling (temperature 1.0, top_p 0.95, top_k 20) and MTP speculation, which BottleCap checked is accuracy-neutral: 98.23% on AIME without it, 98.13% with. Seeds run from 1 to 32, and every test set is complete except MMMLU, a fixed 10,000-question sample. Mean thinking tokens, reported:

The τ²-bench and Terminal-Bench rows count reasoning summed over whole multi-turn episodes.

"37% on average" is a mean of percentages. The card says so in its evaluation notes: the bottom row is "the mean of the per-benchmark reductions, not the ratio of the two token figures beside it". Take the ratio of the two token figures instead, 12,144 against 15,735, and the cut is 22.8% (reasoned). For total tokens, thinking plus answer, it is 17,450 → 13,900, or 20.3%.

Both averages are honest; they answer different questions. The 37.2% is what a typical benchmark sees. The 22.8% is closer to what a workload drawn evenly from these twelve would pay, and the gap has a cause. The longest-running rows shrink least. Terminal-Bench is 38.6% of the pooled thinking tokens and gives up only 10.7% of them.

"Up to 65%" is MMMLU, 65.5%. It comes from a single seed, at a cost of 1.29 points of accuracy. It is not "65% fewer tokens at equal accuracy". Every row in the table is measured at an equal setting, xhigh on both sides, and accuracy moves where it moves. Ten of twelve benchmarks stay within two points. AIME 2026 loses 3.85, and the two intervals do not overlap (98.13 ± 0.74 against 94.27 ± 1.47). AA-LCR gains 2.25.

The equal-accuracy comparison that does hold up is against the dial the base model already has. Qwen3.8's chat template takes reasoning_effort as xhigh, medium or low, and turning it down is free. BottleCap ran both models at all three settings:

expand

That is the real result. The dial buys its savings steeply: 9.16 points for 52.1%. ThinkingCap takes 37.2% for 0.86. At medium and low the fine-tune sits 7–8 points further right than the base at the same setting, for about one more point of accuracy, so the two cut different things and stack.

The other view is a budget: cap the tokens a response may generate, count anything unfinished as wrong, and sweep the cap.

expand

Under a 16K-token cap per response, ThinkingCap is more accurate than the base. If your stack already truncates, that is the version of the claim that pays.

Two side effects, both reported: truncation (a trace that never closes) drops from 0.51% to 0.34%, and MTP draft acceptance is unchanged, 53% against the base's 54%.

Fewer bits: OrcaSAQ-2

What SAQ is

The card describes "a proprietary sensitivity-aware mixed-precision quantization system", hence SAQ, sensitivity-aware quantization, and says the methodology is "not currently disclosed". The files disclose more than the card. config.json says, in part:

"quantization_config": {
  "quant_method": "exl3", "version": "1.5.1", "bits": 3, "head_bits": 6,
  "calibration": { "rows": 250, "cols": 2048 }, "codebook": "mul1",
  "mtp_bits": 4, "head_quant": "exl3-6bit", "embed_quant": "int8",
  "bits_per_weight": 3.21
}

exl3 is ExLlamaV3's format. It is a streamlined QTIP (arXiv 2406.11235): incoherence processing (blockwise Hadamard transforms, with the per-channel suh and svh vectors stored beside every tensor), then trellis-coded quantization of 16×16 tiles (the .trellis int16 tensors). OrcaRouter's own serving repository, Continuum-AI-Corp/OrcaSAQ2-kernel, says it outright: "a QTIP-style trellis code with a searched mixed-precision allocation", with "the trellis kernels" coming "from exllamav3". The codec is public. What is OrcaRouter's is the search that decides which tensor gets how many bits, and that search is unpublished.

The allocation can be read from the headers anyway. A 16×16 tile holds 256 weights, so at $K$ bits it packs $16K$ int16 values. The trellis tensor's last dimension is therefore $16K$: 48 means 3 bits, 56 means 3.5, 64 means 4. The half rates come from the mul1 codebook, according to a comment in the kernel repository. I read all four shard headers and checked every one of the 400 decoder projections against the per-tensor bits_per_weight in quantization_config.json. All 400 agree. (The model card above shows 6.77B parameters because the Hub counts stored elements, not weights: 5,465,702,400 of them are int16 trellis words.)

Measured, over the 400 decoder projections:

• 231 tensors are at 3 bits, 120 at 3.5, 43 at 4 and 6 at 2. By parameters that is 61.4%, 26.2%, 10.2% and 2.2%.
• Some choices are made by role and are the same in every block. Every attention v_proj gets 4 bits, every k_proj 3.5, every o_proj 3, and every Gated DeltaNet in_proj_qkv 3.
• Other choices are made by depth, in bands of about six blocks. Blocks 0–11 sit at about 3 bits. The only 2-bit tensors are the MLP gate_proj of blocks 12–17. Blocks 18–52 mix 3 and 3.5. The MLPs of blocks 53–63 get 3.5 and 4. Blocks 0–52 average 3.12 bits and blocks 53–63 average 3.68.
• Nothing protects the front of the network. Block 0 averages 3.12, the same as blocks 1, 2, 4 and 5.

Penjing-27B, another Qwen3.8-27B quant claiming a sensitivity-driven allocation, turned out to use a fixed first-and-last rule, identical at three bit budgets. OrcaSAQ-2 is not that. It spends its spare bits at the back of the network, cuts one tensor type in one band of the middle, and treats attention by role, which is what a search would plausibly produce. Without the search's output I cannot check that it was one.

Outside the blocks, the embedding is INT8 with a per-row scale (1.27 GB), the LM head is 6-bit EXL3 (954 MB), and the MTP head is 4-bit (213 MB). There is no vision tower: OrcaSAQ-2 is text-only.

The headline numbers

The launch post reads "55.59 → 12.06 GB — 78.3% smaller / 4.61×, 3.21 bpw · 93.2% Top-1 agreement". Each number, checked:

All measured. The 12.06 is the only split of the files I could find that produces that number; OrcaRouter does not say how it was computed. Count the same things on both sides, text weights plus MTP head, and 54.64 GB → 12.27 GB is 4.45×, 77.5% smaller. The model card's own figures, 54 GB → 12.3 GB and "4.4× smaller", are the like-for-like version. The launch post took the larger numerator from one count and the smaller denominator from another.

93.2% top-1 agreement is token-level agreement with BF16's top choice over 16,376 predicted WikiText-2 tokens, from the same pass as the perplexity. One predicted token in about fifteen differs from what BF16 would have picked. The mean KL divergence is 0.031. Perplexity goes from 5.6468 to 5.6482, or +0.02%, and that is the weakest of the three: errors in both directions cancel inside a perplexity. That is how it can move by 0.02% while 6.8% of the argmaxes flip. These are fidelity numbers, not benchmark scores, and they are the right kind to publish.

70.0 on SWE-bench Verified and 58.4 on Terminal-Bench 2.1 are placed on the card next to Claude, Gemini and GPT scores, not next to Qwen3.8-27B in BF16. The harness is not stated. Three parties have published BF16 numbers for this exact base on Terminal-Bench 2.1:

58.4 sits 11.3 to 17.4 points under every published BF16 number, and 70.0 sits 10.6 under PrismML's. I cannot attribute that gap to quantization, because I do not know the harness. But same-harness BF16 against the quant is the comparison a quantization release exists to make, and it is the one missing. The card's own limitations list says as much: "Long-horizon comparisons should use a controlled same-harness evaluation."

OrcaRouter's two documents also disagree on serving. The card reports 90.1 tok/s single-stream with MTP under a 15.7 GiB cap; the kernel repository reports 111.2 tok/s and a KV pool of 35,617 tokens against the card's 14,563, and its last commit reads "the pool figure came from a 16K run, not the 32K it ships". Neither names the physical card, so the roofline check is not possible.

Fewer bits on fewer tokens: ThinkingCap's own builds

BottleCap quantized its own fine-tune five ways. Four of the five repositories are not gated, so those headers are readable:

The MLX "4-bit" build is 6.04 bits per weight on the decoder's linear layers. The card is candid: only the MLP projections of the first 56 blocks are 4-bit, and self-attention, the wide Gated DeltaNet projections, the last eight blocks' MLPs, the LM head and the embeddings are 8-bit. With group-64 affine scales and biases, 4-bit costs 4.5 bits and 8-bit costs 8.5, and only 61.6% of the decoder's weights are in the 4-bit set. The headers sum to 18,360,565,760 bytes of decoder linears, exactly what that layout predicts. It is a deliberate trade, but the label hides a consequence: this build reads 19.76 GB per token, 21% more than NVFP4's 16.28 GB. The launch post's "21 GB, down from 52 GB" is GiB on both sides (20.99 against 51.75), so it is consistent.

Quantizing also moved $N$ a little. On NVFP4, mean completion tokens fell 9.5% on IFBench and 12.2% on AA-LCR, with both intervals excluding zero, while the medians rose 7–9%: fewer very long answers, not uniformly shorter ones. NVFP4 accuracy stays within 1.7 points of the BF16 fine-tune on four benchmarks and loses 3.0 on AA-LCR, every interval including zero. MLX lands within about two points either way. All reported, on 100 to 1,500 questions per benchmark.

Different words: Hemmingway-1

Hemmingway-1 is the control group. It is BF16 and text-only (Qwen3_5ForCausalLM, vision tower dropped). Its headers list exactly the base's 866 text and MTP tensors, with the same shapes and dtype, renamed from model.language_model.* to model.*. The chat template is byte-identical to the base's, so it thinks at xhigh by default. $B$ is unchanged at 51.25 GB per token (measured), and nobody has published $N$.

The card's "Code" link is a GitHub repository with a README, the model card and six chart images; there is no code and no word on data or method. The license is CC BY-NC 4.0, stricter than the base's. The claims are about style, on three internal benchmarks that the card flags up front: CommunicationBench is 80 real requests judged pairwise and blind, in both orders, by an outside model. There it scores 1026 against Fable 5.1's 1024, a two-point lead on 80 requests. On EQ-Bench 4, the one public benchmark, it places third at 1330, and the base model is not on the chart. One chart does bear on cost:

expand

The fine-tune wraps its answer twice as often as the model it came from, 39 against 19. More wrapper probably means more answer tokens (reasoned; no token counts are published). Hemmingway-1 changes what the tokens say, not how many there are or what each one costs. Others pulled the bits lever for it: a Hub search for "Hemmingway-1" returns 52 repositories besides Altworld's own, most of them quantized builds.

What nobody measured

• Token counts for OrcaSAQ-2 and Hemmingway-1. The calculator assumes OrcaSAQ-2 thinks as long as the base.
• Same-harness agentic scores for OrcaSAQ-2 against BF16, for a model sold on long-horizon agents.
• ThinkingCap's training method. The recipe is one sentence from a previous release.
• Both levers together. ThinkingCap's traces at OrcaSAQ-2's 10.79 GB per token would floor at 31.2 s per averaged answer on an H200, against 186.3 s for the base in BF16. Nobody has built it.

The practical reading is short. For one user on one card, bytes per token is the number to shop on, and OrcaSAQ-2's 10.79 GB is the smallest here. For a full server, tokens per answer is the number, and ThinkingCap is the only release that moved it, with seeds and intervals to show for it. BottleCap is also the only publisher that measured both levers, and in its own table the build that pulls both is the fastest.