# Qwen3.8-2B-Distill: the filter wrote most of the headline

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/qwen3-8-2b-distill
> date: 2026-08-26
> tags: qwen, distillation, quantization, gguf, evaluation, edge
[empero-ai/Qwen3.8-2B-Distill-GGUF](https://huggingface.co/empero-ai/Qwen3.8-2B-Distill-GGUF) is a full-parameter distillation of [Qwen3.8 2.4T A95B](/articles/qwen3-8-open-weights) into the smallest member of the Qwen3.5 family, quantized for llama.cpp. It has 155,055 downloads and 93 likes eleven days after upload. The pitch is compact and appealing: the reasoning curriculum of a 2.4-trillion-parameter teacher, MMLU 0.283 → 0.548, GSM8K 0.330 → 0.640, 262k context, function calling, 1.3 GB on a phone.

I went and read the files. Not the card — the five GGUF headers, the safetensors index, the `config.json`, the chat template, llama.cpp's memory allocator, the `lm-evaluation-harness` task YAMLs that produced those numbers, and the `training_args.bin` that Empero shipped alongside the weights and probably did not mean to publish as evidence.

Two things came out of it. The distillation is real and cheap, and it did something worth having. And the benchmark table is a much weaker claim than it looks, in a way the parent model card itself discloses and the GGUF card does not reprint.

| | |
|---|---|
| What it is | Qwen3.5-2B, full-parameter SFT on ~30,000 Qwen3.8 teacher traces, converted to GGUF |
| Repo | five quants, **nothing under 1 GB** — Q4_K_M is **1,312,164,224 bytes** |
| Real parameter count | **2,274,069,824** in the safetensors; **1,942,653,248** in the GGUF; **1,373,265,728** in the transformer that does the work |
| Vision | the base is a VLM. The 331M-parameter ViT is in the download and **absent from every GGUF** |
| Advertised context | **262,144** — and the GGUF header agrees, `qwen35.context_length = 262144` |
| Context it was trained at | **8,192** — `max_length` in the shipped `training_args.bin` |
| KV cache at 262,144 | **3.221 GB**, or 2.45× the weights it serves |
| Training run | 1 epoch · lr 3e-5 cosine · 5 warmup steps · `paged_adamw_8bit` · **one GPU** · seed 42 |
| The headline metric | lm-eval **flexible-extract**, which takes the *last* number in the output |
| The column not reprinted | GSM8K **strict-match: 0.545 → 0.640**, a gain of **+0.095**, not +0.310 |
| Same recipe at 4B / 9B | GSM8K **−0.065** and **−0.015**. The distillation makes the larger siblings worse |
| License | Apache-2.0, inherited. Genuinely open |

<ModelCard repo="empero-ai/Qwen3.8-2B-Distill-GGUF" />

## What is actually in the repo

Start with the byte sizes, because they are the one thing nobody can spin. The Hugging Face blob listing gives exact lengths, and each file's own GGUF tensor table gives the breakdown.

<Figure
  src="/articles/qwen3-8-2b-distill/fig1.png"
  alt="A horizontal stacked bar chart of the five GGUF files in the repository — Q4_K_M, Q5_K_M, Q6_K, Q8_0 and BF16 — each split into the token embedding table, other tensors held at the file's top precision, tensors at the nominal quant, and the tokenizer header. A dashed red line at 1 GB sits to the left of every bar."
  caption="Every file in empero-ai/Qwen3.8-2B-Distill-GGUF, decomposed by summing its own tensor table. The dashed line is the circulating ‘~1 GB on a phone’ claim; the smallest file in the repo is 1.312 GB. (Rendered from the HF blob listing and the GGUF headers.)"
/>

There is no Q3, no Q2, no IQ quant. The smallest file on offer is Q4_K_M at 1.312 GB — 1.222 GiB if you prefer binary — and it is the one the card recommends. So the "~1 GB" figure is not a rounding of anything in this repository. It has no file behind it.

The reason is structural, and it is the most interesting thing about the ladder. I decoded the tensor table of each file and summed the ggml block sizes; for Q4_K_M the total lands on 1,312,164,210 bytes against a 1,312,164,224-byte file, a 14-byte alignment remainder. Inside that total, **the Q6_K bytes (657,162,240) outweigh the Q4_K bytes (641,802,240)**. Half the byte volume of a file called Q4_K_M is at six bits, and its true rate is **5.404 bits per weight**, not 4.5.

<WeightLedger />

`token_embd.weight` alone is 417,177,600 bytes — 31.8% of the download — and llama.cpp holds it at Q6_K. That is not a quirk of this publisher's recipe. Qwen3.5's vocabulary is 248,320 tokens because it covers 201 languages plus vision and tool sentinels, and `config.json` sets `tie_word_embeddings: true`, so that matrix is also the output projection. llama.cpp will not take the output projection to four bits. The floor is 417 MB before a single transformer weight is quantized, and it is why the entire K-quant ladder is compressed into 1.31–1.61 GB.

The other surprise in that ledger is what is missing. The parent is `Qwen3_5ForConditionalGeneration` — the base is a vision-language model, and the safetensors carries a 24-layer ViT at `model.visual.*`, 331,416,576 parameters. The GGUF parameter count is 1,942,653,248, which is exactly the safetensors minus that tower, to the parameter. No GGUF in the repo contains a single `v.*` or `mm.*` tensor and there is no `mmproj` file. **663 MB of the bf16 download does nothing in llama.cpp.** The parent card is upfront that the fine-tune is text-only; the GGUF card does not mention vision at all.

<Callout type="note">
Credit where it is due: the parent card at [empero-ai/Qwen3.8-2B-Distill](https://huggingface.co/empero-ai/Qwen3.8-2B-Distill) is honest about several things the derivative card omits — that the fine-tune is text-only, that 262k is *inherited* from the base, and, crucially, that there are two eval columns. The problem is distribution. The GGUF repo has 155,055 downloads to the parent's 6,002. **96% of this model's audience reads the short card.**
</Callout>

## The metric is doing the work

Here is the table as the GGUF card prints it:

| Task | Qwen3.5-2B (base) | Qwen3.8-2B | Δ |
|---|---:|---:|---:|
| mmlu (CoT, 57 subjects) | 0.283 | **0.548** | +0.265 |
| gsm8k_cot | 0.330 | **0.640** | +0.310 |

And here is the parent card's version of the same table, which has twice as many rows:

| Task | Metric | base | student | Δ |
|---|---|---:|---:|---:|
| gsm8k_cot | exact_match (flexible) | 0.330 | 0.640 | +0.310 |
| gsm8k_cot | exact_match (**strict**) | **0.545** | 0.640 | **+0.095** |
| mmlu (CoT) | acc (flexible-extract) | 0.283 | 0.548 | +0.265 |
| mmlu (CoT) | acc (**strict-match**) | **0.004** | 0.225 | +0.221 |

Look at the second row. Under lm-eval's strict-match filter the **base model already solves 54.5% of GSM8K**, and the distilled student reaches 64.0%. The gain is +0.095. The headline +0.310 exists because the base scores 0.330 under the *other* filter — 21.5 points lower than under the strict one.

If your intuition says a "flexible" filter should never score lower than a "strict" one, that intuition is wrong, and the reason is one line of YAML.

<FilterAnatomy />

From `lm_eval/tasks/gsm8k/gsm8k-cot.yaml`, verbatim:

```yaml
filter_list:
- filter:
  - function: regex
    regex_pattern: The answer is (\-?[0-9\.\,]+).
  - function: take_first
  name: strict-match
- filter:
  - function: regex
    group_select: -1
    regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
  - function: take_first
  name: flexible-extract
```

`group_select: -1`. lm-eval's `RegexFilter` does `match = self.regex.findall(resp)` then `match = match[self.group_select]`, so `-1` means **the last match anywhere in the generation**. flexible-extract is not a relaxed strict-match; it is an unrelated rule that says "the answer is the last number-shaped substring in the output." (`take_first` is a separate filter that picks the first of N sampled responses. With `repeats: 1` it does nothing, and it has no bearing on which match wins.)

So a model that solves the problem, states the answer in the required sentence, and then adds one more sentence of sanity-checking will be scored on whatever number it happened to mention last. That is exactly the shape of an instruction-tuned base model that has not been taught to shut up. And a model that solves the problem but never writes "The answer is" scores zero under strict-match no matter how right it was. Neither filter dominates; which one flatters a model depends on how it ends its answer.

<GainSplit />

The decomposition is an identity, not an interpretation. Define each model's *extraction gap* as flexible minus strict on the same generations. Then

$$\Delta_{\text{flex}} \;=\; \Delta_{\text{strict}} \;+\; \big(\text{gap}_{\text{student}} - \text{gap}_{\text{base}}\big)$$

with nothing left over. For GSM8K at 2B: $+0.310 = +0.095 + 0.215$. **69% of the advertised GSM8K gain is the base model's extraction gap closing** — the student stops talking once it has answered, so both filters land on the same token and its two columns agree to three decimals.

## The siblings settle it

Empero released the same recipe at three scales on the same day, and published all three tables. That is a natural experiment they handed us for free.

<Figure
  src="/articles/qwen3-8-2b-distill/fig3.png"
  alt="Two grouped bar charts, gsm8k_cot and MMLU CoT, each showing base and distilled scores under strict-match and flexible-extract for the 2B, 4B and 9B models, with the delta printed above each pair. GSM8K deltas are positive only at 2B and negative at 4B and 9B; MMLU deltas are positive everywhere."
  caption="The same teacher, the same curriculum, three student sizes. GSM8K improves only where the base had an extraction gap to give back. (Rendered from the empero-ai Qwen3.8-2B / -4B / -9B model cards.)"
/>

The base models' GSM8K extraction gaps, flexible minus strict:

| Base | gap | reported Δ after distillation |
|---|---:|---:|
| Qwen3.5-2B | **−0.215** | **+0.310** |
| Qwen3.5-4B | 0.000 | **−0.065** |
| Qwen3.5-9B | +0.010 | **−0.015** |

The only scale with a headline GSM8K gain is the only scale whose base had a large extraction penalty to recover. At 4B and 9B, where the base already stopped at its answer, the identical recipe with the identical teacher makes the student **worse** — 6.5 and 2.5 points worse under a fixed extraction rule. If what was transferred were reasoning, it would not evaporate at 4B, where the student has twice the capacity to hold it.

MMLU is a different and more favourable case, and I want to be careful here because the sceptical reading does not extend cleanly. MMLU CoT improves at all three scales under both filters. But the base's strict-match is **0.004**. That is not a knowledge measurement — the `mmlu_flan_cot_zeroshot` strict filter is four lookbehinds demanding a literal "The answer is (X)" (or one of three near-identical variants) as the last thing on its line, zero-shot, with no exemplar ever shown. The base essentially never writes it. So on MMLU, strict-match cannot see the base at all, and *neither column isolates reasoning*. The +0.265 is real as a harness result. How much of it is knowledge and how much is the student learning where to put the answer is a question these four numbers cannot answer, and the control that would answer it — rescoring the base few-shot, or with a format-forcing instruction — is not on the card.

The one hint we have points the same way as everything else: `gsm8k_cot` is an **8-shot** task whose YAML bakes in eight exemplars that all end "The answer is N." MMLU CoT is **zero-shot**. The task where the base is shown the format eight times yields +0.095. The task where it is never shown the format yields +0.265.

## What the eval report leaves out

The card says "Measured with `lm-evaluation-harness`, HF backend, identical settings for base and student." That is more than most publishers write down. It is still not enough to reproduce a number, and the gaps are specific:

- **No shot count.** The task defaults are 8-shot for `gsm8k_cot` and 0-shot for `mmlu_flan_cot_zeroshot`; the card names neither.
- **No harness version or commit.** Both YAMLs are at `metadata.version: 3.0`, and gsm8k's filter list has changed across releases. A bare "lm-evaluation-harness" does not pin a filter.
- **Decoding contradicts the task.** The card states `temperature=0.6, top_p=0.95, top_k=20`. Both task YAMLs set `do_sample: false`. To get sampling you must override with `--gen_kwargs`, which makes the run stochastic — and no seed and no confidence interval are reported. If they did not override, the numbers are greedy and the stated sampling is decoration.
- **No `max_gen_toks`.** The HF backend's default is **256** (`lm_eval/models/huggingface.py`, `max_gen_toks` returns `256`). This is a model whose card says "every answer opens with a `<think>` block" and recommends `max_new_tokens=16384`. Those two facts cannot both be true of the same run. Either the budget was raised — and a reasoning model's score is a function of its thinking budget, so the value matters — or every generation was cut at 256 tokens.
- **Which MMLU?** `mmlu_flan_cot_zeroshot` has `validation_split: validation` and no `test_split`, so it runs on MMLU's **1,531-question validation split**, not the 14,042-question test set. The card's "~1,700 questions" is in that neighbourhood. These are not the MMLU numbers anyone else reports.

That last point is worth pulling out, because it reframes the base model entirely. Qwen's own card for [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) reports **MMLU-Redux 69.2** in non-thinking mode and **79.6** in thinking mode, and MMLU-Pro 55.3 / 66.5. Whatever "0.283" measures about Qwen3.5-2B, it is not that model's knowledge; it is that model's score under one zero-shot generative harness whose extractor takes the last parenthesised capital letter it can find. The distilled student's 0.548 sits fourteen points below the base's own published MMLU-Redux.

## The training run, from the file they shipped

`training_args.bin` is in the parent repo — 5,777 bytes, a `torch.save` of TRL's `SFTConfig`. Unpickle it and the entire run is there:

```python
learning_rate      = 3e-05        lr_scheduler_type = cosine
num_train_epochs   = 1            warmup_steps      = 5
per_device_train_batch_size = 8   gradient_accumulation_steps = 8
max_length         = 8192         truncation_mode   = keep_start
packing            = True         packing_strategy  = bfd
assistant_only_loss = False       loss_type         = nll
optim              = paged_adamw_8bit
bf16 = True   gradient_checkpointing = True   seed = 42
_n_gpu = 1    distributed_type = NO    fsdp = None    deepspeed = None
```

Four things fall out of that block.

**It is a genuine full fine-tune.** `paged_adamw_8bit` with gradient checkpointing on a single GPU is precisely what you reach for when you are fitting every parameter of a 2.27B bf16 model plus optimizer state onto one card. Nobody pages an 8-bit optimizer to train an adapter. The repo has one 4.55 GB `model.safetensors` and no `adapter_config.json`. The "full-parameter SFT, not LoRA" claim survives contact with the artifacts.

**It is about a hundred optimizer steps.** Effective batch is 8 × 8 = 64 sequences of 8,192 packed tokens, one epoch over ~30,000 traces. If a CoT trace averages 1,000–3,000 tokens, that is 30–90M tokens, 3,700–11,000 packed bins, and **roughly 60–170 optimizer steps**. `warmup_steps: 5` and `save_steps: 25` with `save_total_limit: 3` are consistent with a run in exactly that range. Whatever "the full reasoning curriculum of a 2.4T model" means, this is a short, cheap style transfer on one GPU — which is a perfectly good thing to build, and is not what the phrase suggests.

**The loss is on everything.** `assistant_only_loss: False` with `packing: True`: sequences are bin-packed into 8,192-token bins and trained with plain next-token NLL across the whole bin, prompts included. A meaningful share of the gradient is spent learning to predict user turns.

**`max_length: 8192`.** Every token this model ever saw during its reasoning fine-tune lived in an 8,192-token window, with anything longer truncated from the end. The card advertises 262,144. That is 3.1% coverage, and the card is careful to say the context is *inherited* — but "262k context with real reasoning" is a claim about two things that were never trained together.

## The context is real, and it is not free

To its credit, the 262k claim survives the check I expected it to fail. The GGUF header does not quietly say 32768:

```
qwen35.context_length         = 262144
qwen35.block_count            = 25
qwen35.attention.head_count   = 8
qwen35.attention.head_count_kv = 2
qwen35.attention.key_length   = 256
qwen35.rope.freq_base         = 10000000.0
qwen35.rope.dimension_count   = 64
qwen35.full_attention_interval = 4
```

`config.json` agrees: `max_position_embeddings: 262144`, `rope_theta: 1e7`. Function calling is likewise real — the embedded chat template renders a `<tools>` block and a `<tool_call><function=…>` grammar, and the tokenizer carries `<tool_call>` and `</tool_call>` as dedicated tokens at ids 248058–248059. Both are inherited wholesale from the base, and neither was evaluated by Empero. The base's own card reports BFCL-V4 43.6 and TAU2-Bench 48.8; those are Qwen's numbers for the base, and they are the only tool-use numbers that exist for this model.

What the card does not put next to 262,144 is the memory bill.

<PhoneBudget />

The arithmetic is fixed by the config and it is short. `num_key_value_heads: 2`, `head_dim: 256`, and `layer_types` is eighteen `linear_attention` layers to six `full_attention` — so per token the cache holds 2 (K and V) × 2 heads × 256 dims × 6 layers = **6,144 values, or 12,288 bytes at f16**. I checked that llama.cpp really does skip the recurrent layers rather than allocating for all twenty-four; in `src/llama-model.cpp` the hybrid memory filter for `LLM_ARCH_QWEN35` is

```cpp
filter_attn = [&](uint32_t il) {
    return il < hparams.n_layer() && !hparams.is_recr(il);
};
```

which also excludes `blk.24`, the MTP block, since its index is not less than `n_layer`. The cache is sized to `cparams.n_ctx_seq` — the full `--ctx-size` — and allocated when the model loads, not as the conversation grows.

So at the advertised context: 262,144 × 12,288 = **3,221,225,472 bytes**. The KV cache is **2.45× the Q4_K_M weights**, and the two are equal at **106,784 tokens**.

<Figure
  src="/articles/qwen3-8-2b-distill/fig2.png"
  alt="A log-x line chart of memory against context length from 2k to 262k tokens, showing KV cache curves for f16, q8_0 and q4_0 against a flat line for the 1.312 GB Q4_K_M weights, plus a dashed curve for a hypothetical all-attention model, with markers at the 8,192-token training window and the 262,144-token advertised context."
  caption="KV bytes per token derived from config.json; the allocation policy read out of llama.cpp's hybrid memory filter for LLM_ARCH_QWEN35. The f16 cache overtakes the whole model file at 107k tokens. (Rendered from the shipped config and llama.cpp source.)"
/>

The eighteen [Gated DeltaNet](/articles/ltc-gated-delta) layers are the reason this is survivable at all. They keep a fixed recurrent state — 16 heads × 128 × 128 in f32 plus a 6,144-channel convolution window, 20.2 MB total — that does not grow with context. Had all twenty-four layers been full attention, 262k would want 12.9 GB of cache and no handset would be in the conversation. That saving is Qwen's architecture, not the distillation's, and the card that advertises 262k credits it to nobody.

The practical lever is not the quant, it is `--cache-type-k`. Going to `q8_0` cuts the cache to 1.71 GB at full context; `q4_0` takes it to 0.91 GB — below the weights again. [Rotate-then-quantize](/articles/turboquant-kv-cache) is the current state of the art for doing that without wrecking the attention scores, and on a model with this weights-to-cache ratio it matters more than the difference between Q4_K_M and Q6_K.

## The reasoning is off by default

One last thing, and it is the kind that only shows up if you open the file. The GGUF card says:

> The model is a reasoning model: every answer opens with a `<think>` block, so allow a generous `-n` and strip the `<think>...</think>` span for end users. Use the built-in chat template (`-cnv`).

The built-in chat template ends like this:

```jinja
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is true %}
        {{- '<think>\n' }}
    {%- else %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}
```

Thinking is **off** unless the caller passes `enable_thinking=true`. The default branch pre-fills a closed, empty think block, which is Qwen3.5's way of switching reasoning off. And `chat_template.jinja` in the distill repo is byte-identical to the base's — 7,755 bytes, same MD5. Diff the two `config.json` files and the *only* substantive change in the entire repository is `eos_token_id` moving from 248044 (`<|endoftext|>`) to 248046 (`<|im_end|>`), which is a correct and sensible fix for a chat SFT.

Which means the artifact does not agree with its own instructions. A model sold on distilled chain-of-thought ships a template whose default suppresses it, and a one-line change to that template would have fixed it. It also raises a question about the evals that the card cannot answer: lm-eval only applies a chat template when you pass `--apply_chat_template`, and passing `enable_thinking=true` through it needs `--chat_template_args`. The card says neither. If the template was applied without that flag, both models were scored with thinking disabled.

## The ledger

**What is genuinely here.** A real full-parameter SFT, Apache-2.0, of a strong small base on ~30,000 teacher traces, for what looks like a hundred optimizer steps on one GPU — and it visibly taught the model to state an answer and stop. That is worth having. Output discipline is most of what separates a base checkpoint from something you can put behind an API, and this run bought it for the price of a weekend. The GGUFs are correctly built, the header metadata is complete and truthful, the tensor tables reconcile to fourteen bytes of padding, and the SHA256SUMS file is there.

**What the numbers do not support.** "1 GB on a phone" — the smallest file is 1.312 GB, a third of it is one embedding table that llama.cpp will not quantize below Q6_K, and the weights are the cheap part once you ask for context. "GSM8K 33 → 64" — the same card prints 54.5 → 64.0 under a fixed extraction rule, and 69% of the advertised gain is the base's extraction penalty closing. "The full reasoning curriculum of a 2.4T flagship" — one epoch, 30k traces, 8,192-token window, loss on the prompts too. "262k with real reasoning" — the context is inherited and genuine, the reasoning was trained at 8k, and the two have never met.

**What I could not settle.** How much of MMLU's +0.265 is knowledge and how much is format. The base's strict-match of 0.004 means the strict column cannot see it, so the split that works cleanly on GSM8K does not transfer, and the ablation that would decide it — the base rescored few-shot or with a format-forcing prompt — was not run. I also could not verify the teacher traces, which come from "internal Qwen3.8 distillation datasets" that are not published, or check whether the eval used the chat template at all.

**What I would watch.** Publishers who report both lm-eval filters, and readers who look at the second one. Empero printed the strict-match column on all three parent cards and it undercuts their own headline at every scale — that is more disclosure than most releases manage. The failure is that the number 96% of their users see is the derivative card, which reprints only the flattering column. The fix is not more honesty from the publisher. It is that `flexible-extract` is a badly named filter that people read as "strict-match, but fairer", and it is neither.