Qwen3.8-2B-Distill: the filter wrote most of the headline

empero-ai/Qwen3.8-2B-Distill-GGUF is a full-parameter distillation of Qwen3.8 2.4T A95B into the smallest member of the Qwen3.5 family, quantized for llama.cpp. It has 155,055 downloads and 93 likes eleven days after upload. The pitch is compact and appealing: the reasoning curriculum of a 2.4-trillion-parameter teacher, MMLU 0.283 → 0.548, GSM8K 0.330 → 0.640, 262k context, function calling, 1.3 GB on a phone.

I went and read the files. Not the card — the five GGUF headers, the safetensors index, the config.json, the chat template, llama.cpp's memory allocator, the lm-evaluation-harness task YAMLs that produced those numbers, and the training_args.bin that Empero shipped alongside the weights and probably did not mean to publish as evidence.

Two things came out of it. The distillation is real and cheap, and it did something worth having. And the benchmark table is a much weaker claim than it looks, in a way the parent model card itself discloses and the GGUF card does not reprint.

What is actually in the repo

Start with the byte sizes, because they are the one thing nobody can spin. The Hugging Face blob listing gives exact lengths, and each file's own GGUF tensor table gives the breakdown.

expand

There is no Q3, no Q2, no IQ quant. The smallest file on offer is Q4_K_M at 1.312 GB — 1.222 GiB if you prefer binary — and it is the one the card recommends. So the "~1 GB" figure is not a rounding of anything in this repository. It has no file behind it.

The reason is structural, and it is the most interesting thing about the ladder. I decoded the tensor table of each file and summed the ggml block sizes; for Q4_K_M the total lands on 1,312,164,210 bytes against a 1,312,164,224-byte file, a 14-byte alignment remainder. Inside that total, the Q6_K bytes (657,162,240) outweigh the Q4_K bytes (641,802,240). Half the byte volume of a file called Q4_K_M is at six bits, and its true rate is 5.404 bits per weight, not 4.5.

token_embd.weight alone is 417,177,600 bytes — 31.8% of the download — and llama.cpp holds it at Q6_K. That is not a quirk of this publisher's recipe. Qwen3.5's vocabulary is 248,320 tokens because it covers 201 languages plus vision and tool sentinels, and config.json sets tie_word_embeddings: true, so that matrix is also the output projection. llama.cpp will not take the output projection to four bits. The floor is 417 MB before a single transformer weight is quantized, and it is why the entire K-quant ladder is compressed into 1.31–1.61 GB.

The other surprise in that ledger is what is missing. The parent is Qwen3_5ForConditionalGeneration — the base is a vision-language model, and the safetensors carries a 24-layer ViT at model.visual.*, 331,416,576 parameters. The GGUF parameter count is 1,942,653,248, which is exactly the safetensors minus that tower, to the parameter. No GGUF in the repo contains a single v.* or mm.* tensor and there is no mmproj file. 663 MB of the bf16 download does nothing in llama.cpp. The parent card is upfront that the fine-tune is text-only; the GGUF card does not mention vision at all.

The metric is doing the work

Here is the table as the GGUF card prints it:

And here is the parent card's version of the same table, which has twice as many rows:

Look at the second row. Under lm-eval's strict-match filter the base model already solves 54.5% of GSM8K, and the distilled student reaches 64.0%. The gain is +0.095. The headline +0.310 exists because the base scores 0.330 under the other filter — 21.5 points lower than under the strict one.

If your intuition says a "flexible" filter should never score lower than a "strict" one, that intuition is wrong, and the reason is one line of YAML.

Olivia started with 23 dollars. Five bagels at 3 dollars each cost 5 * 3 = 15 dollars. So 23 - 15 = 8. The answer is 8.
Checking that: 5 bagels at 3 dollars is 15 dollars of spending, and 23 - 15 leaves 8, so she keeps roughly a third of the 23 she walked in with.

From lm_eval/tasks/gsm8k/gsm8k-cot.yaml, verbatim:

filter_list:
- filter:
  - function: regex
    regex_pattern: The answer is (\-?[0-9\.\,]+).
  - function: take_first
  name: strict-match
- filter:
  - function: regex
    group_select: -1
    regex_pattern: (-?[$0-9.,]{2,})|(-?[0-9]+)
  - function: take_first
  name: flexible-extract

group_select: -1. lm-eval's RegexFilter does match = self.regex.findall(resp) then match = match[self.group_select], so -1 means the last match anywhere in the generation. flexible-extract is not a relaxed strict-match; it is an unrelated rule that says "the answer is the last number-shaped substring in the output." (take_first is a separate filter that picks the first of N sampled responses. With repeats: 1 it does nothing, and it has no bearing on which match wins.)

So a model that solves the problem, states the answer in the required sentence, and then adds one more sentence of sanity-checking will be scored on whatever number it happened to mention last. That is exactly the shape of an instruction-tuned base model that has not been taught to shut up. And a model that solves the problem but never writes "The answer is" scores zero under strict-match no matter how right it was. Neither filter dominates; which one flatters a model depends on how it ends its answer.

The decomposition is an identity, not an interpretation. Define each model's extraction gap as flexible minus strict on the same generations. Then

$\Delta_{\text{flex}} \;=\; \Delta_{\text{strict}} \;+\; \big(\text{gap}_{\text{student}} - \text{gap}_{\text{base}}\big)$

with nothing left over. For GSM8K at 2B: $+0.310 = +0.095 + 0.215$. 69% of the advertised GSM8K gain is the base model's extraction gap closing — the student stops talking once it has answered, so both filters land on the same token and its two columns agree to three decimals.

The siblings settle it

Empero released the same recipe at three scales on the same day, and published all three tables. That is a natural experiment they handed us for free.

expand

The base models' GSM8K extraction gaps, flexible minus strict:

The only scale with a headline GSM8K gain is the only scale whose base had a large extraction penalty to recover. At 4B and 9B, where the base already stopped at its answer, the identical recipe with the identical teacher makes the student worse — 6.5 and 2.5 points worse under a fixed extraction rule. If what was transferred were reasoning, it would not evaporate at 4B, where the student has twice the capacity to hold it.

MMLU is a different and more favourable case, and I want to be careful here because the sceptical reading does not extend cleanly. MMLU CoT improves at all three scales under both filters. But the base's strict-match is 0.004. That is not a knowledge measurement — the mmlu_flan_cot_zeroshot strict filter is four lookbehinds demanding a literal "The answer is (X)" (or one of three near-identical variants) as the last thing on its line, zero-shot, with no exemplar ever shown. The base essentially never writes it. So on MMLU, strict-match cannot see the base at all, and neither column isolates reasoning. The +0.265 is real as a harness result. How much of it is knowledge and how much is the student learning where to put the answer is a question these four numbers cannot answer, and the control that would answer it — rescoring the base few-shot, or with a format-forcing instruction — is not on the card.

The one hint we have points the same way as everything else: gsm8k_cot is an 8-shot task whose YAML bakes in eight exemplars that all end "The answer is N." MMLU CoT is zero-shot. The task where the base is shown the format eight times yields +0.095. The task where it is never shown the format yields +0.265.

What the eval report leaves out

The card says "Measured with lm-evaluation-harness, HF backend, identical settings for base and student." That is more than most publishers write down. It is still not enough to reproduce a number, and the gaps are specific:

• No shot count. The task defaults are 8-shot for gsm8k_cot and 0-shot for mmlu_flan_cot_zeroshot; the card names neither.
• No harness version or commit. Both YAMLs are at metadata.version: 3.0, and gsm8k's filter list has changed across releases. A bare "lm-evaluation-harness" does not pin a filter.
• Decoding contradicts the task. The card states temperature=0.6, top_p=0.95, top_k=20. Both task YAMLs set do_sample: false. To get sampling you must override with --gen_kwargs, which makes the run stochastic — and no seed and no confidence interval are reported. If they did not override, the numbers are greedy and the stated sampling is decoration.
• No max_gen_toks. The HF backend's default is 256 (lm_eval/models/huggingface.py, max_gen_toks returns 256). This is a model whose card says "every answer opens with a <think> block" and recommends max_new_tokens=16384. Those two facts cannot both be true of the same run. Either the budget was raised — and a reasoning model's score is a function of its thinking budget, so the value matters — or every generation was cut at 256 tokens.
• Which MMLU? mmlu_flan_cot_zeroshot has validation_split: validation and no test_split, so it runs on MMLU's 1,531-question validation split, not the 14,042-question test set. The card's "~1,700 questions" is in that neighbourhood. These are not the MMLU numbers anyone else reports.

That last point is worth pulling out, because it reframes the base model entirely. Qwen's own card for Qwen/Qwen3.5-2B reports MMLU-Redux 69.2 in non-thinking mode and 79.6 in thinking mode, and MMLU-Pro 55.3 / 66.5. Whatever "0.283" measures about Qwen3.5-2B, it is not that model's knowledge; it is that model's score under one zero-shot generative harness whose extractor takes the last parenthesised capital letter it can find. The distilled student's 0.548 sits fourteen points below the base's own published MMLU-Redux.

The training run, from the file they shipped

training_args.bin is in the parent repo — 5,777 bytes, a torch.save of TRL's SFTConfig. Unpickle it and the entire run is there:

learning_rate      = 3e-05        lr_scheduler_type = cosine
num_train_epochs   = 1            warmup_steps      = 5
per_device_train_batch_size = 8   gradient_accumulation_steps = 8
max_length         = 8192         truncation_mode   = keep_start
packing            = True         packing_strategy  = bfd
assistant_only_loss = False       loss_type         = nll
optim              = paged_adamw_8bit
bf16 = True   gradient_checkpointing = True   seed = 42
_n_gpu = 1    distributed_type = NO    fsdp = None    deepspeed = None

Four things fall out of that block.

It is a genuine full fine-tune. paged_adamw_8bit with gradient checkpointing on a single GPU is precisely what you reach for when you are fitting every parameter of a 2.27B bf16 model plus optimizer state onto one card. Nobody pages an 8-bit optimizer to train an adapter. The repo has one 4.55 GB model.safetensors and no adapter_config.json. The "full-parameter SFT, not LoRA" claim survives contact with the artifacts.

It is about a hundred optimizer steps. Effective batch is 8 × 8 = 64 sequences of 8,192 packed tokens, one epoch over ~30,000 traces. If a CoT trace averages 1,000–3,000 tokens, that is 30–90M tokens, 3,700–11,000 packed bins, and roughly 60–170 optimizer steps. warmup_steps: 5 and save_steps: 25 with save_total_limit: 3 are consistent with a run in exactly that range. Whatever "the full reasoning curriculum of a 2.4T model" means, this is a short, cheap style transfer on one GPU — which is a perfectly good thing to build, and is not what the phrase suggests.

The loss is on everything. assistant_only_loss: False with packing: True: sequences are bin-packed into 8,192-token bins and trained with plain next-token NLL across the whole bin, prompts included. A meaningful share of the gradient is spent learning to predict user turns.

max_length: 8192. Every token this model ever saw during its reasoning fine-tune lived in an 8,192-token window, with anything longer truncated from the end. The card advertises 262,144. That is 3.1% coverage, and the card is careful to say the context is inherited — but "262k context with real reasoning" is a claim about two things that were never trained together.

The context is real, and it is not free

To its credit, the 262k claim survives the check I expected it to fail. The GGUF header does not quietly say 32768:

qwen35.context_length         = 262144
qwen35.block_count            = 25
qwen35.attention.head_count   = 8
qwen35.attention.head_count_kv = 2
qwen35.attention.key_length   = 256
qwen35.rope.freq_base         = 10000000.0
qwen35.rope.dimension_count   = 64
qwen35.full_attention_interval = 4

config.json agrees: max_position_embeddings: 262144, rope_theta: 1e7. Function calling is likewise real — the embedded chat template renders a <tools> block and a <tool_call><function=…> grammar, and the tokenizer carries <tool_call> and </tool_call> as dedicated tokens at ids 248058–248059. Both are inherited wholesale from the base, and neither was evaluated by Empero. The base's own card reports BFCL-V4 43.6 and TAU2-Bench 48.8; those are Qwen's numbers for the base, and they are the only tool-use numbers that exist for this model.

What the card does not put next to 262,144 is the memory bill.

The arithmetic is fixed by the config and it is short. num_key_value_heads: 2, head_dim: 256, and layer_types is eighteen linear_attention layers to six full_attention — so per token the cache holds 2 (K and V) × 2 heads × 256 dims × 6 layers = 6,144 values, or 12,288 bytes at f16. I checked that llama.cpp really does skip the recurrent layers rather than allocating for all twenty-four; in src/llama-model.cpp the hybrid memory filter for LLM_ARCH_QWEN35 is

filter_attn = [&](uint32_t il) {
    return il < hparams.n_layer() && !hparams.is_recr(il);
};

which also excludes blk.24, the MTP block, since its index is not less than n_layer. The cache is sized to cparams.n_ctx_seq — the full --ctx-size — and allocated when the model loads, not as the conversation grows.

So at the advertised context: 262,144 × 12,288 = 3,221,225,472 bytes. The KV cache is 2.45× the Q4_K_M weights, and the two are equal at 106,784 tokens.

expand

The eighteen Gated DeltaNet layers are the reason this is survivable at all. They keep a fixed recurrent state — 16 heads × 128 × 128 in f32 plus a 6,144-channel convolution window, 20.2 MB total — that does not grow with context. Had all twenty-four layers been full attention, 262k would want 12.9 GB of cache and no handset would be in the conversation. That saving is Qwen's architecture, not the distillation's, and the card that advertises 262k credits it to nobody.

The practical lever is not the quant, it is --cache-type-k. Going to q8_0 cuts the cache to 1.71 GB at full context; q4_0 takes it to 0.91 GB — below the weights again. Rotate-then-quantize is the current state of the art for doing that without wrecking the attention scores, and on a model with this weights-to-cache ratio it matters more than the difference between Q4_K_M and Q6_K.

The reasoning is off by default

One last thing, and it is the kind that only shows up if you open the file. The GGUF card says:

The model is a reasoning model: every answer opens with a <think> block, so allow a generous -n and strip the <think>...</think> span for end users. Use the built-in chat template (-cnv).

The built-in chat template ends like this:

{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
    {%- if enable_thinking is defined and enable_thinking is true %}
        {{- '<think>\n' }}
    {%- else %}
        {{- '<think>\n\n</think>\n\n' }}
    {%- endif %}
{%- endif %}

Thinking is off unless the caller passes enable_thinking=true. The default branch pre-fills a closed, empty think block, which is Qwen3.5's way of switching reasoning off. And chat_template.jinja in the distill repo is byte-identical to the base's — 7,755 bytes, same MD5. Diff the two config.json files and the only substantive change in the entire repository is eos_token_id moving from 248044 (<|endoftext|>) to 248046 (<|im_end|>), which is a correct and sensible fix for a chat SFT.

Which means the artifact does not agree with its own instructions. A model sold on distilled chain-of-thought ships a template whose default suppresses it, and a one-line change to that template would have fixed it. It also raises a question about the evals that the card cannot answer: lm-eval only applies a chat template when you pass --apply_chat_template, and passing enable_thinking=true through it needs --chat_template_args. The card says neither. If the template was applied without that flag, both models were scored with thinking disabled.

The ledger

What is genuinely here. A real full-parameter SFT, Apache-2.0, of a strong small base on ~30,000 teacher traces, for what looks like a hundred optimizer steps on one GPU — and it visibly taught the model to state an answer and stop. That is worth having. Output discipline is most of what separates a base checkpoint from something you can put behind an API, and this run bought it for the price of a weekend. The GGUFs are correctly built, the header metadata is complete and truthful, the tensor tables reconcile to fourteen bytes of padding, and the SHA256SUMS file is there.

What the numbers do not support. "1 GB on a phone" — the smallest file is 1.312 GB, a third of it is one embedding table that llama.cpp will not quantize below Q6_K, and the weights are the cheap part once you ask for context. "GSM8K 33 → 64" — the same card prints 54.5 → 64.0 under a fixed extraction rule, and 69% of the advertised gain is the base's extraction penalty closing. "The full reasoning curriculum of a 2.4T flagship" — one epoch, 30k traces, 8,192-token window, loss on the prompts too. "262k with real reasoning" — the context is inherited and genuine, the reasoning was trained at 8k, and the two have never met.

What I could not settle. How much of MMLU's +0.265 is knowledge and how much is format. The base's strict-match of 0.004 means the strict column cannot see it, so the split that works cleanly on GSM8K does not transfer, and the ablation that would decide it — the base rescored few-shot or with a format-forcing prompt — was not run. I also could not verify the teacher traces, which come from "internal Qwen3.8 distillation datasets" that are not published, or check whether the eval used the chat template at all.

What I would watch. Publishers who report both lm-eval filters, and readers who look at the second one. Empero printed the strict-match column on all three parent cards and it undercuts their own headline at every scale — that is more disclosure than most releases manage. The failure is that the number 96% of their users see is the derivative card, which reprints only the flattering column. The fix is not more honesty from the publisher. It is that flexible-extract is a badly named filter that people read as "strict-match, but fairer", and it is neither.