# Whittle MoE 27B: the routers moved 4.5 degrees

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/qwen3-8-whittle-moe
> date: 2026-08-26
> tags: moe, model-compression, open-weights, evaluation, quantization
The pitch for [Whittle MoE 27B](https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B) is one of the better ones I have read this year. Take a dense Qwen3.8-27B. Cut every feed-forward layer into 64 experts. Freeze every neuron. Train **only the routers**, and watch a model that scored 4 out of 39 on a knowledge quiz come back to 28. A model that heals its own router.

The routers moved 4.50 degrees.

I measured that by pulling the shipped `mlp.gate.weight` out of the safetensors shards and comparing it, row by row, against the k-means centroid matrix the build script initialised it with — which the author also published, in a different repository, as `moe27_plan_gen2.pt`. Across all 64 layers and all 4096 router rows the mean angular change is 4.50°, the median is 4.02°, 99.3% of rows sit inside 10°, and the row norms are 1.00062 against an initialisation of exactly 1.0. The router that supposedly did the healing is, to three decimal places, the untrained one.

That is not the end of the story, and it is not the whole criticism either. Underneath a model card that is wrong in most of its particulars is a 675-line research log that is better than almost any model card I have read — it publishes its failed runs, catches its own train/test contamination before spending compute, and reasons its way to a genuinely sharp result about repetition. The interesting artefact in this repository is not the model.

| | |
|---|---|
| Model | [logic65/Qwen3.8-Whittle-MoE-27B-A17.8B](https://huggingface.co/logic65/Qwen3.8-Whittle-MoE-27B-A17.8B), Apache-2.0 · 13.8k downloads · created 18 Aug 2026 |
| Parent | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — see [Qwen3.8, weights in hand](/articles/qwen3-8-open-weights) |
| Method | MoEfication-style FFN partition: 17408 = 5120 shared + 64 × 192 routed, top-16 |
| Total, summed from tensor shapes | **26,917,297,664** — matches the index's `total_parameters` and the GGUF element count |
| Active per token | **17,857,601,024 = 66.3%** of the model. The repo name (A17.8B) is right; the card's title (A18B) and the findings log (18.03B) are not |
| Params the MoE adds | 21,299,200 — **0.079%** of the model |
| The "router-only" adapter | 337,890,304 trainable, of which the routers are **6.2%** |
| Router drift from init | **4.50° mean**, norms 1.0006 |
| The eval | `loop_test.py` and `q38_battery2.py`, both public, both standard-library, both seeded |

<ModelCard repo="logic65/Qwen3.8-Whittle-MoE-27B-A17.8B" />

## The arithmetic comes first

`config.json` gives the shape without ambiguity: `hidden_size` 5120, `intermediate_size` 17408, `num_hidden_layers` 64, `num_experts` 64, `num_experts_per_tok` 16, `moe_intermediate_size` 192, `shared_expert_intermediate_size` 5120.

And `64 × 192 + 5120 = 17408`. The parent's dense FFN width, to the neuron. Nothing invented, nothing dropped.

That last line is what everyone quotes, and it is true. What it hides is that 5120 of those 17408 neurons — 29.4% of every FFN — sit in an **always-on shared expert** that no routing decision can skip. Top-16 of the remaining 64 slivers adds 3072 more, so 8192 of 17408 run per token: 47.1% of the FFN.

Then you have to count the rest of the model, and this is where the label falls apart. I summed all 1107 tensor shapes by reading the safetensors headers of the 15 shards over HTTP range requests (eight bytes for the header length, then the JSON header — no need to move 54 GB to learn what is in it):

<Figure
  src="/articles/qwen3-8-whittle-moe/fig1.png"
  alt="Two panels. The top panel is a horizontal stacked bar of 26.92 billion parameters, split into embeddings and head 2.54B, full attention 1.68B, gated DeltaNet 5.56B, a shared expert of 5.03B, 3.02B of routed experts that fire, and 9.06B of routed experts that are skipped, drawn hatched in red. A green arrow spans the first five segments and is labelled active per token 17.86B equals 66.3 percent of the model. The bottom panel plots active parameters against k, the number of experts routed per token, as a straight line rising from 15.03B at k equals 1 to 26.92B at k equals 64, with a shaded always-on floor at 14.84B that the line never approaches and a marked point at the shipped k equals 16, 17.86B."
  caption="Summed from all 1107 tensor shapes in model.safetensors.index.json and the 15 shard headers. The routed pool is 12.08B of 26.92B, so three quarters of it being skipped saves a third of the model. (Own measurement from the published checkpoint.)"
/>

The routed expert pool is 12,079,595,520 parameters — 44.9% of the model. Everything else is always on: 14,837,702,144. So

$$
\text{active}(k) = 14{,}837{,}702{,}144 + k \cdot 188{,}743{,}680
$$

and at the shipped `k = 16` that is **17,857,601,024 active parameters, 66.34% of the total**.

Two things follow. First, the repository name is the only place the number is stated correctly: 17.86B rounds to A17.8B or A17.9B, not the A18B in the card's title, and not the "27.09B total and 18.03B active" in the findings log — the total is 26.92B and the active count is 0.17B lower than claimed. Second, and much more important: **there is no value of k that makes this model sparse.** At `k = 1` it still runs 15.03B parameters, 55.8% of the weights. The floor is set by the shared expert and the attention stack, and routing cannot touch either.

<FfnSplit />

For scale, on the same definition and their publishers’ own totals: DeepSeek-V3 runs 37B of 671B, 5.5% active; Mixtral 8×7B ran 12.9B of 46.7B, 27.6%; and the [Switch Transformer](/articles/switch-transformer) that started this line of work routed each token to exactly one expert of thousands. At 66.3%, calling this a mixture of experts is a category claim the arithmetic does not support. It is a dense 27B that skips a third of its FFN.

## A partition, not a rebuild

Here is the part I want to be precise about, because it is genuinely elegant.

The shared expert holds 5,033,164,800 parameters and the routed pool holds 12,079,595,520. Their sum is 17,112,760,320, which is exactly `64 layers × 3 matrices × 5120 × 17408` — the parent's entire FFN, to the parameter. The carve is lossless in weight space. What it adds is 64 router matrices at `64 × 5120` each and 64 shared-expert gates at `1 × 5120`: **21,299,200 parameters, 0.079% of the model.**

So the parent's text stack is 26,895,998,464 parameters and the MoE is 26,917,297,664. The transformation is a re-indexing of the same tensors plus a gate. That is a real and clean idea, and it is the reason the technique has a literature — MoEfication, sparse upcycling, LLaMA-MoE-style FFN splitting — none of which the card cites, though the findings log names the lineage correctly.

Which neurons go where is decided by `moe27_plan.py`, and the method is sensible. It streams sixteen real token sequences through the model, layer by layer, computing `|silu(x·Wgate) ⊙ (x·Wup)|` per neuron per token. A neuron's "hotness" is how often it lands in a token's top-5% activation slice. The hottest 5120 become the shared expert; the remaining 12288 go through a balanced k-means on their `gate_proj` rows — the input direction that switches each neuron on — with a capacity-constrained reassignment so every expert holds exactly 192. The cluster centroids become the router.

I checked whether that clustering finds anything. Pulling eight experts' worth of gate rows out of layer 32 by byte range, the mean pairwise cosine *within* an expert is +0.0280 against +0.0118 for a random 192-neuron grouping of the same rows. So the structure is real — 2.4× more coherent than chance — and also very weak in absolute terms: 192 neurons at a mean pairwise cosine of 0.028 are nearly orthogonal to each other. The hot-neuron sets are properly layer-specific, too: the overlap between layer 0's shared set and layer 63's is 1611 of 5120, against a chance expectation of 1506.

<Callout type="note">
The neuron-to-expert assignment, the centroid routers and the hotness plan for the shipped geometry are all in `moe/moe27_plan_gen2.pt` in [logic65/Qwen3.8-Whittle-dev](https://huggingface.co/logic65/Qwen3.8-Whittle-dev). Publishing that file is what makes everything below this line checkable, and it is the single most useful thing in the whole project.
</Callout>

## The fold that makes a fresh carve broken

Now the mechanism, which the card does not mention and the build script explains in four lines of comment.

A dense FFN **sums** the contributions of its neurons. An MoE layer takes a normalised weighted **average** over the k experts it selected. Those are not the same operation, so `moe27_build.py` corrects for it on the way out:

```python
# The shared expert is multiplied by sigmoid(gate), and a zero gate gives exactly
# 0.5, so its down_proj is doubled and the product comes back to the original.
# The routed branch normalises its top-k weights, making it an average of the k
# experts rather than their sum, so their down_proj carries a factor of k. Both
# are exact at uniform weights and the distillation pass refines the rest.
add(out + "shared_expert.down_proj.weight", (d[:, sh].float()*2.0).to(torch.bfloat16))
...
dn.append((d[:, sel].float()*A.topk).to(torch.bfloat16))
```

Read the last clause again. **Both are exact at uniform weights.**

The shared-expert fold is fine: the gate is initialised to zeros, `sigmoid(0) = 0.5`, and the ×2 cancels it exactly. I checked whether training moved it — it did, but only to an absolute maximum of about 0.005 across the layers I sampled, so `sigmoid` is still 0.5 to three decimals and the fold still holds.

The routed fold does not hold. Multiplying every routed `down_proj` by `k = 16` converts an average back into a sum only if the router puts weight exactly 1/16 on each of its sixteen picks. A softmax never does that. Every selected expert enters the residual stream scaled by `k·wⱼ` instead of 1, and the more confident the router, the further from 1 those coefficients are.

<ScaleFold />

This reframes the "gibberish" baseline entirely. A freshly carved model is not broken because the router is untrained and picks nonsense — the router is initialised to cluster centroids, which is a perfectly reasonable guess. It is broken because **the reconstruction identity only holds at the one router configuration that carries no information**. Damage arrives before the question of which experts to pick is even asked. And the repair job is correspondingly different from the one advertised: not teaching the router better taste, but absorbing a scale error the build introduced.

## What "trained only the routers" actually trained

The healing adapter is published, so this needs no inference. `router-heal-adapter/adapter_config.json` lists `modules_to_save` — trained at full rank — as all 64 `mlp.gate` matrices, all 64 `mlp.shared_expert_gate` vectors, and the layernorms. So far so good. It also lists `target_modules` for LoRA at rank 64, alpha 128:

```
["k_proj", "in_proj_b", "q_proj", "in_proj_qkv", "o_proj", "out_proj",
 "up_proj", "gate_proj", "in_proj_z", "in_proj_a", "v_proj", "down_proj"]
```

That is every attention projection, every gated-DeltaNet projection, and — because the routed experts are stored as fused 3-D parameters that LoRA cannot target, while `up_proj`/`gate_proj`/`down_proj` exist only inside `mlp.shared_expert` — **the shared expert's entire FFN, in all 64 layers.**

Summing the adapter's own tensor shapes gives the split:

<Figure
  src="/articles/qwen3-8-whittle-moe/fig3.png"
  alt="Two panels. The top panel shows the 26.92 billion parameter model as a long grey bar with a tiny amber sliver at the left labelled everything the router-heal adapter trains, 337.9M equals 1.255 percent of the model, and a smaller green sliver labelled the 64 routers alone, 20.97M equals 0.078 percent. The bottom panel shows the two published adapters on a common scale: the router-heal adapter totals 337.9M, made of 190.1M of LoRA on attention and DeltaNet, 125.8M of LoRA on the shared expert, and 21.0M of routers, which is 6.2 percent of what it trains; the v2 anti-loop adapter totals 1030.3M, made of 629.1M of full-rank shared-expert FFN in layers 56 to 63, 380.2M of LoRA on attention and DeltaNet, and the same 21.0M of routers, which is 2.0 percent."
  caption="Tensor-by-tensor sums of the two published adapter_model.safetensors files, grouped by the module each LoRA pair or full-rank tensor targets. (Own measurement from the published adapters.)"
/>

The routers are 20,971,520 parameters — 6.2% of what the "router-heal" adapter moves, and 0.078% of the model. The other 93.8% is LoRA on the attention stack, the DeltaNet stack, and the shared expert's feed-forward weights. The neurons were not frozen. The *routed* neurons were frozen, which is 44.9% of the model; the 5.03B-parameter shared expert, sitting in the same FFN, was trained through a rank-64 adapter in every layer.

The anti-loop round that produced "69% → 8%" is further still from the story. Its adapter trains 1,030,291,456 parameters, 61% of which is the shared-expert FFN of layers 56–63 at **full rank**. The routers are 2.0% of it.

And the author knows all this. From `WHITTLE_FINDINGS.md`, under "Architecture notes worth keeping":

> Training routers alone is harmful. An earlier round that trained only routers doubled the number of facts the model answered as Unknown. Routers are co-adapted with their experts and must not be moved independently.

The research log says router-only training was tried and made the model worse. The model card, the GGUF card and the quantization notes all say router-only training is what saved it.

## The routers moved 4.5 degrees

The claim is checkable because `moe27_plan_gen2.pt` contains the exact centroid matrix each router was born as. It is a torch pickle over a zip archive, so reading it needs no torch — an unpickler that understands `_rebuild_tensor_v2` and a numpy view over the storage blobs is enough. Its `layers[L]["router"]` is `[64, 5120]` float32 with every row at unit norm, which is what `torch.nn.functional.normalize` guarantees.

Against that, every `mlp.gate.weight` in the shipped v2.1 checkpoint:

<Figure
  src="/articles/qwen3-8-whittle-moe/fig2.png"
  alt="Left panel: a scatter of 4096 points, one per expert per layer, showing the angle between each shipped router row and the k-means centroid it was initialised as, against layer index. Points cluster tightly between 3 and 5 degrees for the first forty layers and spread to between 4 and 10 degrees in the last fifteen, with a green per-layer mean line rising gently from 3.3 to about 6.8 degrees and a dashed horizontal line at the overall mean of 4.50 degrees. Right panel: a histogram of all 4096 angles, sharply peaked between 3 and 5 degrees, with a red dashed line at 10 degrees marking that 99.3 percent of rows fall below it."
  caption="Shipped v2.1 mlp.gate.weight against the untrained centroid router in moe27_plan_gen2.pt, all 64 layers and all 64 experts. Row norms 1.00062 against an initialisation of exactly 1.0. (Own measurement.)"
/>

Mean 4.50°, median 4.02°, p95 7.39°, worst single row 15.28°. The drift is monotone with depth — 3.41° averaged over layers 0–15, 6.28° over layers 48–63 — which is exactly where the findings log independently located the degeneration behaviour.

Almost all of that belongs to the healing round, which is the point. Across nine layers sampled through the stack, `v1/`'s routers already sit 4.25° from the centroids and v2.1's sit 4.74°, so the two later training rounds together rotate them by 1.89°. The round that is supposed to have taken a model from gibberish to conversation moved its routers about four degrees and left their norms at 1.0.

You can argue about how much a 4.5° rotation of a decision boundary is worth. What you cannot argue is that it carried a model from 4/39 to 27/39 while 316.9M other parameters were also being trained. The parsimonious reading is the one the author's own findings log already reached: **the routers are co-adapted with their experts, and what repaired the model was the LoRA on everything else.**

## Where the 39 comes from

A denominator of 39 is not a benchmark, so I went looking for it. It is `moe/q38_battery2.py` in the dev repository, and its docstring is honest about what it is:

> Broad 40-prompt robustness battery, auto-scored by expected substring. … Scores are rough (substring in 40 tokens) but identical across variants, so the DELTA between variants is meaningful even where the absolute bar is crude.

Forty items. One of them — `story2`, "The old lighthouse keeper climbed the stairs one last time," — carries an empty expectation list and is scored `----` rather than pass or fail. That is where the 39 comes from: `scored` counts only the items with an expectation.

The mechanics: a raw `/completion` call at `temperature 0.0, top_k 1, top_p 1.0, seed 7`, 40 tokens, marked pass if any expected string appears anywhere in the output, case-sensitively. Twenty facts, eight arithmetic items, eight code items, three commonsense. Some of the bars are low — `sql_count` passes on the substring `COUNT` or `count`, `html_link` on `<a` or `href`, and `c_loop` passes if the completion contains either the character `0` or the character `9`.

None of that makes it useless. A fixed, greedy, seeded instrument applied identically to every variant is a reasonable way to detect a model that has stopped working, and the author says so explicitly. What it cannot do is resolve small differences. On 39 binary items, the Wilson 95% interval around 28/39 runs from 21.9 to 32.5 items. The card's own table moves from 28 to 27 to 28 across two training rounds; a two-sided Fisher exact test on 28/39 against 27/39 returns p = 1.00. The card is straight about this too — it calls the change "inside the measured noise floor" — and then the headline elsewhere reads "4/39 → 28/39" anyway.

## The loop rate is a failure rate, and the sampler is not the trick

My first suspicion on reading "loops 69% → 8%" was the obvious one: repetition rate is enormously sensitive to sampling, and a `repetition_penalty` appearing between two releases can move it that far on its own. `loop_test.py` closes that off completely. The sampler is hard-coded in the harness at `temperature 0.7, top_p 0.8, top_k 20` with fixed seeds `(1, 2, 3)`, it matches `generation_config.json` exactly, and there is no repetition penalty anywhere in the file. Same prompts, same seeds, same settings across every release. Credit where it is due — that is more rigour than most model cards manage.

Two things do need saying about it.

The first is what "loop rate" means. A generation is FAILED if `rep4 > 0.15` **or** duplicate lines exceed 0.20 **or** repeated line-openers exceed 0.40 **or** the answer is under a word floor. It is a composite failure rate, not a repetition rate, and the floor is load-bearing: the findings log records two separate occasions when a near-zero repetition score turned out to be a model that had learned to answer "Sure" and stop. Adding a metric that gets worse when the fix overshoots is the correct instinct, and the log states the general rule it learned from it.

The second is resolution.

<GateResolution />

The single-turn section is 12 prompts × 3 seeds = 36 generations, so 69% is 25 failures and 8% is 3. The gap between v2's 11% and v2.1's 8% is one generation. The structured section holds 18, the multi-turn section 28, the late-turn section 12 — and 12 generations can only report 0%, 8%, 17%, 25%, 33%, 42%, 50%, 58%. The card's late-turn baseline of **56% is not on that list**, so it was measured on a differently shaped run than the harness the repository ships. The same is true of the "~75%" structured baseline, which comes from a 12-generation cell in the k-sweep rather than the 18-generation gate section its 22% successor was measured on.

There is also a stale number. The card's results table gives v2.1 a 22% structured-output failure rate; the "Honest limitations" section three paragraphs above still says "39 percent of SQL, HTML and markdown table generations degenerate", which is the v2 figure. The prose was not updated with the table.

## The control that is missing

Every number on this card compares the model to an earlier version of itself. The natural control — `Qwen/Qwen3.8-27B`, the dense model all of these weights came out of, which is Apache-2.0 and one `llama-server` invocation away — is not measured on the battery, not measured on the loop gate, and not mentioned as a baseline anywhere in the repository.

That matters because of what the "before" model is. A freshly split MoE with a k-fold scale error is an artefact of the splitting procedure. Measuring recovery against it measures how much damage the method did, not how much capability the method added.

The comparison the project does publish is, if anything, worse for it. `Qwen3.8-Whittle-dev`'s README carries a table it calls "The ladder": fifteen shapes carved from the same parent, "all measured against the same 80-probe knowledge atlas and the same 39-prompt generation battery". Top of the ladder:

| shape | params | battery | how it was made |
|---|---|---|---|
| Whittle-16B v1 heal | 16.8B | **36/39** | 44L cut, 25% width prune, 3h QLoRA |
| 48L cut | 20.8B | 35/39 | 16 layers dropped by block pricing, no training |
| restored 18.3B | 18.3B | 34/39 | 44L trunk with blocks 8-11 spliced back |
| *Whittle MoE 27B v2.1* | *27B / 17.86B active* | *28/39* | *this article* |

<WhichBaseline />

A 16.8B model from the same author — 44 of 64 layers kept, every MLP cut to 75% width, three hours of QLoRA to repair it —, on the same instrument, scores eight items higher than the 27B MoE — and unlike the 27-versus-28 wobble, that difference clears significance (Fisher exact, p = 0.036). The per-item run is in the repository as `research/q38_battery2_healed.json`. On the author's own evidence, the cheapest way to get a good small model out of Qwen3.8-27B is to cut layers off it, not to turn it into a mixture of experts.

## The 192-wide tax

One more consequence of the geometry, and this one is a straightforward engineering cost that shows up in every download.

`ffn_down_exps` has a reduction dimension of 192. llama.cpp's k-quants operate on blocks of 256 weights, and 192 is not divisible by 256, so that tensor cannot be k-quantized at all. Parsing the GGUF headers by range request confirms what the fallback picks:

| build | `ffn_gate_exps` / `ffn_up_exps` | `ffn_down_exps` |
|---|---|---|
| Q4_K_M | Q4_K, 4.50 bpw | Q8_0 on 32 layers, **Q5_0 on 32 layers — 7.00 bpw** |
| DQ4_K_XL | Q4_K, 4.50 bpw | Q8_0 / Q5_0, 7.00 bpw |
| DQ3_K_XL | Q3_K, 3.44 bpw | Q5_0 / Q5_1, ~5.5 bpw |

In the Q4_K_M build, one tensor class holding 15% of the parameters takes 3.52 GB of a 17.36 GB file — 20% of the download, at 7 bits per weight in a model whose overall rate is 5.16. Reconstructing the file size from the header's tensor types and dimensions gives 17.36 GB against a published blob of 17.37 GB, so the accounting is sound. The card describes the mechanism correctly and names the wrong fallback type: it says `q5_1`, and the shipped files use `Q5_0` and `Q8_0`.

Two smaller notes from the same headers. The card presents "the router stays at F16 in every DQ tier" as a property of its dynamic-quant recipe; `ffn_gate_inp` is F32 in the plain Q4_K_M build too, which is stock llama.cpp behaviour, not a DQ choice. And the main repository is tagged `gguf` and tells you to download `Whittle-MoE-27B-A18B-v2.1-Q4_K_M.gguf`, which is not in it — the GGUF files live in a separate repository that has more downloads than this one.

## What is real

I have spent a lot of words on what does not survive checking, so let me be equally specific about what does. This is a self-funded, one-person project whose card opens by saying the compute budget is exhausted, and whose limitations section says "Evaluated by one person on a small harness. Treat every number as a workshop measurement, not a benchmark." Held to that standard rather than the headline's, a lot of it is good.

**The partition is exact and the plan is published.** `64 × 192 + 5120 = 17408`, verified in the shapes. The neuron assignment, the centroid routers and the hotness plan are all in a downloadable file, which is why this article could check anything at all. Most model cards make claims you cannot test.

**The eval hygiene is better than the model.** A public, standard-library, fixed-seed, fixed-sampler harness that writes per-generation scores to disk so you can inspect individual failures. A length floor added specifically because the author was twice fooled by a near-zero repetition score that meant the model had gone quiet. And an audit, run *before* generating training data, that found 11 of 12 gate prompts sitting inside the planned training set and rebuilt the set disjoint from the gate. That last one is a discipline plenty of funded labs skip.

**The negative results are published and they are load-bearing.** Five training runs mapped in one table, including phase C, which masked the EOS token globally and made looping *worse* than doing nothing (90% against a 69% baseline), and phase E, whose best-in-campaign 2% loop rate turned out to be a model with a multi-turn median of one word. A neuron-ablation attempt following [arXiv:2606.13705](https://arxiv.org/abs/2606.13705) that returned two nulls, with the confound named and the causal check declared unrunnable on the available hardware rather than fudged.

**The EOS-bias dose-response curve is the best experiment in the project.** Sweeping a logit bias on the end-of-turn token at inference, on the released model, 12 generations per point: at bias 0 the model loops in 11 of 12 generations with a median of 278 words; at +7.5 it stops looping entirely and answers in a median of 14 words; at +9, 7 words. The dial slides from all-loops to all-silence without passing through a healthy point. That upgrades "repetition and stopping are one axis" from a training anecdote to a property of the weights, and it is the kind of cheap, decisive experiment that ought to be standard.

**And the k-sweep kills the convenient hypothesis.** When the community suggested the looping was capacity starvation at top-16, a positive result would have handed the project a free config-level fix — recommend k = 24 and ship. The author served the same quantized weights three times with only `expert_used_count` overridden, verified the change was real from the throughput signature (18.8, 17.2, 15.9 tok/s at k = 16, 24, 32), and reported that k = 24 changes nothing row for row while k = 32 is worse. Publishing the negative, with the confound stated up front, is the thing.

## The ledger

**What is genuinely new.** Nothing methodological — this is MoEfication with a hotness-selected shared expert, and the technique is a decade of literature deep. What is new is the *artefact*: a complete, downloadable record of one post-hoc MoE conversion, plan file included, with the failed runs attached. That has real value to anyone who wants to try this, and almost none of it is in the model card.

**What is convergent.** The finding underneath all the training rounds — that repetition and premature stopping are one axis, and the missing signal is *when* an answer is complete rather than whether to stop — is not specific to MoEs or to this model. It is a statement about what teacher-forced distillation corpora do and do not contain, and it lines up with what the [looped-model](/articles/looped-models-done-right) and pruning literature keeps rediscovering: [Bonsai](/articles/bonsai-27b) found capability falling unevenly under extreme quantization, and this project finds the same thing under structural surgery. Damage is not scalar.

**What I would not repeat.** Calling a 66.3%-active model a mixture of experts. Reporting recovery against a baseline your own method created. And writing "trained only the routers" on a card whose adapter file, in the same repository, shows the routers are 6.2% of what moved — especially when your own findings log says router-only training was tried and was harmful.

**What I would watch.** Whether anyone runs the obvious experiment. `Qwen/Qwen3.8-27B` is Apache-2.0, `loop_test.py` is 174 lines of standard library, and `q38_battery2.py` is 110 more. Half an hour on a rented card produces the two numbers this entire campaign is missing, and they would settle whether a post-hoc MoE carved out of a dense model is worth anything at all relative to the model it came from. My prior, from the ladder in the author's own dev README, is that it is not — and I would rather be shown the measurement than keep the prior.

The sentence I keep coming back to is the one in `moe27_build.py`: *both are exact at uniform weights and the distillation pass refines the rest*. Everything downstream — the gibberish, the healing, the looping, five training runs and a lot of one person's evenings — is the cost of "refines the rest" doing a lot of work in that sentence.