# Husky's 4.5× over MLX: the method is published, and it is speculative decoding

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/husky-msi
> date: 2026-09-22
> tags: explainer, llm, on-device-inference, on-device, benchmarks, inference
Two days ago [a 50× turned out to be 1.39×](/articles/laya-mlx), and the reason
was not dishonesty — it was a denominator. Somebody divided a local warm
inference by a hosted HTTPS round trip and the Atlantic did the work.

[Husky](https://husky.underdog.ai) makes a claim of the same shape, on the same
kind of hardware, in the same week: *a model-specific inference engine up to 4.5×
faster than Apple's MLX*, running Underdog's Woof at *up to 730 tokens a second
on a MacBook*. Same-machine, same-weights, so it is checkable in principle.

It turns out to be checkable in practice, which I did not expect. The page ships
a sixteen-row table with both engines' medians, a method paragraph naming the
chip, the OS build, both library versions and the run protocol, and a `Cite`
block. That is more denominator than most benchmark posts carry, so the
interesting question is no longer *is this a real number* but *what is it a
number about*.

Short answer: **it is a speculative-decoding result, the page says so, and the
baseline was not given the speculative decoding it already has.**

<Callout type="note">
There is no Apple silicon in the machine I write these on, so **nothing here was
executed**. What I could check independently is the model — Woof is public on the
Hub and its file sizes, layer counts and quantisation reproduce from its config —
and the arithmetic that connects it to the published table. Every latency and
throughput figure below is Reported, read off husky.underdog.ai on 22 September
2026.
</Callout>

## What is being made faster

Before the ratio, the denominator's denominator: what Woof is.

<ModelCard repo="ConwayResearch/Underdog-Woof-4B-1.1" note="The model both engines run, and the one place this story has an artefact I can inspect. Architecture Qwen3_5ForConditionalGeneration — so Woof is a Qwen3.5-4B derivative, not a from-scratch family. 4-bit affine MLX quantisation at group size 64. Apache-2.0." />

The config settles several things the blog post states without support, and they
all hold:

```bash
$ curl -s https://huggingface.co/ConwayResearch/Underdog-Woof-4B-1.1/raw/main/config.json \
  | python3 -c "import sys,json,collections as c; t=json.load(sys.stdin)['text_config']; \
      print(len(t['layer_types']), c.Counter(t['layer_types']))"
32 Counter({'linear_attention': 24, 'full_attention': 8})
```

Twenty-four linear-attention layers, which is exactly the number Husky's *resident
grid for the recurrence* section says it runs on a persistent threadgroup grid,
and 32 layers total, which is the number its whole-step-megakernel paragraph
names. Hidden size 2560, intermediate 9216, `full_attention_interval` 4, vocab
248,320. The quantisation is 4-bit affine at group size 64, so each weight costs
4 bits plus an fp16 scale and bias per group of 64 — 4.5 bits effective:

$$
4{,}205{,}751{,}296 \times \tfrac{1}{2} \;+\; \frac{4{,}205{,}751{,}296}{64}\times 4
= 2.3657\ \text{GB}
$$

against a published `model.safetensors` of **2,367,237,149 bytes**. That is 0.06%
apart, which for a formula with no fitted terms means the layout is exactly what
the config says. Husky's *"all 2.4 GB of Woof"* is the real file, rounded up by
33 MB.

That matters because the whole argument turns on it.

## The bus, and why a kernel cannot win this

At batch 1 a decode step reads every weight. There is no way around it: the
matmuls are tall-skinny, arithmetic intensity is about one, and the step is a
streaming read with a little multiply attached. Husky states the consequence in
its own second paragraph, and it is the most useful sentence on the page:

> Producing a token means streaming all 2.4 GB of Woof from unified memory
> through the 40 GPU cores, about 6 ms on the M5 Max, and **no engine reads
> faster than the bus**. What Husky changes is how many tokens come out of each
> read.

Take that seriously and the headline becomes arithmetic. MLX's median across the
sixteen rows is 159 tok/s — 6.29 ms a token, an effective 377 GB/s on a 2.367 GB
read. Husky puts the floor for a plain streaming read at 4.5 ms, which is 526
GB/s. So the **entire** headroom available to any engine that emits one token per
read is

$$
\frac{6.29\ \text{ms}}{4.5\ \text{ms}} = 1.40\times
$$

and no amount of kernel work can go past it.

<ReadBudget />

Husky, without its draft model, has a median gain of **1.125×** and a best prose
row of 1.21×. Both sit under the line, where they have to. The page is candid
about where that comes from and it is not a kernel: *"Every step used to cost
almost half a millisecond of encoding and waiting between GPU commands; Husky now
encodes the next step while the current one runs."* Half a millisecond off 6.3 is
8%, and the measured prose gains are 2% to 21%. It is a host-latency fix and it
reads like one.

Everything above the line is **tokens per read**, which is speculative decoding.
Two kinds of it:

- **Prompt lookup.** When the last few tokens Husky wrote also appear in the
  prompt, it proposes the seven that followed there and Woof keeps the ones it
  agrees with — *"typically five or six"*. No draft model, no training. This is
  why an edit flies.
- **The Flash draft.** A single small layer reading Woof's hidden states from five
  of its layers, self-distilled on Woof's replies to 139,000 conversations, about
  two hours on one B200. It proposes seven tokens; Woof still verifies every one,
  so the output is identical.

Both ride the same eight-row verify step, which Husky prices at **1.5× the cost
of one row**. That gives clean ceilings from the published acceptance rates —
5.4 accepted on edits is 5.4 / 1.5 = 3.60×, 2.0 on prose is 1.33×, 5.0 on code
from scratch is 3.33× — and the measured rows land just under each, which is what
a real system that also pays host costs looks like. The one place the measurement
*beats* its own ceiling is the function-edit row at 3.77×, which is what you get
when acceptance on a particular file runs above the 5.4 average.

## Sixteen rows, and which one the headline is

<TaskLadder />

Read down the chart rather than at the top of it.

**MLX is flat.** 151 to 164 tok/s across every task in the suite — a 9% spread
over prompts from 38 to 857 tokens. That is exactly what a bandwidth-bound
single-token decoder should do, and it is a quiet validation of the whole setup:
a baseline that moved around with the prompt would mean something else was wrong.

**Husky alone is a tie on prose and a rout on edits.** Seven of the sixteen rows
are between 1.02× and 1.09×. Four are between 2.91× and 3.89×, and all four are
edits — function edit, JSON field, SQL column rename, typo fix — where the reply
repeats long runs of the prompt verbatim and prompt lookup has something to copy.

**Flash on beats MLX everywhere, by 1.30× to 4.48×, median 1.76×.**

**The 4.5× is the first row.** Function edit, 368 tokens in, Flash on, 730 against
163. It is a real median of three runs. It is also the single most favourable cell
in a sixteen-cell table, on the task class the engine was explicitly built for —
Husky says so: *"Underdog asks Woof for edits like these all day, so this is not a
corner case, it is the common one."* Fair enough for their product. It is still a
maximum, and "up to" is doing the work in the title.

The first-token numbers are the least-quoted and hold up best. 137–177 ms for MLX
against 29–39 ms for Husky on a continued chat, a median **4.24× sooner**, on a
comparison the page sets up to disfavour itself: both engines run from their
caches, and it separately reports that MLX's plain `generate` call — the one
without a prompt cache, which is what a naive caller gets — takes 145 to 295 ms.
It quotes the cached number in the headline. That is the right choice and almost
nobody makes it.

<Video
  src="/articles/husky-msi/race"
  poster="/articles/husky-msi/race-poster.jpg"
  alt="A four-second screen animation. First a panel headed 'One more message. MLX rereads the chat. Husky answers from where it left off' shows two time-to-first-word bars filling: MLX reaching 157 milliseconds and Husky 34. Then a side-by-side generation replay headed 'real time', with a running clock, both engines answering 'Fix the spelling and grammar in this paragraph'. MLX's pane fills with text at a counter reading about 148 tokens a second; Husky's pane, marked Flash on, fills much faster at 537 tokens a second and stops with 'done in 0.53 s' while the clock reads 1.70 seconds and MLX is still writing."
  caption="Underdog's own launch animation, clipped to the two evidence-bearing beats and otherwise unaltered. It is a replay at the speed the stream arrived, not a screen recording — the counters are animated from the same measurements as the table. The prompt is the 'Fix typos in a paragraph' row, where the table reads MLX 159 / Husky 463 / Flash on 535. The internal arithmetic is consistent: a reply Husky finishes in 0.53 s at 537 tok/s is about 285 tokens, which at 148 tok/s takes 1.93 s, and the clip ends at 1.70 s with MLX unfinished. (husky.underdog.ai, husky-launch-v15.mp4, 20 September 2026.)"
/>

## The baseline that was not run

Here is the one thing I would change, and it is the same shape as the question
worth asking of any framework comparison: was the baseline given its own best
options?

Mostly yes. Both engines get the same weight file — *"Husky repacks it without
changing a value, and greedy output is compared token by token against MLX after
every change"*. The MLX version is **mlx-lm 0.31.3 on MLX 0.32.2**, and 0.31.3 is
the current release on PyPI, so this is not a stale baseline. MLX gets its prompt
cache in the first-token comparison rather than the naive path. Runs that
disagreed with their own repeats were discarded. Order was alternated, a canary
ran first, repeats had to agree within 12%.

But the headline ratio is a speculative-decoding ratio, and **mlx-lm has shipped
speculative decoding the whole time**:

```bash
$ curl -s https://raw.githubusercontent.com/ml-explore/mlx-lm/main/mlx_lm/generate.py \
  | grep -n "draft-model\|num-draft-tokens\|def speculative"
210:        "--draft-model",
216:        "--num-draft-tokens",
476:def speculative_generate_step(
```

So the comparison in the table is *Husky with an eight-token verify step* against
*MLX without one*. That is not a small asymmetry — it is the entire mechanism
above 1.40×.

Two honest caveats on my own complaint, because they are real:

1. **Flash is not portable to MLX.** It is not a standalone language model; it
   reads Woof's own hidden states from five of its layers. You cannot hand it to
   `--draft-model`. A fair MLX speculative baseline would need somebody to train
   or pick a small standalone draft for Woof, which is work Underdog did for their
   engine and nobody has done for MLX.
2. **mlx-lm has no prompt-lookup decoding.** I grepped for it; there are no hits
   for lookup or n-gram anywhere in `generate.py`. So the 2.9–3.9× on the four
   edit rows is a capability MLX genuinely does not have, draft model or not, and
   that part of the gap is not a configuration choice.

Which leaves a precise version of the claim. *Husky's engine is about 1.1× MLX on
prose. Husky's prompt lookup is 2.9–3.9× on edits and MLX cannot do it at all.
Husky's trained draft adds 1.3–1.7× on prose, and MLX has a slot for a draft that
this benchmark left empty.* All three are worth having. Only the first is a claim
about kernels.

<Callout type="tip">
The most interesting paragraph on the page is the one reporting a negative result,
and it is the kind of thing that only appears when somebody is writing for
engineers. Husky built the first piece of a whole-step megakernel — a layer's
entire MLP as one persistent dispatch with a grid barrier — and *"measured it
honestly: identical output, and no faster. On this GPU, dispatches placed back to
back in one encoder already run with almost no gap."* They then say where it
would pay: overlapping the next stage's weight loads with the current stage's
tail, which is the only lever left when you are already at the bus. A page that
prints its own null result is a page I trust more on the rest.
</Callout>

## What 730 tokens a second is, exactly

Since it is the number that will travel, here it is with every qualifier attached:

- **decode only.** Tokens per second while writing the reply, after the prompt is
  processed. Prefill is a separate figure (5,150 tok/s at the caller, 5,460 inside
  the engine) and first-token latency is a third.
- **one row of sixteen.** The function edit, 368 tokens in. The median row with
  Flash on is 279 tok/s; the slowest is 210.
- **Flash on.** Without the trained draft, that row is 614 and the slowest row is
  164 — three tokens a second above MLX.
- **the task where the reply copies the prompt**, so most of those tokens are
  verified from prompt lookup rather than generated.
- **batch 1**, which the page does not state and does not need to: a single-user
  Mac app, and 159 tok/s on a 2.367 GB model is 377 GB/s, which is only a
  single-stream number. At any real batch the weight read amortises and both
  engines' tokens per second go up a lot.
- **M5 Max, 40-core GPU, 128 GB, macOS 26.5.1.** On a base M5 or an M-series Air
  the bus is narrower and every figure in the table moves together.

None of that makes 730 wrong. It makes it one cell.

## What is not checkable

The engine. Husky is not open source, has no repository, and runs only inside the
Underdog app — *"Husky is live in Underdog on the Mac today, with no setup."* The
method section says *"the receipts are kept"*; they are not published.

So the reproducibility is one-sided in a specific way. Anyone with a Mac can
reproduce the **baseline** exactly: `pip install mlx-lm==0.31.3`, pull
`ConwayResearch/Underdog-Woof-4B-1.1`, run the sixteen prompts, and check whether
MLX really lands at 151–164 tok/s and 137–177 ms to first token on their machine.
If it does, the denominator is sound and the only unverified column is Husky's
own. If it does not, nothing else matters.

That is a much better position than the 50× was in. It is still short of the bar
[Laya on Apple silicon](/articles/laya-mlx) set two days earlier, which committed
every timing sample, every rejected run and every export manifest to a public
tree — including a run it threw away whole because one power-sensor reading came
back as 38 kilowatts.

## What I would actually ship

- **If your replies copy your prompts, build prompt lookup first.** It is the
  cheapest large win in this entire post: no draft model, no training, no extra
  weights, and 2.9× to 3.9× on the four edit tasks. It costs an n-gram index over
  the prompt and an eight-row verify step. Every assistant that rewrites text the
  user already sent is leaving this on the floor.
- **Do not expect kernels to move a batch-1 decoder.** Measure your model's weight
  bytes, divide by your machine's achievable bandwidth, and that is your token
  time. If you are within 40% of it, the remaining work is speculation or batching,
  not tiling.
- **Enable speculative decoding on the baseline before you publish a ratio
  against it.** `mlx_lm.generate --draft-model` exists. vLLM and SGLang have their
  own. A speculative engine measured against a non-speculative one is comparing
  two methods, not two implementations.
- **Quote the median, not the maximum.** Flash on is 1.76× MLX across sixteen
  representative tasks and that is a good result for a from-scratch
  engine. "Up to 4.5×" is the same data reported less usefully.
- **Watch what happens to MSI on CUDA.** The interesting bet in the post is the
  last line of *What's next* — the same compile-time-shapes approach written for
  NVIDIA. On a datacentre GPU the arithmetic changes completely: batching is
  available, the bus is faster, and general engines are far more tuned. That is
  where model-specific inference either earns its name or does not.

<ChangeMyMind>

<Falsifier claim="Woof is a Qwen3.5-4B derivative, 4-bit affine at group size 64, and its weight file is 2.367 GB.">
`curl -s https://huggingface.co/ConwayResearch/Underdog-Woof-4B-1.1/raw/main/config.json` prints `"architectures": ["Qwen3_5ForConditionalGeneration"]` and `"quantization": {"group_size": 64, "bits": 4, "mode": "affine"}`; the Hub API gives `model.safetensors` at 2,367,237,149 bytes and 4,205,751,296 parameters. If a later revision reshapes or requantises the checkpoint, or if the engine ships a different repack whose byte count differs from the Hub file, the 2.367 GB per read is wrong and the 1.40× bus ceiling moves with it.
</Falsifier>

<Falsifier claim="The memory bus caps any one-token-per-read engine at about 1.40x over MLX on this machine.">
It is one division: MLX's 159 tok/s median is 6.29 ms a token, and Husky's own stated plain-streaming-read floor for the same bytes is 4.5 ms. Measure the floor yourself — a bare `mx.eval` over a 2.37 GB array on an M5 Max — and if it comes back under 4.5 ms the ceiling rises, or if MLX is leaving more than 40% on the table for a reason other than bandwidth, the whole framing is wrong. The strong version: run Husky with Flash off and prompt lookup disabled on a task whose reply shares nothing with its prompt. It should not beat 1.40x.
</Falsifier>

<Falsifier claim="Husky without its draft model is a median 1.125x over MLX, and a tie on seven of sixteen tasks.">
Sum the published Husky column against the published MLX column, row by row. I get 1.02, 1.04, 1.04, 1.07, 1.07, 1.08, 1.09, 1.10, 1.15, 1.21, 1.27, 1.82, 2.91, 3.08, 3.77, 3.89 — median 1.125. If the table is revised, or if I have mapped the Husky column to the wrong condition and it already includes a draft, the ladder above is wrong about which mechanism does the work.
</Falsifier>

<Falsifier claim="The baseline is mlx-lm without its own speculative decoding.">
`mlx_lm/generate.py` defines `speculative_generate_step` and the CLI takes `--draft-model` and `--num-draft-tokens`; the method paragraph names no draft on the MLX side. If Underdog did run MLX with a draft model and simply did not say so, the asymmetry disappears and I owe them a correction. The test that settles it for everyone: convert any small Qwen3.5-class checkpoint to a 4-bit MLX draft, run `mlx_lm.generate --draft-model` over the same sixteen prompts, and publish the column. If MLX plus a draft lands near Flash on, most of the 1.76x median was the method rather than the engine.
</Falsifier>

<Falsifier claim="mlx-lm has no prompt-lookup decoding, so the edit speedup is a capability MLX lacks.">
`grep -i "lookup\|ngram" mlx_lm/generate.py` returns nothing on main as of 22 September 2026. If it lands, or if it is reachable through a logits processor or a custom sampler in a way I missed, then even the four edit rows become a configuration choice rather than a capability gap, and the honest ratio for Husky drops to something near the bus ceiling on every row.
</Falsifier>

<Falsifier claim="730 tok/s is decode-only, Flash on, on the single most favourable of sixteen tasks.">
It is the top-left cell of Table 1 — Function edit, 368 tokens in, Flash-on column — and the table's own header says the column is tokens per second while writing the reply. Second place is 672. If a later run publishes a higher median or a distribution rather than a maximum, quote that instead. If somebody measures 730 sustained across a mixed workload, the headline stops being a maximum and becomes a throughput, which is a different and better claim.
</Falsifier>

</ChangeMyMind>

---

*Nothing here was executed. There is no Apple silicon in the machine this was written on, so every latency and throughput figure is Reported: read off [husky.underdog.ai](https://husky.underdog.ai) on 22 September 2026, by Sigil Wen, Underdog / Conway Research, published 20 September 2026. What I checked independently is the model — [`ConwayResearch/Underdog-Woof-4B-1.1`](https://huggingface.co/ConwayResearch/Underdog-Woof-4B-1.1), Apache-2.0, through the Hugging Face API and its raw `config.json` — and the baseline's capabilities, from [`ml-explore/mlx-lm`](https://github.com/ml-explore/mlx-lm) on `main` and the PyPI release index. Every ratio, median and byte count in this piece is arithmetic over those two sources and is reproducible from them. The clip is Underdog's own launch animation, trimmed to its two evidence-bearing beats and otherwise unaltered. There is no figure here because there is none to take: the launch page publishes its numbers as one HTML table and draws its three charts in the browser from that same table, so there is no image to mirror, and the only image file it serves is `icon.png`, the product mark. The model card carries no figure either — no chart, no diagram, nothing but the Hub's own avatar — and Underdog publishes no paper or post beyond the page. All sixteen rows of that table are transcribed unaltered into the ladder above instead. Companion piece: [Laya on Apple silicon](/articles/laya-mlx), for the same question asked of a different engine in the same week, and for what a fully published benchmark tree looks like.*