Husky's 4.5× over MLX: the method is published, and it is speculative decoding

Two days ago a 50× turned out to be 1.39×, and the reason was not dishonesty — it was a denominator. Somebody divided a local warm inference by a hosted HTTPS round trip and the Atlantic did the work.

Husky makes a claim of the same shape, on the same kind of hardware, in the same week: a model-specific inference engine up to 4.5× faster than Apple's MLX, running Underdog's Woof at up to 730 tokens a second on a MacBook. Same-machine, same-weights, so it is checkable in principle.

It turns out to be checkable in practice, which I did not expect. The page ships a sixteen-row table with both engines' medians, a method paragraph naming the chip, the OS build, both library versions and the run protocol, and a Cite block. That is more denominator than most benchmark posts carry, so the interesting question is no longer is this a real number but what is it a number about.

Short answer: it is a speculative-decoding result, the page says so, and the baseline was not given the speculative decoding it already has.

What is being made faster

Before the ratio, the denominator's denominator: what Woof is.

The config settles several things the blog post states without support, and they all hold:

$ curl -s https://huggingface.co/ConwayResearch/Underdog-Woof-4B-1.1/raw/main/config.json \
  | python3 -c "import sys,json,collections as c; t=json.load(sys.stdin)['text_config']; \
      print(len(t['layer_types']), c.Counter(t['layer_types']))"
32 Counter({'linear_attention': 24, 'full_attention': 8})

Twenty-four linear-attention layers, which is exactly the number Husky's resident grid for the recurrence section says it runs on a persistent threadgroup grid, and 32 layers total, which is the number its whole-step-megakernel paragraph names. Hidden size 2560, intermediate 9216, full_attention_interval 4, vocab 248,320. The quantisation is 4-bit affine at group size 64, so each weight costs 4 bits plus an fp16 scale and bias per group of 64 — 4.5 bits effective:

against a published model.safetensors of 2,367,237,149 bytes. That is 0.06% apart, which for a formula with no fitted terms means the layout is exactly what the config says. Husky's "all 2.4 GB of Woof" is the real file, rounded up by 33 MB.

That matters because the whole argument turns on it.

The bus, and why a kernel cannot win this

At batch 1 a decode step reads every weight. There is no way around it: the matmuls are tall-skinny, arithmetic intensity is about one, and the step is a streaming read with a little multiply attached. Husky states the consequence in its own second paragraph, and it is the most useful sentence on the page:

Producing a token means streaming all 2.4 GB of Woof from unified memory through the 40 GPU cores, about 6 ms on the M5 Max, and no engine reads faster than the bus. What Husky changes is how many tokens come out of each read.

Take that seriously and the headline becomes arithmetic. MLX's median across the sixteen rows is 159 tok/s — 6.29 ms a token, an effective 377 GB/s on a 2.367 GB read. Husky puts the floor for a plain streaming read at 4.5 ms, which is 526 GB/s. So the entire headroom available to any engine that emits one token per read is

and no amount of kernel work can go past it.

Husky, without its draft model, has a median gain of 1.125× and a best prose row of 1.21×. Both sit under the line, where they have to. The page is candid about where that comes from and it is not a kernel: "Every step used to cost almost half a millisecond of encoding and waiting between GPU commands; Husky now encodes the next step while the current one runs." Half a millisecond off 6.3 is 8%, and the measured prose gains are 2% to 21%. It is a host-latency fix and it reads like one.

Everything above the line is tokens per read, which is speculative decoding. Two kinds of it:

• Prompt lookup. When the last few tokens Husky wrote also appear in the prompt, it proposes the seven that followed there and Woof keeps the ones it agrees with — "typically five or six". No draft model, no training. This is why an edit flies.
• The Flash draft. A single small layer reading Woof's hidden states from five of its layers, self-distilled on Woof's replies to 139,000 conversations, about two hours on one B200. It proposes seven tokens; Woof still verifies every one, so the output is identical.

Both ride the same eight-row verify step, which Husky prices at 1.5× the cost of one row. That gives clean ceilings from the published acceptance rates — 5.4 accepted on edits is 5.4 / 1.5 = 3.60×, 2.0 on prose is 1.33×, 5.0 on code from scratch is 3.33× — and the measured rows land just under each, which is what a real system that also pays host costs looks like. The one place the measurement beats its own ceiling is the function-edit row at 3.77×, which is what you get when acceptance on a particular file runs above the 5.4 average.

Sixteen rows, and which one the headline is

Read down the chart rather than at the top of it.

MLX is flat. 151 to 164 tok/s across every task in the suite — a 9% spread over prompts from 38 to 857 tokens. That is exactly what a bandwidth-bound single-token decoder should do, and it is a quiet validation of the whole setup: a baseline that moved around with the prompt would mean something else was wrong.

Husky alone is a tie on prose and a rout on edits. Seven of the sixteen rows are between 1.02× and 1.09×. Four are between 2.91× and 3.89×, and all four are edits — function edit, JSON field, SQL column rename, typo fix — where the reply repeats long runs of the prompt verbatim and prompt lookup has something to copy.

Flash on beats MLX everywhere, by 1.30× to 4.48×, median 1.76×.

The 4.5× is the first row. Function edit, 368 tokens in, Flash on, 730 against 163. It is a real median of three runs. It is also the single most favourable cell in a sixteen-cell table, on the task class the engine was explicitly built for — Husky says so: "Underdog asks Woof for edits like these all day, so this is not a corner case, it is the common one." Fair enough for their product. It is still a maximum, and "up to" is doing the work in the title.

The first-token numbers are the least-quoted and hold up best. 137–177 ms for MLX against 29–39 ms for Husky on a continued chat, a median 4.24× sooner, on a comparison the page sets up to disfavour itself: both engines run from their caches, and it separately reports that MLX's plain generate call — the one without a prompt cache, which is what a naive caller gets — takes 145 to 295 ms. It quotes the cached number in the headline. That is the right choice and almost nobody makes it.

The baseline that was not run

Here is the one thing I would change, and it is the same shape as the question worth asking of any framework comparison: was the baseline given its own best options?

Mostly yes. Both engines get the same weight file — "Husky repacks it without changing a value, and greedy output is compared token by token against MLX after every change". The MLX version is mlx-lm 0.31.3 on MLX 0.32.2, and 0.31.3 is the current release on PyPI, so this is not a stale baseline. MLX gets its prompt cache in the first-token comparison rather than the naive path. Runs that disagreed with their own repeats were discarded. Order was alternated, a canary ran first, repeats had to agree within 12%.

But the headline ratio is a speculative-decoding ratio, and mlx-lm has shipped speculative decoding the whole time:

$ curl -s https://raw.githubusercontent.com/ml-explore/mlx-lm/main/mlx_lm/generate.py \
  | grep -n "draft-model\|num-draft-tokens\|def speculative"
210:        "--draft-model",
216:        "--num-draft-tokens",
476:def speculative_generate_step(

So the comparison in the table is Husky with an eight-token verify step against MLX without one. That is not a small asymmetry — it is the entire mechanism above 1.40×.

Two honest caveats on my own complaint, because they are real:

1. Flash is not portable to MLX. It is not a standalone language model; it reads Woof's own hidden states from five of its layers. You cannot hand it to --draft-model. A fair MLX speculative baseline would need somebody to train or pick a small standalone draft for Woof, which is work Underdog did for their engine and nobody has done for MLX.
2. mlx-lm has no prompt-lookup decoding. I grepped for it; there are no hits for lookup or n-gram anywhere in generate.py. So the 2.9–3.9× on the four edit rows is a capability MLX genuinely does not have, draft model or not, and that part of the gap is not a configuration choice.

Which leaves a precise version of the claim. Husky's engine is about 1.1× MLX on prose. Husky's prompt lookup is 2.9–3.9× on edits and MLX cannot do it at all. Husky's trained draft adds 1.3–1.7× on prose, and MLX has a slot for a draft that this benchmark left empty. All three are worth having. Only the first is a claim about kernels.

What 730 tokens a second is, exactly

Since it is the number that will travel, here it is with every qualifier attached:

• decode only. Tokens per second while writing the reply, after the prompt is processed. Prefill is a separate figure (5,150 tok/s at the caller, 5,460 inside the engine) and first-token latency is a third.
• one row of sixteen. The function edit, 368 tokens in. The median row with Flash on is 279 tok/s; the slowest is 210.
• Flash on. Without the trained draft, that row is 614 and the slowest row is 164 — three tokens a second above MLX.
• the task where the reply copies the prompt, so most of those tokens are verified from prompt lookup rather than generated.
• batch 1, which the page does not state and does not need to: a single-user Mac app, and 159 tok/s on a 2.367 GB model is 377 GB/s, which is only a single-stream number. At any real batch the weight read amortises and both engines' tokens per second go up a lot.
• M5 Max, 40-core GPU, 128 GB, macOS 26.5.1. On a base M5 or an M-series Air the bus is narrower and every figure in the table moves together.

None of that makes 730 wrong. It makes it one cell.

What is not checkable

The engine. Husky is not open source, has no repository, and runs only inside the Underdog app — "Husky is live in Underdog on the Mac today, with no setup." The method section says "the receipts are kept"; they are not published.

So the reproducibility is one-sided in a specific way. Anyone with a Mac can reproduce the baseline exactly: pip install mlx-lm==0.31.3, pull ConwayResearch/Underdog-Woof-4B-1.1, run the sixteen prompts, and check whether MLX really lands at 151–164 tok/s and 137–177 ms to first token on their machine. If it does, the denominator is sound and the only unverified column is Husky's own. If it does not, nothing else matters.

That is a much better position than the 50× was in. It is still short of the bar Laya on Apple silicon set two days earlier, which committed every timing sample, every rejected run and every export manifest to a public tree — including a run it threw away whole because one power-sensor reading came back as 38 kilowatts.

What I would actually ship

• If your replies copy your prompts, build prompt lookup first. It is the cheapest large win in this entire post: no draft model, no training, no extra weights, and 2.9× to 3.9× on the four edit tasks. It costs an n-gram index over the prompt and an eight-row verify step. Every assistant that rewrites text the user already sent is leaving this on the floor.
• Do not expect kernels to move a batch-1 decoder. Measure your model's weight bytes, divide by your machine's achievable bandwidth, and that is your token time. If you are within 40% of it, the remaining work is speculation or batching, not tiling.
• Enable speculative decoding on the baseline before you publish a ratio against it. mlx_lm.generate --draft-model exists. vLLM and SGLang have their own. A speculative engine measured against a non-speculative one is comparing two methods, not two implementations.
• Quote the median, not the maximum. Flash on is 1.76× MLX across sixteen representative tasks and that is a good result for a from-scratch engine. "Up to 4.5×" is the same data reported less usefully.
• Watch what happens to MSI on CUDA. The interesting bet in the post is the last line of What's next — the same compile-time-shapes approach written for NVIDIA. On a datacentre GPU the arithmetic changes completely: batching is available, the bus is faster, and general engines are far more tuned. That is where model-specific inference either earns its name or does not.

Nothing here was executed. There is no Apple silicon in the machine this was written on, so every latency and throughput figure is Reported: read off husky.underdog.ai on 22 September 2026, by Sigil Wen, Underdog / Conway Research, published 20 September 2026. What I checked independently is the model — ConwayResearch/Underdog-Woof-4B-1.1, Apache-2.0, through the Hugging Face API and its raw config.json — and the baseline's capabilities, from ml-explore/mlx-lm on main and the PyPI release index. Every ratio, median and byte count in this piece is arithmetic over those two sources and is reproducible from them. The clip is Underdog's own launch animation, trimmed to its two evidence-bearing beats and otherwise unaltered. There is no figure here because there is none to take: the launch page publishes its numbers as one HTML table and draws its three charts in the browser from that same table, so there is no image to mirror, and the only image file it serves is icon.png, the product mark. The model card carries no figure either — no chart, no diagram, nothing but the Hub's own avatar — and Underdog publishes no paper or post beyond the page. All sixteen rows of that table are transcribed unaltered into the ladder above instead. Companion piece: Laya on Apple silicon, for the same question asked of a different engine in the same week, and for what a fully published benchmark tree looks like.