WEBVTT

00:00:01.290 --> 00:00:02.194
Hi, I'm Axolotl!

00:00:02.194 --> 00:00:06.411
Husky claims up to four and a half times Apple's MLX on a Mac.

00:00:06.411 --> 00:00:07.014
Here's how.

00:00:09.517 --> 00:00:13.271
At batch one, every token streams all two point four gigabytes of

00:00:13.271 --> 00:00:14.836
weights through the memory bus.

00:00:14.836 --> 00:00:17.026
So Husky gets more tokens per read.

00:00:19.775 --> 00:00:21.966
First, the ceiling for one token per read.

00:00:22.386 --> 00:00:26.016
MLX's median is six point two nine milliseconds a token.

00:00:26.436 --> 00:00:29.268
Husky puts the floor for a plain read at four point five.

00:00:29.872 --> 00:00:32.656
So kernels win at most one point four times.

00:00:32.656 --> 00:00:36.986
Without a draft, Husky sits under that line, at one point one two five.

00:00:38.550 --> 00:00:42.368
Guess one, prompt lookup: if the last few tokens appear in the

00:00:42.368 --> 00:00:44.277
prompt, propose the seven that followed.

00:00:44.277 --> 00:00:45.231
No model needed.

00:00:45.651 --> 00:00:50.390
Guess two: the Flash draft, a single small layer reading Woof's own hidden states.

00:00:50.390 --> 00:00:51.744
It proposes seven too.

00:00:52.164 --> 00:00:55.520
Woof checks every guess in one eight-row step, for about one and

00:00:55.520 --> 00:00:57.478
a half times a single row's cost.

00:00:57.898 --> 00:01:01.780
It keeps the tokens it agrees with, so the output never changes.

00:01:01.780 --> 00:01:04.044
On edits, typically five or six survive.

00:01:05.300 --> 00:01:09.176
Across sixteen tasks: one point one two five without a draft, one

00:01:09.176 --> 00:01:10.790
point seven six with Flash.

00:01:10.790 --> 00:01:13.697
The four point four eight is one edit row.

00:01:15.133 --> 00:01:19.551
Underdog's own launch animation replays one typo-fixing edit, from measured timings.

00:01:19.551 --> 00:01:21.961
It is not a screen recording.

00:01:22.381 --> 00:01:27.000
With Flash on, Husky is done at five hundred and thirty-seven tokens a second.

00:01:27.000 --> 00:01:28.320
MLX is still writing.

00:01:29.933 --> 00:01:32.926
At batch one, kernels can't outrun the bus.

00:01:32.926 --> 00:01:37.042
Speed comes from tokens per read, so compare speculation with speculation.

00:01:38.400 --> 00:01:42.620
The bus caps every read, and speculation fills each read with more tokens.

00:01:43.040 --> 00:01:45.272
Every source is in the full article.

00:01:45.272 --> 00:01:45.910
I'm Axolotl.

00:01:45.910 --> 00:01:46.229
Bye!
