WEBVTT

00:00:01.290 --> 00:00:02.532
Hi, I'm Piston!

00:00:02.532 --> 00:00:08.326
FreeToken serves a seven hundred and fifty-three billion parameter model from one workstation GPU.

00:00:08.326 --> 00:00:09.153
Here's how.

00:00:10.517 --> 00:00:12.853
The idea: don't pick an offloading strategy.

00:00:12.853 --> 00:00:16.857
Measure two bandwidths, and let them split each step's cache misses between

00:00:16.857 --> 00:00:18.525
the bus and the CPU.

00:00:20.717 --> 00:00:22.900
At each layer, the router picks experts.

00:00:22.900 --> 00:00:26.642
Hits in the GPU cache run at once; the cache follows the

00:00:26.642 --> 00:00:28.201
router, least recently used out.

00:00:28.621 --> 00:00:31.927
The m misses can be fetched over the bus, or computed on

00:00:31.927 --> 00:00:33.855
the CPU, where their weights already live.

00:00:34.275 --> 00:00:38.138
FreeToken does both at once, then merges the partial sums exactly.

00:00:39.942 --> 00:00:42.306
Balance the two branches so they finish together.

00:00:42.726 --> 00:00:45.414
On top, the bus bandwidth, measured on this machine.

00:00:45.967 --> 00:00:47.864
Below, the host's expert bandwidth.

00:00:49.208 --> 00:00:53.263
Expert size cancels, so the split belongs to the machine, not the model.

00:00:54.842 --> 00:00:58.878
On ordinary consumer machines, the right share to fetch swings from

00:00:58.878 --> 00:01:00.346
twenty-five to ninety-one percent.

00:01:03.692 --> 00:01:07.557
Prefill loads the next whole layer while the GPU computes this one,

00:01:07.557 --> 00:01:09.168
before routing is even known.

00:01:09.588 --> 00:01:14.051
Recurrent state is checkpointed at special tokens, where agent harnesses edit, so

00:01:14.051 --> 00:01:16.283
only the new suffix is recomputed.

00:01:16.703 --> 00:01:20.552
And the GPU cache can resize mid-session, because the CPU pool is

00:01:20.552 --> 00:01:21.836
the source of truth.

00:01:23.017 --> 00:01:27.211
Replace decisions with measurements: the bandwidths on this machine, not a spec

00:01:27.211 --> 00:01:28.609
sheet, decide the split.

00:01:30.567 --> 00:01:34.975
To recap: follow the router, measure the bus, and split the misses by arithmetic.

00:01:35.395 --> 00:01:37.575
Every source is in the full article.

00:01:37.575 --> 00:01:38.197
I'm Piston.

00:01:38.197 --> 00:01:38.509
Bye!
