# A 320B MoE on a free Kaggle TPU: I audited the bytes, not the tok/s

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/kaggle-tpu-320b
> date: 2026-09-18
> tags: tpu, jax, inference, quantization, mixture-of-experts, glm

Kaggle gives a phone-verified account free time on a TPU v5e-8 — the repository
says about twenty hours a week, in sessions Kaggle cuts off at nine, and I have
no account to check that against. The machine is eight chips with sixteen
gigabytes of HBM each.
[`ARahim3/kaggle-tpu-lab`](https://github.com/ARahim3/kaggle-tpu-lab) uses that
allocation to serve **GLM-5.3-Flash** — 320B total parameters, 18B active — from
a notebook, behind a Cloudflare tunnel, speaking both the OpenAI and the
Anthropic APIs, so you can point Claude Code at it. The README says ~64 tok/s on
one stream, 262k context, about sixteen minutes from pressing Run to a URL.

Nothing off the shelf runs that model on a TPU. GLM-5.3-Flash is
[45 layers, 34 of them KDA linear attention and 11 sparse MLA](/articles/glm-5-3-flash),
and the README's explanation is that vLLM's TPU backend has no kernels for the
linear-attention half — which I cannot prove, but which matches
`vllm-project/tpu-inference`'s own release notes, where MLA and XL-MoE support
are still listed as landing. So the author wrote an engine. It is 5,562 lines of
JAX and Pallas under `glm53-flash/engine/glm53/`, plus 1,989 lines of tests, and
the first commit in the repository is dated 1 September 2026. The GLM engine
landed on the 15th. Two weeks.

I do not have a TPU. Everything in this article that is a throughput number is
theirs, and I say so each time. What I can do is check the parts that are made
of bytes: the checkpoint, the memory budget, the quantization, and the code.
Those checks are the article.

The short version: the memory arithmetic is honest to within a rounding step,
the engineering is real, and the README oversells the decode kernel's mechanism,
misstates the model's native context by a factor of four, and rounds off a
quantization loss it has not measured. I measured it.

<HbmBudget />

## Why 320 billion parameters is the whole problem

A v5e-8 is 128 GiB of HBM in total, spread over eight chips that talk over ICI.
The engine's own `hbm()` helper reads `bytes_limit` off chip 0 and the kernel
logs **16.9 GB** usable per chip, so the real ceiling is about 135 GB across the
board.

<Figure
  src="https://ai.thesatyajit.com/articles/kaggle-tpu-320b/fig2.png"
  alt="Diagram titled TPU host in a v5e. Two CPUs, CPU0 and CPU1, sit above eight TPU chips arranged in two rows of four, labelled Chip0 through Chip7. Green lines between the chips are labelled ICI network; white lines from the CPUs down to the chips are labelled PCIe. A dashed box around Chip0 is labelled 1-Chip VM, a dashed box around Chip4 through Chip7 is labelled 4-Chip VM, and a grey dashed box around all eight is labelled 8-Chip VM. The left half is labelled NUMA 0 and the right half NUMA 1."
  caption="The machine the whole design is pinned to: eight chips, two NUMA nodes, one host. kaggle-tpu-lab takes the 8-Chip VM and shards every routed-expert matrix across all of it. (Google Cloud, TPU v5e documentation.)"
/>

GLM-5.3-Flash's released weights are native FP8: 314.4B parameters at one byte
plus 6.9B at bf16, which is 328 GB, or 306 GiB. That is not a near miss; it is
2.4× the machine. And the model's shape makes the problem worse
in a specific way. `config.json` gives `n_routed_experts: 288`,
`moe_intermediate_size: 2048`, `first_k_dense_replace: 3`, so 42 of the 45
layers hold 288 experts of three 4096×2048 matrices each:

```
42 layers × 288 experts × 3 matrices × 4096 × 2048 = 304,405,807,104 parameters
```

Against the Hub's own count of 321,323,031,390 total, that is **94.7%** of the
model in the routed experts alone. The README says "the routed experts are 95%
of the weights, so the whole design follows from where they can live." That
checks out to a decimal place.

Their answer is to take the experts from
[Unsloth's GGUF quantization](https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF)
at roughly three bits, keep them resident in HBM in a layout a Pallas kernel can
read, and take everything else from the original FP8 checkpoint at int8.

## The 110 GB claim, checked against the tensor index

The README's load-bearing number is one sentence: *"Unsloth's UD-IQ3_XXS GGUF
gives 110 GB of expert weights for the whole model."* Everything else in the
design follows from whether that is true.

You do not need to download 120 GB to find out. A GGUF file opens with a magic
word, a version, a tensor count, a key-value block, and then a tensor index that
names every tensor with its dims, its quantization type and its offset. Two
range requests per shard gets you the whole inventory. I read the four
UD-IQ3_XXS shards that way — 1,412 tensors — and summed the 126 routed-expert
tensors of layers 3 through 44, which are exactly the ones the serve kernel
loads, because `kernel/serve_glm53.py` loops `for i in range(N_LAYERS)` with
`N_LAYERS = tc.num_hidden_layers = 45` and never touches `blk.45`, the MTP
layer.

| | bytes | GB | per chip |
|---|---:|---:|---:|
| routed experts, `blk.3`–`blk.44` | 109,867,696,128 | 109.87 | 13.73 |
| after the engine's planar repack | 110,717,042,688 | 110.72 | **13.84** |
| MTP layer experts, never loaded | 2,623,537,152 | 2.62 | — |
| whole GGUF, all four shards | 120,367,571,715 | 120.37 | — |

**110 GB is right.** It is 109.87 as GGUF blocks and 110.72 once the engine has
finished with it, and the README's number sits between them.

The 0.77% the repack adds is worth a paragraph, because it is the kind of detail
that is usually wrong in a README and is not wrong here. `engine/glm53/planes.py`
splits each GGUF block into dense 32-bit planes with the matrix row on the lane
axis, so a Pallas kernel can dequantize an expert without byte-slicing at odd
offsets. Its docstring claims "no expansion" except for the `d` field, which is
stored as pairs of f16 bit patterns, one `u32` per two blocks. I reimplemented
`plane_rows` against my tensor index and got exactly that: the only growth is
the down projections whose per-chip slice is a single block, where two bytes of
scale round up to four. 849 MB total. 106 MB per chip. The docstring is precise.

The rest of the budget I had to derive, and the widget above shows how far the
derivation gets. Non-expert parameters come to 8.921B over layers 0–44 plus the
untied embedding and `lm_head`; at int8 that is 1.12 GB/chip. The vision tower
is the 0.563B of parameters that appear in the FP8 checkpoint but not in the
GGUF at all, which at bf16 is 0.141 GB/chip — and the config comment in
`serve_glm53.py` says `vision: False` "saves ~1 min and 0.14 GB/chip". That is
two independent routes to the same number.

Add it up and I reach 15.13 GB/chip against the 15.4 the kernel prints. The
0.28 GB I cannot itemise is XLA's own footprint and the chart shows it as a
residual rather than quietly absorbing it.

## The format was not a preference

Here is the part I did not expect to be so clean. Unsloth publishes ten
dynamic-quant builds of this model. I pulled the expert byte count out of every
one of them the same way, and the ladder decides the design on its own.

<QuantLadder />

UD-IQ3_XXS is **the largest build on the ladder that fits**. The next rung up,
UD-Q3_K_XL, puts 16.80 GB of experts on each chip — over the whole 16.9 GB
before a single non-expert weight, the vision tower or one token of KV cache.
UD-IQ4_XS is 17.96. So the codebook formats are not a taste; above 2.6 bits per
weight they are the only thing on the shelf that goes in.

That single fact sets up everything that follows, because a codebook format is
cheap to store and expensive to read, and the engine pays that bill on every
decoded token.

## What "3-bit" costs, measured

The README is careful here, and I want to quote it before I take a swing:

> The output is that of a 3-bit-expert quantization, not the bf16 model. Agent
> tasks, long-context retrieval and tool calling behave as expected in our use;
> a formal comparison against the full model is on the list.

The end-to-end half of that comparison needs a TPU and an eval rig. The
weight-space half needs neither, and it is the half nobody ever runs. Unsloth
ships both a UD-IQ3_XXS build and a BF16 build of the same model in the same
repository, so the ground truth is sitting at a known byte offset.

So I range-read matched slices — the same 512 rows of the same expert of the
same tensor, out of both files — dequantized the quantized one with the engine's
own `glm53.iqquant`, and measured.

**Receipts.** kaggle-tpu-lab's README says of its 3-bit expert quantization: "The output is that of a 3-bit-expert quantization, not the bf16 model... a formal comparison against the full model is on the list." This is the weight-space half of that comparison, which does not need a TPU. I pulled matched byte ranges of the same expert matrices out of Unsloth's UD-IQ3_XXS GGUF and Unsloth's BF16 GGUF, dequantized the first with the engine's own glm53.iqquant, and measured the error against the second. The format that carries the gate and up matrices in 41 of the 42 served layers, IQ2_S at 2.5625 bits per weight, reproduces its bf16 counterpart with a 25% relative Frobenius error and a cosine similarity of 0.970. The down matrices, at IQ3_S, land at 13.9%. The three layers Unsloth spent extra bits on (11, 12, 44) get IQ4_XS down projections at 7.6%.

| tensor | expert | format | bits/w | rel Frobenius | cosine | matvec rel |
| :--- | ---: | :--- | ---: | ---: | ---: | ---: |
| blk.20.ffn_gate_exps | 0 | IQ2_S | 2.5625 | 0.2500 | 0.970034 | 0.2487 |
| blk.20.ffn_gate_exps | 100 | IQ2_S | 2.5625 | 0.2499 | 0.970074 | 0.2503 |
| blk.20.ffn_gate_exps | 287 | IQ2_S | 2.5625 | 0.2501 | 0.970011 | 0.2500 |
| blk.20.ffn_down_exps | 0 | IQ3_S | 3.4375 | 0.1385 | 0.991102 | 0.1387 |
| blk.20.ffn_down_exps | 100 | IQ3_S | 3.4375 | 0.1384 | 0.991111 | 0.1395 |
| blk.20.ffn_down_exps | 287 | IQ3_S | 3.4375 | 0.1397 | 0.990931 | 0.1399 |
| blk.11.ffn_gate_exps | 0 | IQ3_S | 3.4375 | 0.1375 | 0.991242 | 0.1364 |
| blk.11.ffn_gate_exps | 100 | IQ3_S | 3.4375 | 0.1375 | 0.991239 | 0.1378 |
| blk.11.ffn_gate_exps | 287 | IQ3_S | 3.4375 | 0.1377 | 0.991218 | 0.1394 |
| blk.11.ffn_down_exps | 0 | IQ4_XS | 4.2500 | 0.0764 | 0.997078 | 0.0763 |
| blk.11.ffn_down_exps | 100 | IQ4_XS | 4.2500 | 0.0767 | 0.997051 | 0.0762 |
| blk.11.ffn_down_exps | 287 | IQ4_XS | 4.2500 | 0.0763 | 0.997086 | 0.0760 |

This is weight-space error, not end-to-end quality: a 25% relative error on a gate matrix does not mean the served model is 25% worse, and the routing, the attention stack and the shared expert are untouched by it. It is a lower bound on how far the served weights are from the ones every published GLM-5.3-Flash benchmark was run on, and it is the number the README's "3-bit" rounds off. bits/weight is the GGUF block size: 82 bytes per 256 weights for IQ2_S, 110 for IQ3_S, 136 for IQ4_XS.

> method: For each tensor: parse the GGUF tensor index from the shard header by HTTP range read, compute the absolute byte offset of expert e row 0, then range-read 512 rows in the quantized file and the same 512 rows in the BF16 file. Dequantize the quantized rows with glm53.iqquant.dequant_rows (the engine's own reader, which I first checked against gguf.quants.dequantize by running the repo's own test_iqquant.py: 62 tests pass on CPU). rel_fro = ||W_q - W_bf16||_F / ||W_bf16||_F over the 512x n_in block; cos is the full-block cosine similarity; matvec is ||(W_q - W_bf16) X||_F / ||W_bf16 X||_F for X a 64-column standard normal draw at seed 0. Three experts per tensor (0, 100, 287). No full weight files downloaded: 34 MB of range reads in total.
> source: https://huggingface.co/unsloth/GLM-5.3-Flash-GGUF
> captured: 2026-09-18
> data: https://ai.thesatyajit.com/articles/kaggle-tpu-320b/data/quant-error.json (12 rows)

The number that matters is the first one. **IQ2_S, at 2.5625 bits per weight,
carries the gate and up projections of 41 of the 42 served layers, and it
reproduces bf16 with a 25.0% relative Frobenius error.** The down projections at
IQ3_S are 13.9%. The three layers Unsloth spent extra bits on — 11, 12 and 44 —
get IQ4_XS down projections at 7.6%.

Two honest caveats. This is weight-space error, not quality: the routing, the
attention stack, the shared expert and the hyper-connections are all untouched,
and a 25% perturbation of a gate matrix inside a SwiGLU does not translate into
anything like a 25% capability drop. And these are three experts out of 288 on
two of 42 layers; the variance across the experts I sampled was under 0.1% of
the value, which is reassuring but is not a full sweep.

What the measurement does establish is the distance. Every GLM-5.3-Flash
benchmark you have read was run on the bf16 or FP8 weights. The ones on this
endpoint are a quarter of the way to somewhere else, on the matrices that make
up most of the model.

### "A dynamic 3-bit mix by layer" is dynamic by matrix, not by layer

While I had the index open I tabulated which format each of the 126 expert
tensors actually uses. The README describes it as "Unsloth's dynamic 3-bit mix
(IQ2_S / IQ3_S / IQ4_XS by layer)". The mix is real. The *by layer* is not.

| layers | gate | up | down |
|---|---|---|---|
| 3–10, 13–43 (39 layers) | IQ2_S | IQ2_S | IQ3_S |
| 11 | IQ3_S | IQ3_S | IQ4_XS |
| 12, 44 | IQ2_S | IQ2_S | IQ4_XS |

Thirty-nine of forty-two layers are identical. The allocation varies by the
matrix's *role* — down projections always get a wider format than gate and up —
and only three layers deviate at all. That is a more interesting fact than "by
layer", and it is Unsloth's design decision rather than this repository's, but
the README passes it along unexamined.

## The kernel, and why 64 tok/s is not a bandwidth number

This is the part of the repository I most enjoyed reading, and the part the
README describes least accurately.

A v5e chip is a TensorCore with one scalar unit, one vector unit and four matrix
multiplication units hanging off HBM:

<Figure
  src="https://ai.thesatyajit.com/articles/kaggle-tpu-320b/fig1.png"
  alt="Block diagram of a TPU v5e chip. On the left, a tall cyan box labelled High Bandwidth Memory. A double-headed arrow connects it to a large box on the right labelled TensorCore, which contains a Scalar Unit and a Vector Unit along the top and four boxes labelled Matrix Multiplication Unit below them."
  caption="One v5e chip: 16 GB of HBM, 800 GiB/s to reach it, four MXUs and one vector unit. At decode the 320B model's experts are unpacked from a codebook on that single vector unit while the four MXUs wait. (Google Cloud, TPU v5e documentation.)"
/>

`engine/glm53/pallas_moe.py` fuses dequantization into the matvec so a
dequantized expert never exists in memory. Per weight, the IQ2_S path looks up a
codebook entry with an in-vreg lane gather, unpacks two level bits and a sign
bit with shifts, converts to float and multiply-accumulates against the
activation broadcast along lanes. The codebook for IQ2_S has 1,024 entries of
8 weights, packed into eight `(8, 128)` table rows, so `_lookup` issues eight
`take_along_axis` gathers and seven selects for each group of eight weights.
IQ3_S is four gathers per group of four. Either way that is **about one dynamic
lane gather per weight**.

The author clearly knows this is the hot path, because the serve config pins the
runtime for it:

```python
# glm53-flash/kernel/serve_glm53.py
"libtpu": "0.0.42.*",            # the image's runtime is 140x slower on gathers
                                 # and cannot run Pallas kernels
"max_len": 262144,               # context capacity (tokens); a multiple of 32
"streams": 4,                    # requests decoded together (one program set per
                                 # batch size, ~2.5 min compile each)
```

Now put a roofline on it. The notebook's config cell publishes three rates:
"one stream ~65 tok/s, two ~40 each, three ~30 each". Fitting a step time that
is linear in the number of (token, expert) pairs a step touches — 8 experts per
token per layer, 42 sparse layers, so 336 pairs per stream — gives
`step(N) = 6.62 ms + 8.97 ms × N`, with residuals under 1.5% on all three
points.

<DecodeCost />

At one stream the expert half of a step moves 384 MB per chip and takes 8.97 ms.
At the v5e's 800 GiB/s that many bytes is **0.45 ms**. The engine is running the
expert path at about 5% of the memory bandwidth available to it, and the whole
step at 11%.

That is not a criticism. It is the point. A 2.89-bit weight is a codebook index,
and the engine is issuing roughly 117.8 billion lane gathers per second per chip
to turn those indices back into numbers. Whether that is the vector unit's
ceiling I cannot say without a profile on real hardware. What the arithmetic
does say is that the bottleneck is not HBM: 384 MB in 8.97 ms is not a bandwidth
wall on an 800 GiB/s chip. The format that makes 320B fit on a free TPU is the
same format that caps decode at 64 tok/s, and the ladder above says there was no
other format to pick. If you want the MXUs busy you need weights the MXU can
eat, and those do not fit on the chip.

## Three things the README gets wrong

### The decode kernel's cost is not "distinct experts"

The README describes the two Pallas paths like this:

> custom Pallas kernels read them straight into the matrix multiply: a fused
> dequant-matvec for decode, where the cost is the number of distinct experts a
> step touches, and a grouped GEMM over the routed rows for prefill.

The prefill half is right. `_apply_ragged_kernel` calls `K.ragged_plan`, which
sorts (token, expert) slots by expert, and the sweep path calls
`K.active_slots`, whose docstring is explicit: *"unique routed experts first
(ascending), the rest of the slots repeat the last active id"*. Deduplication,
by construction.

The decode half has no deduplication anywhere. Here is the dispatch, in
`engine/glm53/resident.py`:

```python
# glm53-flash/engine/glm53/resident.py — verbatim, with elisions marked
    def apply(self, p, x, idx, w, layer, limit):
        """x [N,D]; idx [N,k] int32 into E; w [N,k] fp32 -> partial MoE output [N,D] ..."""
        gate_q, up_q, down_q = p["gate_q"], p["up_q"], p["down_q"]
        N, k = idx.shape
        if N * k <= self.gather_max_rows:
            if self.use_pallas:
                return self._apply_kernel(x, idx, w, layer, gate_q, up_q, down_q, limit)
            ...

    def _apply_kernel(self, x, idx, w, layer, gate_q, up_q, down_q, limit):
        N, k = idx.shape
        flat = idx.reshape(-1)
        qt_gu, qt_dn = self.qtypes[layer]["gate_q"], self.qtypes[layer]["down_q"]
        Xg = jnp.repeat(PL.pm_x(qt_gu, x.astype(jnp.float32)), k, axis=0)   # [Nk, W, C]
        mv = lambda planes, qt, key, X: K.moe_matvec(planes, qt, self.nblk[key], flat, X, ...)
```

`gather_max_rows` is 64 in the serve config and `N * k` is 8 per stream, so
every decode step takes the first branch. `flat` is `idx.reshape(-1)`: every
(token, expert) pair, in token order. No sort, no unique. And `moe_matvec` builds its
Pallas grid straight off it — `grid=(Nk, n_lb)`, one grid step per pair — with
the module docstring saying as much: *"computes, for every expert slot i (one
(token, expert) pair)"*. Pallas will skip a block DMA when consecutive grid
steps ask for the same block, but `flat` is token-major, so two streams that
happen to route to the same expert land nowhere near each other, and the
dequantization is redone per pair regardless.

At one stream the two descriptions agree, because a token's eight experts are
already distinct. At three streams they do not. With 288 experts the gap is
small — under uniform routing, three tokens touch about 23.3 distinct experts
against 24 pairs, so the published rates cannot tell the two models apart, and I
am not claiming the numbers are wrong. The *mechanism* is described wrongly, and
the error grows with stream count and shrinks with expert count. Anyone reading
that sentence to plan a batch size is planning against a kernel that does not
exist.

### 262,144 is the engine's context, not the model's

The README says the engine gives you "the model's native **262,144** tokens".
`config.json` says `max_position_embeddings: 1048576`. Z.ai's own model card
reports evaluations run at 300K and at 1M context. 262,144 is
`CFG["max_len"]` in `kernel/serve_glm53.py` — the engine's cap, and a sensible
one, since the chart above shows each 262k context set costs 0.23 GB/chip and
only a few fit. But it is a quarter of the model's native window, described as
all of it.

### 5,400 lines is 5,562

Trivial, and in the honest direction: the README says "about 5,400 lines of JAX
and Pallas in `engine/glm53/`". `wc -l` on the non-test files gives 5,562, and
the tests add another 1,989. I mention it only because it is the one place the
project undersells itself.

## What they got right that almost nobody does

Compile time. On a TPU, an XLA compile can dwarf everything else, and a free-tier
allocation makes it a first-order cost — and it is the thing quietly excluded
from most published TPU numbers. This README does not exclude it. It puts it in
the table:

> Time to live endpoint — **~16 min** with the serve dataset attached (weights
> 4 min, warm-up 9 min, the rest is the runtime, the tunnel and Kaggle's own
> start); ~22 min from the FP8 checkpoint.

Nine of the sixteen minutes are named as compilation, and the "Good to know"
section goes further: the warm-up is ~9 minutes with a prebuilt XLA cache
attached as a Kaggle dataset and ~14 without, because "what the cache cannot
skip is JAX tracing the programs". The warm-up loop in `serve_glm53.py` compiles
two prefill buckets, four batched decode program sets and four snapshot sizes,
which at the config's own "~2.5 min compile each" is where the nine minutes go.

The tok/s figures are steady-state, post-warm-up, on a warm cache. That is the
right way to quote them and the README says which is which. It is the single
most common place TPU claims go soft, and this one does not.

One caveat on that: the advertised sixteen minutes assumes you attach the
author's prebuilt compile-cache dataset. Build it yourself from the FP8
checkpoint and the README's own number is 22.

## What I actually ran

I have no Kaggle TPU, so nothing here is a throughput measurement. What I could
run, I ran, on CPU.

**The test suite, as far as it goes without torch.** The engine claims to run on
eight virtual CPU devices — `conftest.py` sets
`XLA_FLAGS=--xla_force_host_platform_device_count=8` — which is how the tests
work without hardware. Most of the suite needs PyTorch for its reference
implementation and I ran out of disk installing it. The four files that do not:

```
$ XLA_FLAGS=--xla_force_host_platform_device_count=8 python -m pytest \
    glm53/tests/test_iqquant.py glm53/tests/test_planes.py \
    glm53/tests/test_gguf_reader.py glm53/tests/test_sampling.py -q
..............................................................           [100%]
62 passed in 209.26s (0:03:29)
```

That includes `test_real_samples_match_gguf_py`, which checks the engine's
dequantizer against `gguf.quants.dequantize` — llama.cpp's own reference — on
committed sample bytes.

**The committed fixtures are the real weights.** Those samples are five `.bin`
files in `engine/glm53/tests/data/`, described by a `gguf_samples.json` that
names each one's source tensor. I computed each tensor's absolute offset from
the shard index, range-read the same byte count out of Hugging Face, and
compared:

| fixture | tensor | bytes | sha256 (first 16) |
|---|---|---:|---|
| `IQ2_S` | `blk.3.ffn_gate_exps.weight` | 2,624 | `d2439faeaf446dad` ✓ |
| `IQ3_S` | `blk.3.ffn_down_exps.weight` | 3,520 | `81cf2df0336bd009` ✓ |
| `IQ4_XS` | `blk.11.ffn_down_exps.weight` | 4,352 | `20e4affcc9e948ad` ✓ |
| `Q6_K` | `blk.0.attn_k.weight` | 6,720 | `b410a335977ec35c` ✓ |
| `Q8_0` | `blk.0.hc_attn_fn.weight` | 17,408 | `9102b8d1ac7d17db` ✓ |

All five match upstream byte for byte, and the declared dims match the GGUF's.
The test fixtures are real weights at real offsets, not synthetic data with a
plausible name.

**The decode kernel computes the right thing.** Pallas runs under
`interpret=True` on CPU, and `moe_matvec` is written to allow it. So I took two
real experts of `blk.20.ffn_gate_exps.weight` off Hugging Face, packed them with
the engine's own `planes.pack_planes`, ran the shipped kernel, and compared
against dequantize-then-matvec:

```
planes: {'qs': (256,128), 'sg': (256,128), 'qh': (64,128), 'sc': (64,128), 'd': (16,128)}
pallas moe_matvec (interpret, CPU) out (2, 128) in 17.7s
max |kernel - dequant_rows@x| / max|ref| = 2.279e-07
KERNEL MATCHES REFERENCE
```

Not a TPU, not a timing, and interpret mode does not exercise Mosaic's
tile-alignment rules. But the arithmetic the kernel performs on real 3-bit
codebook weights is correct to float32.

## Where this leaves it

Fifteen days separate this repository's first commit from its current HEAD, and
in that time it put a 320B mixture-of-experts on hardware you get for free by
verifying a phone number. Whether it is the first engine to run this model on a
TPU I cannot prove — the project claims it only "as far as we know", and I found
nothing to contradict it. The memory claim, the one everything else hangs on, is
accurate to 0.12% against the tensor index, and the design choice behind it
turns out to have been forced rather than chosen. The code is better than the
README, which is the right way round and rarer than it should be.

The README's decode sentence is wrong about its own kernel. Its context number
is off by 4×. And the quantization it ships is a quarter of the way from bf16 on
the matrices that dominate the model — a number that was sitting in a byte range
the whole time, waiting for someone to subtract.

If you run this on a TPU, the two numbers I would most like from you are a real
decode rate at four streams and a short-context needle test against the bf16
model. I could not get either.

## Three weeks later: a dense 27B in bf16, and where its 130 tok/s comes from

*Added 7 October 2026.*

A post doing the rounds this week calls Qwen3.8-27B "running in full BF16 on a free Kaggle TPU" one
of "the more ridiculous local AI setups", and lists ~130 tok/s decode, ~10,300 tok/s prefill,
262K context, ~540 tok/s over eight streams, and an OpenAI-compatible endpoint for Claude Code,
Codex CLI and OpenCode. It doesn't name the project. The video it embeds does: it is from Abdur
Rahim, and his post links the same repository, which now has a `qwen38-27b/` folder beside the GLM
engine. I cloned it again (HEAD `9a31b85`, 7 October) and did to the Qwen recipe what I did to the
GLM one above: check what is made of bytes, and put a roofline under the rates.

The first thing to know is that there is no engine here this time. The folder's README is plain
about it: "the GDN Pallas kernels and the MTP drafter are theirs; this repo is packaging and
measurement." The model runs on stock vllm-tpu 0.28.0 with tensor parallelism across all eight
chips, and the repo contributes the Kaggle plumbing, a prebuilt compile cache, and one patch that
matters more than anything else in the folder, which I come back to below.

<Figure
  src="https://ai.thesatyajit.com/articles/kaggle-tpu-320b/fig3.jpg"
  alt="Two terminal panes side by side running Claude Code. The left pane, an Anthropic model, is editing a file called bouncing_balls_heptagon.html and shows a diff, with a status line reading Forging, 1m 33s, 7.8k tokens. The right pane's header reads qwen3.8-27b with xhigh effort, API Usage Billing, ~/Codes/tpu_demo; it shows the prompt asking for a single HTML file with 20 balls bouncing inside a spinning heptagon, and a status line reading Sublimating, 1m 35s, almost done thinking with xhigh effort."
  caption="Claude Code pointed at the Kaggle endpoint (right), next to a hosted model on the same prompt (left). At 1 minute 35 seconds the Qwen side is still thinking (Abdur Rahim's launch video, frame at 0:20)."
/>

### Does 27B in bf16 fit, with 262K of context?

The weights are easy to pin down. The Hub's index file gives a `total_size` of 55,562,855,904
bytes, and reading the 18 shard headers shows every one of the 27,781,427,952 parameters is bf16:
17.11B in the MLPs, 5.56B in the gated-DeltaNet layers, 1.68B in the full-attention layers, 1.27B
each for the untied embedding and output head, 0.46B of vision tower and 0.42B for the one MTP
layer. The README's "55.6 GB safetensors" is right. Split eight ways it is 6.95 GB a chip, under
half of the 16.9 GB the GLM engine saw as usable.

The KV cache is where a 27B at 262K would normally die, and Qwen3.8's shape is what saves it.
`config.json` lists 64 layers with `full_attention_interval: 4`, so only 16 have a KV cache at all;
the other 48 are linear attention with a fixed-size state. Those 16 layers have 4 KV heads of
width 256. That is 16 x 2 x 4 x 256 x 2 bytes, 65,536 bytes a token, which matches the README's
"~64 KB/token". A full 262,144-token context is then 17.18 GB for the whole model.

How that lands on a chip depends on something I couldn't confirm: with eight chips and four KV
heads, either the heads are split and each chip holds half a head's worth (2.15 GB a chip at full
length), or, as vLLM does on GPUs when tensor parallelism exceeds the KV head count, each chip
keeps a copy of one head (4.29 GB). The DeltaNet state is small either way: 48 layers x 48 heads x
128 x 128 in float32 is 151 MB a sequence across the whole machine, 19 MB a chip, a few times that
with the speculative-decoding checkpoints the patch adds.

<QwenHbm />

So the claim holds as the README states it. One request at the model's native 262,144 tokens fits
with room to spare, and here "native" is accurate: `max_position_embeddings` is 262144 in this
config, unlike GLM-5.3-Flash, where the same phrase covered a quarter of the model's window. What
doesn't follow is that the default of four concurrent sequences can each hold 262K at once. With
copied heads that is 17 GB of KV cache a chip, more than the chip has. vLLM doesn't reserve a full
context per slot; it sizes one shared pool from whatever memory is left and guarantees a single
full-length request, so four long agent sessions on one endpoint will queue or get preempted
rather than all sit at 262K. That's how it should work, but it's worth knowing before you share
a URL with three friends.

### 130 tok/s is MTP's number, not the memory's

A dense model in decode reads every weight it uses once per step. For Qwen3.8-27B that is
everything except the vision tower, the MTP layer and all but one row of the embedding table:
25.62B parameters, 51.25 GB, or 6.41 GB a chip. The v5e's HBM moves 800 GiB/s, so one step can't
take less than 7.46 ms, which caps plain one-token-at-a-time decoding at 134 tok/s.

The headline rate is 130. That would be 97% of the roofline, which no real decode loop reaches,
and the README doesn't claim it does. Its table row reads "~130 tok/s with MTP speculative
decoding (78 without, measured before the token-bucket change)". 78 tok/s is 58% of the roofline,
an ordinary efficiency for small-batch decode. The 130 comes from Qwen3.8's built-in
multi-token-prediction head drafting three tokens a step for the full model to verify, and the post
I started from leaves that out entirely.

The README's acceptance profile, "87/66/52 % per draft position", says how much that buys. If
those are the share of steps on which each draft position survives, a step yields on average 1 +
0.87 + 0.66 + 0.52 = 3.05 tokens. A verify step costs the full read plus three draft passes, each
of which reads the MTP layer and the 1.27B output head, about 7.68 GB a chip, so the ceiling with
MTP is roughly 341 tok/s. At 130, each step is taking about 23.5 ms against a plain step's 12.8 ms
at 78 tok/s. Nothing about that looks inflated; it looks like speculative decoding working as
designed with a lot of headroom left. It also means the 130 depends on acceptance, and the repo's
self-test measures it on the easiest case there is: `self_test()` in `serve_qwen38.py` times a
192-token completion of "Write a short story about a lighthouse." at temperature 0 with thinking
off. Code under sampling, mid-reasoning, will accept fewer drafts, and the rate will sit somewhere
between 78 and 130.

There's also a reason the MTP gain counts for more than the speed. The README says stock vllm-tpu
0.28.0 "corrupts outputs" with MTP on this model, because rejected draft tokens advance the
DeltaNet recurrent state and nothing rolls it back; its authors saw 0 of 12 greedy prompts match.
The repo ships a 691-line port of an unmerged upstream fix (tpu-inference PR 3178) that keeps a
state checkpoint per draft position and resumes from the last accepted one, and with it reports
12 of 12 exact matches. The script also turns MTP off if the patch fails to apply rather than serve
garbage. That is the most valuable thing in the folder, and it is the part nobody quotes. Twelve
greedy prompts is a small test, though, and I didn't run it.

The concurrency numbers are less tidy. The table gives "~540 tok/s aggregate (~107 tok/s each, MTP
on)" for eight streams, but eight streams at 107 each would be 856, not 540. Both can't describe
the same run. And the default config caps the server at `max_num_seqs: 4`, so eight concurrent
streams needs one of the other configs the repo ships (131,072 tokens and 16 sequences). Batching
can raise aggregate throughput this way, because every stream shares the one weight read per step,
so 540 is plausible. Which config produced it, and which of the two per-stream figures is right, I
couldn't tell from the repo.

### The prefill numbers agree with each other

10,300 tok/s and "a 105k-token prompt in ~10 s" are the same measurement: 105,000 / 10,300 is
10.2 seconds. The second number, 225K in ~28 s, is only 8,036 tok/s, and at first glance that
looks like a contradiction. At a flat 10,300 tok/s it would take 21.8 seconds.

It isn't one. The 16 full-attention layers do work that grows with the square of the prompt. A
causal pass costs about 2 x 24 heads x 256 x n² floating-point operations a layer, 196,608 n²
over the 16 layers, while the matrix multiplies cost about 48.7 GFLOP a token whatever the length.
Adding the two up, 105K tokens is 7.3 PFLOP of work and 225K is 20.9. Over 10 and 28 seconds
that is 728 and 747 TFLOP/s, 46% and 47% of the 1,576 TFLOP/s the eight chips can do in bf16
at peak. Two prompts, one efficiency. The extra six seconds at 225K is attention, as it should be,
and 46% of peak is a believable figure for long-sequence prefill sharded over eight chips. If
anything the agreement is better than two rounded numbers deserve. I left the DeltaNet layers'
own chunked recurrence out of that sum; it is small next to the projections, but it isn't zero.

One caveat the README raises itself: vllm-tpu 0.28.0 turns off prefix caching for hybrid models,
so an agent session re-prefills its whole history every turn. At 10,000 tok/s that's about five
seconds for 50,000 tokens. Fine for a demo, and the first thing I'd want fixed for real use.

### Fast tokens, slow answers

<Figure
  src="https://ai.thesatyajit.com/articles/kaggle-tpu-320b/fig4.jpg"
  alt="Side-by-side renders of the same task, 20 numbered balls inside a spinning heptagon. The left panel is labelled Claude Opus 4.8, high effort, done in 1m 43s. The right panel is labelled Qwen3.8-27B bf16, xhigh effort, Kaggle TPU v5e-8, done in 8m 0s, with a caption line giving omega = 360 degrees per 5 s, g = 980 px/s squared, e = 0.82. Both show the balls resting near the lowest wall."
  caption="The same prompt finished by both models; the Qwen run on the Kaggle endpoint took 8 minutes against 1 minute 43 seconds (Abdur Rahim's launch video, final frame)."
/>

The launch video closes on this frame, and I think it says more than any of the tok/s figures.
Both programs work. The Qwen one took 8 minutes at xhigh reasoning effort, the server's default.
At 130 tok/s, 8 minutes is room for about 62,000 generated tokens, and most of what a reasoning
model generates is thinking you never read. For an agent, wall-clock time is tokens per second
times tokens per task, and the second factor is the model's, not the TPU's. The README's
`--reasoning-effort medium` launch flag is the knob I'd try first.

My verdict on this one is simpler than on the GLM engine. Every number checks out against the
model's config and the chip's specs, and the "full BF16, no quantization" line is literally true.
The post I started from oversold it in one way only: 130 tok/s is a speculative-decoding rate on
an easy prompt, and the last plain rate the repo measured is 78. Twenty TPU hours a week, in sessions under nine hours,
is what the repo says Kaggle's free tier gives; I have no account to check that against, same as
before.

One last thing the repository's top-level table now says: the GLM-5.3-Flash row reads ~165 tok/s
on one stream and ~1,800 tok/s prefill, against the ~64 tok/s I audited above on 18 September. I
haven't re-read the GLM engine since, so the decode-cost section describes the version I read.

<ChangeMyMind>
  <Falsifier claim="110 GB of expert weights is accurate.">
    A recount of the UD-IQ3_XXS tensor index that disagrees with
    109,867,696,128 bytes over `blk.3`–`blk.44`, or evidence that the serve
    kernel loads `blk.45`&rsquo;s experts after all — it would add 2.62 GB and
    0.33 GB/chip. Unsloth requantizing and reuploading the build would also move
    it; my read is dated 18 September 2026.
  </Falsifier>
  <Falsifier claim="The decode path does not deduplicate experts across streams.">
    A sort or a unique-ing step between `idx` and `K.moe_matvec` that I missed,
    or a Pallas revision where the pipeline elides a repeated block index that is
    not adjacent in the grid. Either would make the README&rsquo;s sentence
    right and this section wrong.
  </Falsifier>
  <Falsifier claim="IQ2_S sits 25% from bf16 in relative Frobenius error.">
    Rerunning the same range reads on more experts and more layers and getting
    materially different numbers, or a bug in `glm53.iqquant` that
    `test_iqquant.py` does not catch — though it checks against llama.cpp&rsquo;s
    own reference, so that bug would have to be in both.
  </Falsifier>
  <Falsifier claim="The decode path runs at roughly 5% of HBM bandwidth.">
    A profile on an actual v5e-8 showing the expert kernels DMA-bound rather than
    VPU-bound. My figure is arithmetic on three published rates and a byte count,
    not a trace, and a different fixed/variable split would move it. The
    direction holds — 384 MB in 8.97 ms is not a bandwidth limit on an
    800 GiB/s chip — but the exact percentage is not.
  </Falsifier>
  <Falsifier claim="UD-IQ3_XXS is the largest Unsloth build that fits on a v5e-8.">
    A build I have not counted, a non-Unsloth quantization between 2.9 and
    3.5 bits, or an engine change that frees more than 1.9 GB/chip — offloading
    the vision tower and running one context instead of three would get close.
  </Falsifier>
  <Falsifier claim="Qwen3.8-27B's 130 tok/s depends on MTP; plain decode cannot exceed about 134.">
    A plain, non-speculative decode rate on a v5e-8 well above 78 tok/s, or evidence that some
    of the 51.25 GB I count as read per step is skipped. The roofline is weights over 800 GiB/s
    a chip, so only a smaller read or a faster chip moves it.
  </Falsifier>
</ChangeMyMind>