A 320B MoE on a free Kaggle TPU: I audited the bytes, not the tok/s

mdjsonmcpcopy↗ai

2026-09-18 · 31 min · tpu · jax · inference · quantization · mixture-of-experts · glm

Why read this

High· top 30%

Audits a 320B MoE on a free Kaggle TPU from GGUF bytes: 110 GB holds, IQ2_S sits 25% from bf16, and the decode kernel never dedups experts.

• Original, source-checked analysis
• A new technique
• Concrete numbers to act on

·GPUs, kernels & systems·Needs datacenter GPUs·Practitioner tool

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

3 of 3: Reproduces the headline result, or shows from primary files it is wrong

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

3 of 3: The only place this analysis exists

Score 73 of 100, ranked 86 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Kaggle gives a phone-verified account free time on a TPU v5e-8 — the repository says about twenty hours a week, in sessions Kaggle cuts off at nine, and I have no account to check that against. The machine is eight chips with sixteen gigabytes of HBM each. ARahim3/kaggle-tpu-lab uses that allocation to serve GLM-5.3-Flash — 320B total parameters, 18B active — from a notebook, behind a Cloudflare tunnel, speaking both the OpenAI and the Anthropic APIs, so you can point Claude Code at it. The README says ~64 tok/s on one stream, 262k context, about sixteen minutes from pressing Run to a URL.

Nothing off the shelf runs that model on a TPU. GLM-5.3-Flash is 45 layers, 34 of them KDA linear attention and 11 sparse MLA, and the README's explanation is that vLLM's TPU backend has no kernels for the linear-attention half — which I cannot prove, but which matches vllm-project/tpu-inference's own release notes, where MLA and XL-MoE support are still listed as landing. So the author wrote an engine. It is 5,562 lines of JAX and Pallas under glm53-flash/engine/glm53/, plus 1,989 lines of tests, and the first commit in the repository is dated 1 September 2026. The GLM engine landed on the 15th. Two weeks.

I do not have a TPU. Everything in this article that is a throughput number is theirs, and I say so each time. What I can do is check the parts that are made of bytes: the checkpoint, the memory budget, the quantization, and the code. Those checks are the article.

The short version: the memory arithmetic is honest to within a rounding step, the engineering is real, and the README oversells the decode kernel's mechanism, misstates the model's native context by a factor of four, and rounds off a quantization loss it has not measured. I measured it.

Why 320 billion parameters is the whole problem

A v5e-8 is 128 GiB of HBM in total, spread over eight chips that talk over ICI. The engine's own hbm() helper reads bytes_limit off chip 0 and the kernel logs 16.9 GB usable per chip, so the real ceiling is about 135 GB across the board.

expand

GLM-5.3-Flash's released weights are native FP8: 314.4B parameters at one byte plus 6.9B at bf16, which is 328 GB, or 306 GiB. That is not a near miss; it is 2.4× the machine. And the model's shape makes the problem worse in a specific way. config.json gives n_routed_experts: 288, moe_intermediate_size: 2048, first_k_dense_replace: 3, so 42 of the 45 layers hold 288 experts of three 4096×2048 matrices each:

42 layers × 288 experts × 3 matrices × 4096 × 2048 = 304,405,807,104 parameters

Against the Hub's own count of 321,323,031,390 total, that is 94.7% of the model in the routed experts alone. The README says "the routed experts are 95% of the weights, so the whole design follows from where they can live." That checks out to a decimal place.

Their answer is to take the experts from Unsloth's GGUF quantization at roughly three bits, keep them resident in HBM in a layout a Pallas kernel can read, and take everything else from the original FP8 checkpoint at int8.

The 110 GB claim, checked against the tensor index

The README's load-bearing number is one sentence: "Unsloth's UD-IQ3_XXS GGUF gives 110 GB of expert weights for the whole model." Everything else in the design follows from whether that is true.

You do not need to download 120 GB to find out. A GGUF file opens with a magic word, a version, a tensor count, a key-value block, and then a tensor index that names every tensor with its dims, its quantization type and its offset. Two range requests per shard gets you the whole inventory. I read the four UD-IQ3_XXS shards that way — 1,412 tensors — and summed the 126 routed-expert tensors of layers 3 through 44, which are exactly the ones the serve kernel loads, because kernel/serve_glm53.py loops for i in range(N_LAYERS) with N_LAYERS = tc.num_hidden_layers = 45 and never touches blk.45, the MTP layer.

110 GB is right. It is 109.87 as GGUF blocks and 110.72 once the engine has finished with it, and the README's number sits between them.

The 0.77% the repack adds is worth a paragraph, because it is the kind of detail that is usually wrong in a README and is not wrong here. engine/glm53/planes.py splits each GGUF block into dense 32-bit planes with the matrix row on the lane axis, so a Pallas kernel can dequantize an expert without byte-slicing at odd offsets. Its docstring claims "no expansion" except for the d field, which is stored as pairs of f16 bit patterns, one u32 per two blocks. I reimplemented plane_rows against my tensor index and got exactly that: the only growth is the down projections whose per-chip slice is a single block, where two bytes of scale round up to four. 849 MB total. 106 MB per chip. The docstring is precise.

The rest of the budget I had to derive, and the widget above shows how far the derivation gets. Non-expert parameters come to 8.921B over layers 0–44 plus the untied embedding and lm_head; at int8 that is 1.12 GB/chip. The vision tower is the 0.563B of parameters that appear in the FP8 checkpoint but not in the GGUF at all, which at bf16 is 0.141 GB/chip — and the config comment in serve_glm53.py says vision: False "saves ~1 min and 0.14 GB/chip". That is two independent routes to the same number.

Add it up and I reach 15.13 GB/chip against the 15.4 the kernel prints. The 0.28 GB I cannot itemise is XLA's own footprint and the chart shows it as a residual rather than quietly absorbing it.

The format was not a preference

Here is the part I did not expect to be so clean. Unsloth publishes ten dynamic-quant builds of this model. I pulled the expert byte count out of every one of them the same way, and the ladder decides the design on its own.

UD-IQ3_XXS is the largest build on the ladder that fits. The next rung up, UD-Q3_K_XL, puts 16.80 GB of experts on each chip — over the whole 16.9 GB before a single non-expert weight, the vision tower or one token of KV cache. UD-IQ4_XS is 17.96. So the codebook formats are not a taste; above 2.6 bits per weight they are the only thing on the shelf that goes in.

That single fact sets up everything that follows, because a codebook format is cheap to store and expensive to read, and the engine pays that bill on every decoded token.

What "3-bit" costs, measured

The README is careful here, and I want to quote it before I take a swing:

The output is that of a 3-bit-expert quantization, not the bf16 model. Agent tasks, long-context retrieval and tool calling behave as expected in our use; a formal comparison against the full model is on the list.

The end-to-end half of that comparison needs a TPU and an eval rig. The weight-space half needs neither, and it is the half nobody ever runs. Unsloth ships both a UD-IQ3_XXS build and a BF16 build of the same model in the same repository, so the ground truth is sitting at a known byte offset.

So I range-read matched slices — the same 512 rows of the same expert of the same tensor, out of both files — dequantized the quantized one with the engine's own glm53.iqquant, and measured.

The number that matters is the first one. IQ2_S, at 2.5625 bits per weight, carries the gate and up projections of 41 of the 42 served layers, and it reproduces bf16 with a 25.0% relative Frobenius error. The down projections at IQ3_S are 13.9%. The three layers Unsloth spent extra bits on — 11, 12 and 44 — get IQ4_XS down projections at 7.6%.

Two honest caveats. This is weight-space error, not quality: the routing, the attention stack, the shared expert and the hyper-connections are all untouched, and a 25% perturbation of a gate matrix inside a SwiGLU does not translate into anything like a 25% capability drop. And these are three experts out of 288 on two of 42 layers; the variance across the experts I sampled was under 0.1% of the value, which is reassuring but is not a full sweep.

What the measurement does establish is the distance. Every GLM-5.3-Flash benchmark you have read was run on the bf16 or FP8 weights. The ones on this endpoint are a quarter of the way to somewhere else, on the matrices that make up most of the model.

"A dynamic 3-bit mix by layer" is dynamic by matrix, not by layer

While I had the index open I tabulated which format each of the 126 expert tensors actually uses. The README describes it as "Unsloth's dynamic 3-bit mix (IQ2_S / IQ3_S / IQ4_XS by layer)". The mix is real. The by layer is not.

Thirty-nine of forty-two layers are identical. The allocation varies by the matrix's role — down projections always get a wider format than gate and up — and only three layers deviate at all. That is a more interesting fact than "by layer", and it is Unsloth's design decision rather than this repository's, but the README passes it along unexamined.

The kernel, and why 64 tok/s is not a bandwidth number

This is the part of the repository I most enjoyed reading, and the part the README describes least accurately.

A v5e chip is a TensorCore with one scalar unit, one vector unit and four matrix multiplication units hanging off HBM:

expand

engine/glm53/pallas_moe.py fuses dequantization into the matvec so a dequantized expert never exists in memory. Per weight, the IQ2_S path looks up a codebook entry with an in-vreg lane gather, unpacks two level bits and a sign bit with shifts, converts to float and multiply-accumulates against the activation broadcast along lanes. The codebook for IQ2_S has 1,024 entries of 8 weights, packed into eight (8, 128) table rows, so _lookup issues eight take_along_axis gathers and seven selects for each group of eight weights. IQ3_S is four gathers per group of four. Either way that is about one dynamic lane gather per weight.

The author clearly knows this is the hot path, because the serve config pins the runtime for it:

# glm53-flash/kernel/serve_glm53.py
"libtpu": "0.0.42.*",            # the image's runtime is 140x slower on gathers
                                 # and cannot run Pallas kernels
"max_len": 262144,               # context capacity (tokens); a multiple of 32
"streams": 4,                    # requests decoded together (one program set per
                                 # batch size, ~2.5 min compile each)

Now put a roofline on it. The notebook's config cell publishes three rates: "one stream ~65 tok/s, two ~40 each, three ~30 each". Fitting a step time that is linear in the number of (token, expert) pairs a step touches — 8 experts per token per layer, 42 sparse layers, so 336 pairs per stream — gives step(N) = 6.62 ms + 8.97 ms × N, with residuals under 1.5% on all three points.

At one stream the expert half of a step moves 384 MB per chip and takes 8.97 ms. At the v5e's 800 GiB/s that many bytes is 0.45 ms. The engine is running the expert path at about 5% of the memory bandwidth available to it, and the whole step at 11%.

That is not a criticism. It is the point. A 2.89-bit weight is a codebook index, and the engine is issuing roughly 117.8 billion lane gathers per second per chip to turn those indices back into numbers. Whether that is the vector unit's ceiling I cannot say without a profile on real hardware. What the arithmetic does say is that the bottleneck is not HBM: 384 MB in 8.97 ms is not a bandwidth wall on an 800 GiB/s chip. The format that makes 320B fit on a free TPU is the same format that caps decode at 64 tok/s, and the ladder above says there was no other format to pick. If you want the MXUs busy you need weights the MXU can eat, and those do not fit on the chip.

Three things the README gets wrong

The decode kernel's cost is not "distinct experts"

The README describes the two Pallas paths like this:

custom Pallas kernels read them straight into the matrix multiply: a fused dequant-matvec for decode, where the cost is the number of distinct experts a step touches, and a grouped GEMM over the routed rows for prefill.

The prefill half is right. _apply_ragged_kernel calls K.ragged_plan, which sorts (token, expert) slots by expert, and the sweep path calls K.active_slots, whose docstring is explicit: "unique routed experts first (ascending), the rest of the slots repeat the last active id". Deduplication, by construction.

The decode half has no deduplication anywhere. Here is the dispatch, in engine/glm53/resident.py:

# glm53-flash/engine/glm53/resident.py — verbatim, with elisions marked
    def apply(self, p, x, idx, w, layer, limit):
        """x [N,D]; idx [N,k] int32 into E; w [N,k] fp32 -> partial MoE output [N,D] ..."""
        gate_q, up_q, down_q = p["gate_q"], p["up_q"], p["down_q"]
        N, k = idx.shape
        if N * k <= self.gather_max_rows:
            if self.use_pallas:
                return self._apply_kernel(x, idx, w, layer, gate_q, up_q, down_q, limit)
            ...
 
    def _apply_kernel(self, x, idx, w, layer, gate_q, up_q, down_q, limit):
        N, k = idx.shape
        flat = idx.reshape(-1)
        qt_gu, qt_dn = self.qtypes[layer]["gate_q"], self.qtypes[layer]["down_q"]
        Xg = jnp.repeat(PL.pm_x(qt_gu, x.astype(jnp.float32)), k, axis=0)   # [Nk, W, C]
        mv = lambda planes, qt, key, X: K.moe_matvec(planes, qt, self.nblk[key], flat, X, ...)

gather_max_rows is 64 in the serve config and N * k is 8 per stream, so every decode step takes the first branch. flat is idx.reshape(-1): every (token, expert) pair, in token order. No sort, no unique. And moe_matvec builds its Pallas grid straight off it — grid=(Nk, n_lb), one grid step per pair — with the module docstring saying as much: "computes, for every expert slot i (one (token, expert) pair)". Pallas will skip a block DMA when consecutive grid steps ask for the same block, but flat is token-major, so two streams that happen to route to the same expert land nowhere near each other, and the dequantization is redone per pair regardless.

At one stream the two descriptions agree, because a token's eight experts are already distinct. At three streams they do not. With 288 experts the gap is small — under uniform routing, three tokens touch about 23.3 distinct experts against 24 pairs, so the published rates cannot tell the two models apart, and I am not claiming the numbers are wrong. The mechanism is described wrongly, and the error grows with stream count and shrinks with expert count. Anyone reading that sentence to plan a batch size is planning against a kernel that does not exist.

262,144 is the engine's context, not the model's

The README says the engine gives you "the model's native 262,144 tokens". config.json says max_position_embeddings: 1048576. Z.ai's own model card reports evaluations run at 300K and at 1M context. 262,144 is CFG["max_len"] in kernel/serve_glm53.py — the engine's cap, and a sensible one, since the chart above shows each 262k context set costs 0.23 GB/chip and only a few fit. But it is a quarter of the model's native window, described as all of it.

5,400 lines is 5,562

Trivial, and in the honest direction: the README says "about 5,400 lines of JAX and Pallas in engine/glm53/". wc -l on the non-test files gives 5,562, and the tests add another 1,989. I mention it only because it is the one place the project undersells itself.

What they got right that almost nobody does

Compile time. On a TPU, an XLA compile can dwarf everything else, and a free-tier allocation makes it a first-order cost — and it is the thing quietly excluded from most published TPU numbers. This README does not exclude it. It puts it in the table:

Time to live endpoint — ~16 min with the serve dataset attached (weights 4 min, warm-up 9 min, the rest is the runtime, the tunnel and Kaggle's own start); ~22 min from the FP8 checkpoint.

Nine of the sixteen minutes are named as compilation, and the "Good to know" section goes further: the warm-up is ~9 minutes with a prebuilt XLA cache attached as a Kaggle dataset and ~14 without, because "what the cache cannot skip is JAX tracing the programs". The warm-up loop in serve_glm53.py compiles two prefill buckets, four batched decode program sets and four snapshot sizes, which at the config's own "~2.5 min compile each" is where the nine minutes go.

The tok/s figures are steady-state, post-warm-up, on a warm cache. That is the right way to quote them and the README says which is which. It is the single most common place TPU claims go soft, and this one does not.

One caveat on that: the advertised sixteen minutes assumes you attach the author's prebuilt compile-cache dataset. Build it yourself from the FP8 checkpoint and the README's own number is 22.

What I actually ran

I have no Kaggle TPU, so nothing here is a throughput measurement. What I could run, I ran, on CPU.

The test suite, as far as it goes without torch. The engine claims to run on eight virtual CPU devices — conftest.py sets XLA_FLAGS=--xla_force_host_platform_device_count=8 — which is how the tests work without hardware. Most of the suite needs PyTorch for its reference implementation and I ran out of disk installing it. The four files that do not:

$ XLA_FLAGS=--xla_force_host_platform_device_count=8 python -m pytest \
    glm53/tests/test_iqquant.py glm53/tests/test_planes.py \
    glm53/tests/test_gguf_reader.py glm53/tests/test_sampling.py -q
..............................................................           [100%]
62 passed in 209.26s (0:03:29)

That includes test_real_samples_match_gguf_py, which checks the engine's dequantizer against gguf.quants.dequantize — llama.cpp's own reference — on committed sample bytes.

The committed fixtures are the real weights. Those samples are five .bin files in engine/glm53/tests/data/, described by a gguf_samples.json that names each one's source tensor. I computed each tensor's absolute offset from the shard index, range-read the same byte count out of Hugging Face, and compared:

All five match upstream byte for byte, and the declared dims match the GGUF's. The test fixtures are real weights at real offsets, not synthetic data with a plausible name.

The decode kernel computes the right thing. Pallas runs under interpret=True on CPU, and moe_matvec is written to allow it. So I took two real experts of blk.20.ffn_gate_exps.weight off Hugging Face, packed them with the engine's own planes.pack_planes, ran the shipped kernel, and compared against dequantize-then-matvec:

planes: {'qs': (256,128), 'sg': (256,128), 'qh': (64,128), 'sc': (64,128), 'd': (16,128)}
pallas moe_matvec (interpret, CPU) out (2, 128) in 17.7s
max |kernel - dequant_rows@x| / max|ref| = 2.279e-07
KERNEL MATCHES REFERENCE

Not a TPU, not a timing, and interpret mode does not exercise Mosaic's tile-alignment rules. But the arithmetic the kernel performs on real 3-bit codebook weights is correct to float32.

Where this leaves it

Fifteen days separate this repository's first commit from its current HEAD, and in that time it put a 320B mixture-of-experts on hardware you get for free by verifying a phone number. Whether it is the first engine to run this model on a TPU I cannot prove — the project claims it only "as far as we know", and I found nothing to contradict it. The memory claim, the one everything else hangs on, is accurate to 0.12% against the tensor index, and the design choice behind it turns out to have been forced rather than chosen. The code is better than the README, which is the right way round and rarer than it should be.

The README's decode sentence is wrong about its own kernel. Its context number is off by 4×. And the quantization it ships is a quarter of the way from bf16 on the matrices that dominate the model — a number that was sitting in a byte range the whole time, waiting for someone to subtract.

If you run this on a TPU, the two numbers I would most like from you are a real decode rate at four streams and a short-context needle test against the bf16 model. I could not get either.

Three weeks later: a dense 27B in bf16, and where its 130 tok/s comes from

Added 7 October 2026.

A post doing the rounds this week calls Qwen3.8-27B "running in full BF16 on a free Kaggle TPU" one of "the more ridiculous local AI setups", and lists ~130 tok/s decode, ~10,300 tok/s prefill, 262K context, ~540 tok/s over eight streams, and an OpenAI-compatible endpoint for Claude Code, Codex CLI and OpenCode. It doesn't name the project. The video it embeds does: it is from Abdur Rahim, and his post links the same repository, which now has a qwen38-27b/ folder beside the GLM engine. I cloned it again (HEAD 9a31b85, 7 October) and did to the Qwen recipe what I did to the GLM one above: check what is made of bytes, and put a roofline under the rates.

The first thing to know is that there is no engine here this time. The folder's README is plain about it: "the GDN Pallas kernels and the MTP drafter are theirs; this repo is packaging and measurement." The model runs on stock vllm-tpu 0.28.0 with tensor parallelism across all eight chips, and the repo contributes the Kaggle plumbing, a prebuilt compile cache, and one patch that matters more than anything else in the folder, which I come back to below.

expand

Does 27B in bf16 fit, with 262K of context?

The weights are easy to pin down. The Hub's index file gives a total_size of 55,562,855,904 bytes, and reading the 18 shard headers shows every one of the 27,781,427,952 parameters is bf16: 17.11B in the MLPs, 5.56B in the gated-DeltaNet layers, 1.68B in the full-attention layers, 1.27B each for the untied embedding and output head, 0.46B of vision tower and 0.42B for the one MTP layer. The README's "55.6 GB safetensors" is right. Split eight ways it is 6.95 GB a chip, under half of the 16.9 GB the GLM engine saw as usable.

The KV cache is where a 27B at 262K would normally die, and Qwen3.8's shape is what saves it. config.json lists 64 layers with full_attention_interval: 4, so only 16 have a KV cache at all; the other 48 are linear attention with a fixed-size state. Those 16 layers have 4 KV heads of width 256. That is 16 x 2 x 4 x 256 x 2 bytes, 65,536 bytes a token, which matches the README's "~64 KB/token". A full 262,144-token context is then 17.18 GB for the whole model.

How that lands on a chip depends on something I couldn't confirm: with eight chips and four KV heads, either the heads are split and each chip holds half a head's worth (2.15 GB a chip at full length), or, as vLLM does on GPUs when tensor parallelism exceeds the KV head count, each chip keeps a copy of one head (4.29 GB). The DeltaNet state is small either way: 48 layers x 48 heads x 128 x 128 in float32 is 151 MB a sequence across the whole machine, 19 MB a chip, a few times that with the speculative-decoding checkpoints the patch adds.

So the claim holds as the README states it. One request at the model's native 262,144 tokens fits with room to spare, and here "native" is accurate: max_position_embeddings is 262144 in this config, unlike GLM-5.3-Flash, where the same phrase covered a quarter of the model's window. What doesn't follow is that the default of four concurrent sequences can each hold 262K at once. With copied heads that is 17 GB of KV cache a chip, more than the chip has. vLLM doesn't reserve a full context per slot; it sizes one shared pool from whatever memory is left and guarantees a single full-length request, so four long agent sessions on one endpoint will queue or get preempted rather than all sit at 262K. That's how it should work, but it's worth knowing before you share a URL with three friends.

130 tok/s is MTP's number, not the memory's

A dense model in decode reads every weight it uses once per step. For Qwen3.8-27B that is everything except the vision tower, the MTP layer and all but one row of the embedding table: 25.62B parameters, 51.25 GB, or 6.41 GB a chip. The v5e's HBM moves 800 GiB/s, so one step can't take less than 7.46 ms, which caps plain one-token-at-a-time decoding at 134 tok/s.

The headline rate is 130. That would be 97% of the roofline, which no real decode loop reaches, and the README doesn't claim it does. Its table row reads "~130 tok/s with MTP speculative decoding (78 without, measured before the token-bucket change)". 78 tok/s is 58% of the roofline, an ordinary efficiency for small-batch decode. The 130 comes from Qwen3.8's built-in multi-token-prediction head drafting three tokens a step for the full model to verify, and the post I started from leaves that out entirely.

The README's acceptance profile, "87/66/52 % per draft position", says how much that buys. If those are the share of steps on which each draft position survives, a step yields on average 1 + 0.87 + 0.66 + 0.52 = 3.05 tokens. A verify step costs the full read plus three draft passes, each of which reads the MTP layer and the 1.27B output head, about 7.68 GB a chip, so the ceiling with MTP is roughly 341 tok/s. At 130, each step is taking about 23.5 ms against a plain step's 12.8 ms at 78 tok/s. Nothing about that looks inflated; it looks like speculative decoding working as designed with a lot of headroom left. It also means the 130 depends on acceptance, and the repo's self-test measures it on the easiest case there is: self_test() in serve_qwen38.py times a 192-token completion of "Write a short story about a lighthouse." at temperature 0 with thinking off. Code under sampling, mid-reasoning, will accept fewer drafts, and the rate will sit somewhere between 78 and 130.

There's also a reason the MTP gain counts for more than the speed. The README says stock vllm-tpu 0.28.0 "corrupts outputs" with MTP on this model, because rejected draft tokens advance the DeltaNet recurrent state and nothing rolls it back; its authors saw 0 of 12 greedy prompts match. The repo ships a 691-line port of an unmerged upstream fix (tpu-inference PR 3178) that keeps a state checkpoint per draft position and resumes from the last accepted one, and with it reports 12 of 12 exact matches. The script also turns MTP off if the patch fails to apply rather than serve garbage. That is the most valuable thing in the folder, and it is the part nobody quotes. Twelve greedy prompts is a small test, though, and I didn't run it.

The concurrency numbers are less tidy. The table gives "~540 tok/s aggregate (~107 tok/s each, MTP on)" for eight streams, but eight streams at 107 each would be 856, not 540. Both can't describe the same run. And the default config caps the server at max_num_seqs: 4, so eight concurrent streams needs one of the other configs the repo ships (131,072 tokens and 16 sequences). Batching can raise aggregate throughput this way, because every stream shares the one weight read per step, so 540 is plausible. Which config produced it, and which of the two per-stream figures is right, I couldn't tell from the repo.

The prefill numbers agree with each other

10,300 tok/s and "a 105k-token prompt in ~10 s" are the same measurement: 105,000 / 10,300 is 10.2 seconds. The second number, 225K in ~28 s, is only 8,036 tok/s, and at first glance that looks like a contradiction. At a flat 10,300 tok/s it would take 21.8 seconds.

It isn't one. The 16 full-attention layers do work that grows with the square of the prompt. A causal pass costs about 2 x 24 heads x 256 x n² floating-point operations a layer, 196,608 n² over the 16 layers, while the matrix multiplies cost about 48.7 GFLOP a token whatever the length. Adding the two up, 105K tokens is 7.3 PFLOP of work and 225K is 20.9. Over 10 and 28 seconds that is 728 and 747 TFLOP/s, 46% and 47% of the 1,576 TFLOP/s the eight chips can do in bf16 at peak. Two prompts, one efficiency. The extra six seconds at 225K is attention, as it should be, and 46% of peak is a believable figure for long-sequence prefill sharded over eight chips. If anything the agreement is better than two rounded numbers deserve. I left the DeltaNet layers' own chunked recurrence out of that sum; it is small next to the projections, but it isn't zero.

One caveat the README raises itself: vllm-tpu 0.28.0 turns off prefix caching for hybrid models, so an agent session re-prefills its whole history every turn. At 10,000 tok/s that's about five seconds for 50,000 tokens. Fine for a demo, and the first thing I'd want fixed for real use.

Fast tokens, slow answers

expand

The launch video closes on this frame, and I think it says more than any of the tok/s figures. Both programs work. The Qwen one took 8 minutes at xhigh reasoning effort, the server's default. At 130 tok/s, 8 minutes is room for about 62,000 generated tokens, and most of what a reasoning model generates is thinking you never read. For an agent, wall-clock time is tokens per second times tokens per task, and the second factor is the model's, not the TPU's. The README's --reasoning-effort medium launch flag is the knob I'd try first.

My verdict on this one is simpler than on the GLM engine. Every number checks out against the model's config and the chip's specs, and the "full BF16, no quantization" line is literally true. The post I started from oversold it in one way only: 130 tok/s is a speculative-decoding rate on an easy prompt, and the last plain rate the repo measured is 78. Twenty TPU hours a week, in sessions under nine hours, is what the repo says Kaggle's free tier gives; I have no account to check that against, same as before.

One last thing the repository's top-level table now says: the GLM-5.3-Flash row reads ~165 tok/s on one stream and ~1,800 tok/s prefill, against the ~64 tok/s I audited above on 18 September. I haven't re-read the GLM engine since, so the decode-cost section describes the version I read.