Gemma 4: an open multimodal family, tuned to the KV-cache budget

Gemma 4 is Google DeepMind's next open-weight family: natively multimodal (text, image, audio), released under Apache 2.0, with sizes from 2.3B to 31B. Four dense models — effective 2.3B (E2B), 4.5B (E4B), 12B, 31B — plus a Mixture-of-Experts variant, 26B-A4B, with ~3.8B activated of 26B total. The headline features read like a checklist: a thinking mode, an encoder-free 12B, and a raft of efficiency work aimed at long context. I care about the last part most, because it's the part that decides whether you can actually serve these on the hardware you have.

Where the sizes land

The family splits by how much of the model runs per token, and by how "effective" and "total" parameters differ:

"Effective" is the Gemma 3n move: E2B and E4B keep 5B and 8B parameters on disk but stream per-layer embeddings so only 2.3B / 4.5B are resident in the compute path. It's a memory-placement trick for on-device, not a routing one — orthogonal to the MoE sparsity in 26B-A4B, where a router activates ~3.8B of 26B per token.

Long context is a memory problem

The expensive part of a long prompt isn't the matmul — it's the KV cache. Every past token leaves a key and a value in every attention layer, and under full attention that cache grows linearly with context on every layer:

The factor of 2 is K and V; $L$ is the context length. At 128k tokens this is what fills your accelerator's memory. Gemma 4 attacks it structurally. Most layers are local — sliding-window attention that only looks back a fixed window $W$ — and only 1 in 6 is global, attending over the whole context. The ratio is 5:1 for most models, 4:1 for E2B:

That interleave changes the scaling. Local layers stop growing once the context passes $W$; only the global layers keep paying for length. Then two more moves shrink the global term itself. On global layers Gemma 4 reuses keys as values ($V = K$), storing one tensor where full attention stores two — except on E2B/E4B. Position uses pp-RoPE ($p = 0.25$) on global layers and ordinary RoPE (frequency 10k) on local ones, with global frequency 1M. Combined with KV-cache sharing, the report puts the global KV-cache cut at 37.5%. So the split KV cost looks like:

Slide the context up and watch which term dominates — the sliding window is the structural win, values=keys shaves what's left:

The payoff shows up in the real memory table. At 32k context the int8 KV cache adds only +0.05 GB (E2B), +0.28 GB (12B / 26B-A4B), +1.10 GB (31B) on top of the quantized weights — small enough that the weights, not the cache, stay the budget:

The quantized column is quantization-aware training (QAT), not a post-hoc round: the model trains with fake-quant in the loop so int4/int8 weights land with minimal quality loss. That's how 31B fits in ~19 GB.

The encoder-free 12B

The most unusual model is the 12B, trained from scratch with no vision or audio encoder at all. Normally a multimodal LLM bolts a frozen ViT and a speech encoder onto the token stream. Gemma 4 12B replaces them with projections:

• Vision: it takes raw 48×48×3 RGB patches and projects them with a single 35M-parameter matmul — standing in for the 550M ViT the larger models use. 2D coordinate embeddings and a LayerNorm carry spatial position.
• Audio: the 305M USM conformer encoder is discarded entirely. Raw audio is cut into 40ms chunks at 16kHz (640-dim vectors) and projected straight into the embedding space. Audio is already temporal, so no extra positional encoding.

The bet: give a large enough LLM the raw patches and it learns the encoder's job internally, for less memory and no separate frozen tower to serve. The report's Table 8 argues the 12B stays competitive on audio-text tasks without the dedicated encoder — a genuinely different design point from the encoder-plus-LLM norm, and the honest caveat is that it only did this for one size.

For the models that keep a vision encoder, the input pipeline is worth a look too. Gemma 4 supports variable aspect ratios rather than square-cropping, resizing an image to fit a token budget $N_{\max} \in \{70, 140, 280, 560, 1120\}$ while mostly preserving shape:

expand

Thinking mode

Gemma 4 adds a thinking mode: before answering, the model can emit a reasoning trace, which lifts math and coding. It's a post-training addition on top of a Gemma 3-style recipe, toggled by a control token in a leading system turn:

<|think|>              # activates the reasoning trace for this turn
...user turn...
# IT models close a turn with <turn|>; base (PT) models emit <eos>

Because the trace is optional, you pay for it only on the prompts that need it. The flip side, for anyone reading the tables: the Gemma 3 27B baseline is non-thinking, so a row like AIME (89.2 vs 20.8) is partly measuring the mode, not only the model.

MTP drafter: speculative decoding without prefill

Decoding is memory-bandwidth-bound — one token per forward pass, weights re-read each step. The usual fix is speculative decoding: a small draft model proposes several tokens, the big model verifies them in one pass, and accepted tokens are free. Gemma 4 ships a multi-token-prediction (MTP) drafter head for exactly this.

expand

The drafter is a 4-layer Transformer block (model dim 256 for E2B/E4B, 1024 for 26B-A4B/31B; three local and one global attention layers). It reuses the main model's last-layer activations and cross-attends to the main model's KV cache, so it needs no prefill of its own and supports any draft length. On E2B/E4B there's a further trick: instead of projecting the draft over the full 262k vocabulary, it does a top-k over token clusters, cutting the final matmul from $d \times 262\text{,}000$ to $d \times 4096$ at a similar acceptance rate. The report doesn't publish an end-to-end speedup, so I won't invent one — but the design (no prefill, cross-attention to the live cache) is the part worth copying.

If speculative decoding and MTP are new, I built the idea up from scratch in Multi-Token Prediction.

The numbers

Start with human eval, since it's the least gameable. On Arena Text (blind side-by-side, Elo, as of June 2026), Gemma 4 31B is the top dense open model — but the leaderboard around it is mostly much larger MoE systems, and a closed model tops it:

An Elo of 1451 at 31B dense against 744B–1.6T MoE models a dozen ranks up is the real story: it's punching well above its parameter count, not topping the board. On static reasoning benchmarks the family scales cleanly, and the jump over Gemma 3 27B is large (thinking-mode caveat noted):

The rest of the text suite tells the same story — the 31B leads its own family, the 26B-A4B tracks close behind at a seventh of the active params, and both clear Gemma 3 27B by a wide margin:

Vision holds up (MMMU Pro 76.9 / MATH-Vision 85.6 / InfographicVQA 92.0 for 31B at full resolution), and long context does what the KV work promises — RULER at 128k stays high where Gemma 3 27B falls off:

The take

Gemma 4's contribution isn't a benchmark crown — it's efficiency engineering shipped in the open. The pieces compose: a 5:1 local:global ratio and values=keys on global layers keep the KV cache flat enough that a 128k prompt costs single-digit gigabytes of cache; QAT puts 31B in ~19 GB; the encoder-free 12B deletes two frozen towers; the MTP drafter does speculative decoding without a second prefill. None of these is individually novel, but the combination is a serious on-device and single-accelerator story, released under Apache 2.0 with a 31B dense model that is the top open dense entry on Arena.

The honest caveats are the first-party ones. The benchmarks are Google's own, thinking mode is on for Gemma 4 and off for the Gemma 3 baseline it's measured against, and "leading open model" is true only in the dense category — larger open MoE models (GLM 5.1, DeepSeek V4) rank above it on the same board. The encoder-free design is proven at exactly one size (12B), and there's no published MTP speedup to hold them to. For a team that wants an open, multimodal, long-context model that fits the hardware it already owns, the 12B and 31B are the ones I'd reach for — and the KV-cache design is the part I'd study regardless of which model I ended up serving.

Built on the Gemma 4 Technical Report (Gemma Team, Google DeepMind, 2026), Apache 2.0 model license. All benchmark numbers are first-party from the report (thinking mode unless noted; the Gemma 3 27B baseline is non-thinking). The interactive diagrams are schematic illustrations of the mechanism, not measured traces; the two paper figures are reproduced for commentary.