2026-07-24 · 10 min · quantization · ternary · performance
Why read this
Nichelast 15%A readable tour of ternary QAT on a 15M TinyStories model: signed accumulation, the STE in five lines, and why the FP32 embedding is most of the 43 MB.
- Runs on a laptop CPU
- A lasting reference
- Explained from first principles
Quantization & compressionMITIntro tool
How this was scored
- Is it new?
- 0 of 3: Repackaging or news of a known thing
- Can I trust it?
- 1 of 3: Spot-checks a few numbers
- Can I run it?
- 3 of 3: Open, permissive, runs on reader hardware with instructions
- Will I understand it?
- 2 of 3: Mechanism from first principles with figures
- Can I act on it?
- 1 of 3: General advice
- Will it last?
- 2 of 3: A reference for a year or more
- Does it affect many?
- 0 of 3: A niche paper or one-off repo
- Only here?
- 1 of 3: Some original analysis
Score 42 of 100, ranked 522 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored
Most quantization is an afterthought: train a model in FP16, then squeeze the weights down to 8 or 4 bits for
deployment and hope the accuracy survives. Ternary15M does the opposite,
and takes it to the extreme. It's a 15.19M-parameter, Llama-style language model where every linear layer is
ternary — each weight is one of exactly three values, −1, 0, or +1, scaled by a single per-channel number.
The model is trained that way from scratch, not clipped down after the fact. It's a tiny, readable member of the
BitNet b1.58 family, and it's a clean place to see why three-valued weights are
such an interesting bet.
The architecture is deliberately small and standard: dim 288, 6 layers, 6 query / 6 KV heads, a SwiGLU MLP with hidden size 768, a 32k vocab, and a 256-token context. All 42 of its attention and feed-forward linear layers (6 layers × 7 projections) are ternary; only the embedding table and the RMSNorm gains stay in full precision.
The whole trick is the sign
Here's why anyone cares about {−1, 0, +1} specifically. A neural network is, underneath, a pile of dot products:
each output is a weighted sum of its inputs. In full precision, every one of those weights is an arbitrary float, so
every term in the sum is a hardware multiply. Multiplies are the expensive part — they dominate the energy and the
silicon area of a matmul.
Now make each weight a sign instead of a number. If the weight is +1, the term is just the input — add it. If
it's −1, subtract the input. If it's 0, the input contributes nothing — skip it. The multiplies vanish; the
inner loop of the dot product becomes signed accumulation over the inputs. Toggle between the two below and watch the
multiply count collapse:
That is also how BitNet b1.58, the paper this model descends from, draws the
change. One row of a ternary weight matrix against an input vector stops being a string of multiply-then-add and
becomes x₀ − x₁ − x₂ + x₃ — the same arithmetic the toggle above counts:
The single multiply that survives is the per-output-channel scale — one number s that rescales the whole
accumulated sum back to a sensible magnitude. So a ternary matmul is: signed adds across the row, then one multiply at
the end. On general-purpose GPUs the win is mostly memory-bandwidth (you move far fewer weight bytes); on hardware
designed for it, killing the multiplies is the point. Either way, the arithmetic is genuinely different from
"small floats." BitNet's own accounting puts a number on how much of a matmul's energy the multiply is:
Two things to keep straight about that figure. The 71.4× is arithmetic energy only — the multiply-and-add units, not the memory traffic — which is why the end-to-end numbers on the right are an order of magnitude smaller than it. And none of this was measured on Ternary15M: these are BitNet's projections for 1.3B-to-70B models at a 7nm node, and they are the reason anyone builds ternary models, not evidence about the 15M one below.
1.58 bits, and where the bytes actually go
Three states carry log₂(3) ≈ 1.58 bits of information each — that's where the "b1.58" name comes from, and why a
ternary weight is often quoted as costing ~1.58 bits versus 16 or 32 for a float. Ternary15M doesn't bit-pack that
tightly; its exported checkpoint stores each ternary weight as an int8 (one byte) plus one FP32 scale per output
channel, which the author notes compresses to roughly 2 bits with basic entropy coding. The latent training checkpoint
is 182 MB; the deployed ternary model is 43 MB.
But 43 MB is bigger than 1.58 bits × 15M would suggest, and the reason is worth sitting with:
Teaching a network to live with three values
You can't train ternary weights directly, because rounding to {−1, 0, +1} has a gradient of zero almost everywhere —
nudging a latent weight from 0.31 to 0.32 doesn't change the rounded output, so ordinary backprop would see no signal
and learn nothing. Quantization-aware training gets around this with two ideas working together.
First, the scale. Each output channel's weights are ternarized around their own average magnitude — the mean of the
absolute weights in that row, absmean(W). Dividing by that scale before rounding is what decides which weights round
to ±1 and which collapse to 0, and multiplying the scale back afterward keeps the layer's outputs at roughly the
right size. It's computed per output channel, so every neuron sets its own threshold.
Second, the straight-through estimator (STE). The forward pass uses the ternarized weights — so the network actually experiences quantization while it learns — but the backward pass pretends the rounding was the identity function and passes the gradient straight through to a full-precision latent copy of the weights. Those FP32 latents are what the optimizer updates; the ternary weights are re-derived from them every forward pass. In PyTorch the whole thing is a few lines:
def forward(self, x: torch.Tensor) -> torch.Tensor:
weight = self.weight # FP32 latent weights
scale = weight.abs().mean(dim=1, keepdim=True) # absmean per output channel
safe_scale = scale.clamp_min(torch.finfo(weight.dtype).eps)
qweight = torch.round(torch.clamp(weight / safe_scale, -1, 1)) * scale
weight_ste = weight + (qweight - weight).detach() # STE: value = qweight, grad → weight
return F.linear(x, weight_ste)The weight + (qweight - weight).detach() line is the STE in one expression: numerically it equals qweight (the term
you subtract is detached from the graph), but its gradient with respect to weight is 1, so the optimizer trains the
latent weights as if the quantizer weren't there. At export time the latents are thrown away and the weights are frozen
to int8 in {−1, 0, +1} plus the FP32 scales:
def ternary_components(weight: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
"""Return int8 {-1, 0, 1} weights and FP32 per-output scales."""
scale = weight.detach().float().abs().mean(dim=1, keepdim=True)
safe_scale = scale.clamp_min(torch.finfo(torch.float32).eps)
qweight = torch.round(torch.clamp(weight.detach().float() / safe_scale, -1, 1))
return qweight.to(torch.int8), scaleDoes it work?
The honest, useful result is that at this scale the quantization is nearly free. Trained on TinyStories (a synthetic corpus of simple children's stories) for 655M tokens, the model's validation loss barely moves when you go from the latent weights to the hard-ternary ones:
The bars are almost identical on purpose — that is the finding. Freezing the network to pure ternary weights costs only +0.01 loss over the latent model it was trained as. The quantization the model trained under is the same one it ships with, so there's no distribution shift at export. And the deployed artifact is a fraction of the size:
The whole run cost about $0.70 — ~50 minutes on a single L40S at ~200k tokens/second. That cheapness is a feature: it makes ternary QAT something you can actually reproduce and poke at, not a claim you take on faith.
What a 15M ternary model can and can't do
It's worth being blunt about the ceiling. TinyStories exists precisely so that tiny models can learn something coherent: the vocabulary and grammar are simple, the stories are short, and 256 tokens of context is plenty. Within that box, Ternary15M does the job — it generates grammatical, on-topic little stories, and it does so from weights that are almost entirely signs. That's the point of the artifact.
What it can't do is everything a real LM does. There's no world knowledge, no reasoning, no code, no long context, no instruction following — 15M parameters and a children's-story corpus don't reach any of that, and ternary quantization doesn't change the ceiling in either direction. The value here isn't the model's outputs; it's that the training recipe — BitLinear layers, an absmean scale, an STE, and a from-scratch schedule — demonstrably works end to end and lands within a hundredth of a nat of its full-precision-latent self.
The take
Ternary15M is a good teaching artifact for a genuinely surprising idea: you can restrict every weight in a network to one of three values and, if you train it that way rather than clipping after the fact, pay almost nothing in loss. The mechanism is clean — signs replace floats, so dot products become signed accumulation with one scale multiply per channel; the STE lets gradients flow to a latent copy the quantizer hides; the absmean scale keeps magnitudes sane. The honest caveats are that this is a 15M model on TinyStories with self-reported numbers, and that at this scale the FP32 embedding table, not the ternary matmuls, dominates the file size. But as a from-scratch, $0.70, reproducible window into how BitNet-style quantization actually works, it's about as legible as this idea gets.
Source: the Ternary15M repository (Brian Bell, MIT license) — its
README, MODEL_CARD.md, RESULTS.md, and ternary15m/model.py. The lineage is
BitNet b1.58 (Ma et al.). Code snippets are from the repo; the interactive diagram
is mine. The repo itself ships no figures, so the two reproduced above are BitNet b1.58's own (Figures 1 and 3) and
describe that paper's models, not this one — every number quoted about Ternary15M comes from its RESULTS.md and model
card.