Ternary15M: a language model where every weight is −1, 0, or +1

Most quantization is an afterthought: train a model in FP16, then squeeze the weights down to 8 or 4 bits for deployment and hope the accuracy survives. Ternary15M does the opposite, and takes it to the extreme. It's a 15.19M-parameter, Llama-style language model where every linear layer is ternary — each weight is one of exactly three values, −1, 0, or +1, scaled by a single per-channel number. The model is trained that way from scratch, not clipped down after the fact. It's a tiny, readable member of the BitNet b1.58 family, and it's a clean place to see why three-valued weights are such an interesting bet.

The architecture is deliberately small and standard: dim 288, 6 layers, 6 query / 6 KV heads, a SwiGLU MLP with hidden size 768, a 32k vocab, and a 256-token context. All 42 of its attention and feed-forward linear layers (6 layers × 7 projections) are ternary; only the embedding table and the RMSNorm gains stay in full precision.

The whole trick is the sign

Here's why anyone cares about {−1, 0, +1} specifically. A neural network is, underneath, a pile of dot products: each output is a weighted sum of its inputs. In full precision, every one of those weights is an arbitrary float, so every term in the sum is a hardware multiply. Multiplies are the expensive part — they dominate the energy and the silicon area of a matmul.

Now make each weight a sign instead of a number. If the weight is +1, the term is just the input — add it. If it's −1, subtract the input. If it's 0, the input contributes nothing — skip it. The multiplies vanish; the inner loop of the dot product becomes signed accumulation over the inputs. Toggle between the two below and watch the multiply count collapse:

The single multiply that survives is the per-output-channel scale — one number s that rescales the whole accumulated sum back to a sensible magnitude. So a ternary matmul is: signed adds across the row, then one multiply at the end. On general-purpose GPUs the win is mostly memory-bandwidth (you move far fewer weight bytes); on hardware designed for it, killing the multiplies is the point. Either way, the arithmetic is genuinely different from "small floats."

1.58 bits, and where the bytes actually go

Three states carry log₂(3) ≈ 1.58 bits of information each — that's where the "b1.58" name comes from, and why a ternary weight is often quoted as costing ~1.58 bits versus 16 or 32 for a float. Ternary15M doesn't bit-pack that tightly; its exported checkpoint stores each ternary weight as an int8 (one byte) plus one FP32 scale per output channel, which the author notes compresses to roughly 2 bits with basic entropy coding. The latent training checkpoint is 182 MB; the deployed ternary model is 43 MB.

But 43 MB is bigger than 1.58 bits × 15M would suggest, and the reason is worth sitting with:

Teaching a network to live with three values

You can't train ternary weights directly, because rounding to {−1, 0, +1} has a gradient of zero almost everywhere — nudging a latent weight from 0.31 to 0.32 doesn't change the rounded output, so ordinary backprop would see no signal and learn nothing. Quantization-aware training gets around this with two ideas working together.

First, the scale. Each output channel's weights are ternarized around their own average magnitude — the mean of the absolute weights in that row, absmean(W). Dividing by that scale before rounding is what decides which weights round to ±1 and which collapse to 0, and multiplying the scale back afterward keeps the layer's outputs at roughly the right size. It's computed per output channel, so every neuron sets its own threshold.

Second, the straight-through estimator (STE). The forward pass uses the ternarized weights — so the network actually experiences quantization while it learns — but the backward pass pretends the rounding was the identity function and passes the gradient straight through to a full-precision latent copy of the weights. Those FP32 latents are what the optimizer updates; the ternary weights are re-derived from them every forward pass. In PyTorch the whole thing is a few lines:

def forward(self, x: torch.Tensor) -> torch.Tensor:
    weight = self.weight                                   # FP32 latent weights
    scale = weight.abs().mean(dim=1, keepdim=True)         # absmean per output channel
    safe_scale = scale.clamp_min(torch.finfo(weight.dtype).eps)
    qweight = torch.round(torch.clamp(weight / safe_scale, -1, 1)) * scale
    weight_ste = weight + (qweight - weight).detach()      # STE: value = qweight, grad → weight
    return F.linear(x, weight_ste)

The weight + (qweight - weight).detach() line is the STE in one expression: numerically it equals qweight (the term you subtract is detached from the graph), but its gradient with respect to weight is 1, so the optimizer trains the latent weights as if the quantizer weren't there. At export time the latents are thrown away and the weights are frozen to int8 in {−1, 0, +1} plus the FP32 scales:

def ternary_components(weight: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
    """Return int8 {-1, 0, 1} weights and FP32 per-output scales."""
    scale = weight.detach().float().abs().mean(dim=1, keepdim=True)
    safe_scale = scale.clamp_min(torch.finfo(torch.float32).eps)
    qweight = torch.round(torch.clamp(weight.detach().float() / safe_scale, -1, 1))
    return qweight.to(torch.int8), scale

Does it work?

The honest, useful result is that at this scale the quantization is nearly free. Trained on TinyStories (a synthetic corpus of simple children's stories) for 655M tokens, the model's validation loss barely moves when you go from the latent weights to the hard-ternary ones:

The bars are almost identical on purpose — that is the finding. Freezing the network to pure ternary weights costs only +0.01 loss over the latent model it was trained as. The quantization the model trained under is the same one it ships with, so there's no distribution shift at export. And the deployed artifact is a fraction of the size:

The whole run cost about $0.70 — ~50 minutes on a single L40S at ~200k tokens/second. That cheapness is a feature: it makes ternary QAT something you can actually reproduce and poke at, not a claim you take on faith.

What a 15M ternary model can and can't do

It's worth being blunt about the ceiling. TinyStories exists precisely so that tiny models can learn something coherent: the vocabulary and grammar are simple, the stories are short, and 256 tokens of context is plenty. Within that box, Ternary15M does the job — it generates grammatical, on-topic little stories, and it does so from weights that are almost entirely signs. That's the point of the artifact.

What it can't do is everything a real LM does. There's no world knowledge, no reasoning, no code, no long context, no instruction following — 15M parameters and a children's-story corpus don't reach any of that, and ternary quantization doesn't change the ceiling in either direction. The value here isn't the model's outputs; it's that the training recipe — BitLinear layers, an absmean scale, an STE, and a from-scratch schedule — demonstrably works end to end and lands within a hundredth of a nat of its full-precision-latent self.

The take

Ternary15M is a good teaching artifact for a genuinely surprising idea: you can restrict every weight in a network to one of three values and, if you train it that way rather than clipping after the fact, pay almost nothing in loss. The mechanism is clean — signs replace floats, so dot products become signed accumulation with one scale multiply per channel; the STE lets gradients flow to a latent copy the quantizer hides; the absmean scale keeps magnitudes sane. The honest caveats are that this is a 15M model on TinyStories with self-reported numbers, and that at this scale the FP32 embedding table, not the ternary matmuls, dominates the file size. But as a from-scratch, $0.70, reproducible window into how BitNet-style quantization actually works, it's about as legible as this idea gets.

Source: the Ternary15M repository (Brian Bell, MIT license) — its README, MODEL_CARD.md, RESULTS.md, and ternary15m/model.py. The lineage is BitNet b1.58 (Ma et al.). Code snippets are from the repo; the interactive diagram is mine. The repo ships no figures, so there are none to reproduce here — the numbers above are quoted from its RESULTS.md and model card.