2026-07-15 · 5 min · quantization · inference-optimization · on-device · multimodal · explainer
Most of the work that makes a large language model usable on your own hardware is not a better model — it's a smaller one that behaves like the big one. Bonsai 27B, from PrismML, is a clean example: it takes a full-precision Qwen3.6-27B and re-encodes its weights at close to one bit each, ending up small enough to load inside a phone's memory budget. The intelligence is Qwen's. What Bonsai contributes is the extreme low-bit representation — run end to end, not just on the easy layers — and the custom kernels that make it fast.
The headline is the collapse in size. A 27B model at FP16 is 54 GB; Bonsai ships two quantized variants, and the smaller one is 3.9 GB — "27B-class capability at a footprint smaller than a full-precision 2B model," as PrismML puts it. Flip between the precisions:
The same 27B weights, re-encoded. Ternary spends 1.71 bits each ( {−1, 0, +1} plus an FP16 per-group scale), the 1-bit variant just 1.125 ( {−1, +1}plus a scale) — collapsing a 54 GB model to 3.9 GB, small enough to live inside a phone's app-memory budget. The capability is still Qwen3.6-27B's; what changes is whether it fits.
Two encodings, close to one bit each
The two variants differ only in how each weight is stored. The ternary model uses the
three-value set {−1, 0, +1} with an FP16 scale shared across a group of weights — that works
out to 1.71 effective bits per weight and a 5.9 GB model. The 1-bit model drops the
zero, storing {−1, +1} plus the group scale, for 1.125 bits and 3.9 GB. (The theoretical
floors are log₂3 ≈ 1.58 and 1.0 bits; the group-wise scales are the small overhead on top.)
Everything else — the hybrid-attention architecture, the 262K-token context window, the
Apache-2.0 license — is inherited from the base.
The hard part: every block, not just the MLPs
Quantizing a transformer to a couple of bits is not new. What usually happens is that the sensitive parts — the token embeddings, the attention projections, the LM head — are kept at higher precision, and only the big feed-forward MLPs get squeezed. That protects quality, but it also means the footprint only partly shrinks: a model is not small until its embeddings and head are small too. Bonsai's claim is that the low-bit representation "runs end to end across the language network, embeddings, attention, MLPs, and the LM head," with a compact 4-bit vision tower alongside. Toggle between the two philosophies:
Bonsai runs the low-bit representation through every block — embeddings, attention, MLPs and the LM head — with a 4-bit vision tower alongside. Nothing is left at FP16, which is what lets the whole model reach 3.9 GB. Compare with the mixed-precision approach of native FP4 training, which deliberately keeps some layers higher precision.
Pushing 1-bit weights through the parts everyone else keeps in FP16 is exactly where accuracy usually falls off a cliff — which is why the interesting question is not the size, but what it costs. This is the inference-time mirror image of native FP4 training: there the goal was to keep the math stable during training while deliberately holding some layers higher precision; here the goal is to serve an already-trained model with nothing held back. If you want the mechanics of why low-precision inference is memory-bound in the first place, the how LLM inference works piece sets that up, and TurboQuant covers the complementary problem of quantizing the KV cache rather than the weights.
What survives — and what doesn't
Here is the honest part, and it's the part a size-and-speed announcement tends to bury. PrismML reports that ternary keeps ~95% of full-precision quality and 1-bit keeps ~90%, averaged over a 15-benchmark suite in thinking mode. Both averages check out against their own table — but the average hides a wide spread. Pick a category and watch the three precisions, then read the per-category retention strip:
Averaged over everything, the 1-bit model keeps ~90% of full precision and ternary ~95%— but that average is the story's villain. Math survives almost intact (~96% at 1-bit), while agentic tool-calling and visionshed ~18% each. If your workload is the reasoning that quantizes gracefully, this is a great trade; if it's multi-step tool use, read the fine print.
Math is remarkably robust: the 1-bit model holds ~96% of the full-precision score. But agentic tool-calling falls from 80.0 to 66.0 and vision from 72.6 to 59.6 — roughly 82% retention each, nearly a fifth of the capability gone. Long, multi-step tool use and multimodal perception are precisely the workloads that lean on the fine-grained information that one-bit weights throw away. The overall number:
On the device
The point of all this is where it runs. Bonsai reports up to 163 tok/s for the 1-bit variant on an RTX 5090 (134 for ternary), and up to 87 tok/s on an Apple M5 Max (58 for ternary) — and, the flashiest claim, that the 3.9 GB model fits inside an iPhone 17 Pro's app-memory budget, making it "the first 27B-class model to run on a phone." PrismML frames this as intelligence density — a coined score-per-GB metric on which the 1-bit model scores 0.53/GB, which they call more than 10× the full-precision baseline. It ships with weights on Hugging Face, an MLX path for Apple silicon and CUDA for NVIDIA, and speculative-decoding support for lossless draft-and-verify acceleration.
The takeaway
Bonsai is a bet that for a large slice of real use — on-device assistants, privacy-sensitive tasks, hybrid deployments that route only the hard cases to a frontier API — a model that keeps 90% of a 27B's quality while fitting in 3.9 GB beats a bigger model you can't run locally at all. That bet is strongest where quality degrades gracefully (math, general reasoning) and weakest where it doesn't (agentic, vision). The genuinely impressive engineering is the end-to-end part: getting one-bit embeddings and a one-bit LM head to work is what turns "quantized MLPs" into a model that actually fits on the phone in your pocket.