# Bonsai 27B: a 27B model at 1.125 bits, small enough for a phone
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/bonsai-27b
> date: 2026-07-15
> tags: quantization, inference-optimization, on-device, multimodal, explainer
Most of the work that makes a large language model *usable* on your own hardware is not a
better model — it's a smaller one that behaves like the big one. **Bonsai 27B**, from PrismML,
is a clean example: it takes a full-precision **Qwen3.6-27B** and re-encodes its weights at
close to one bit each, ending up small enough to load inside a phone's memory budget. The
intelligence is Qwen's. What Bonsai contributes is the **extreme low-bit representation** —
run end to end, not just on the easy layers — and the custom kernels that make it fast.
The headline is the collapse in size. A 27B model at FP16 is 54 GB; Bonsai ships two quantized
variants, and the smaller one is **3.9 GB** — "27B-class capability at a footprint smaller than
a full-precision 2B model," as PrismML puts it. Flip between the precisions:
## Two encodings, close to one bit each
The two variants differ only in how each weight is stored. The **ternary** model uses the
three-value set `{−1, 0, +1}` with an FP16 scale shared across a group of weights — that works
out to **1.71 effective bits per weight** and a 5.9 GB model. The **1-bit** model drops the
zero, storing `{−1, +1}` plus the group scale, for **1.125 bits** and 3.9 GB. (The theoretical
floors are log₂3 ≈ 1.58 and 1.0 bits; the group-wise scales are the small overhead on top.)
Everything else — the hybrid-attention architecture, the 262K-token context window, the
Apache-2.0 license — is inherited from the base.
## The hard part: every block, not just the MLPs
Quantizing a transformer to a couple of bits is not new. What usually happens is that the
*sensitive* parts — the token embeddings, the attention projections, the LM head — are kept at
higher precision, and only the big feed-forward MLPs get squeezed. That protects quality, but
it also means the footprint only partly shrinks: a model is not small until its embeddings and
head are small too. Bonsai's claim is that the low-bit representation "runs end to end across
the language network, embeddings, attention, MLPs, and the LM head," with a compact **4-bit
vision tower** alongside. Toggle between the two philosophies:
Pushing 1-bit weights through the parts everyone else keeps in FP16 is exactly where accuracy
usually falls off a cliff — which is why the interesting question is not the size, but what it
costs. This is the *inference-time* mirror image of [native FP4 training](/articles/nemotron-nvfp4):
there the goal was to keep the math stable during training while deliberately holding some layers
higher precision; here the goal is to serve an already-trained model with nothing held back. If
you want the mechanics of why low-precision inference is memory-bound in the first place, the
[how LLM inference works](/articles/how-llm-inference-works) piece sets that up, and
[TurboQuant](/articles/turboquant-kv-cache) covers the complementary problem of quantizing the KV
cache rather than the weights.
## What survives — and what doesn't
Here is the honest part, and it's the part a size-and-speed announcement tends to bury. PrismML
reports that ternary keeps **~95%** of full-precision quality and 1-bit keeps **~90%**, averaged
over a 15-benchmark suite in thinking mode. Both averages check out against their own table — but
the average hides a wide spread. Pick a category and watch the three precisions, then read the
per-category retention strip:
Math is remarkably robust: the 1-bit model holds ~96% of the full-precision score. But
**agentic tool-calling** falls from 80.0 to 66.0 and **vision** from 72.6 to 59.6 — roughly 82%
retention each, nearly a fifth of the capability gone. Long, multi-step tool use and multimodal
perception are precisely the workloads that lean on the fine-grained information that one-bit
weights throw away. The overall number:
## On the device
The point of all this is where it runs. Bonsai reports up to **163 tok/s** for the 1-bit variant
on an RTX 5090 (134 for ternary), and up to **87 tok/s** on an Apple M5 Max (58 for ternary) — and,
the flashiest claim, that the 3.9 GB model fits inside an iPhone 17 Pro's app-memory budget, making
it "the first 27B-class model to run on a phone." PrismML frames this as *intelligence density* — a
coined score-per-GB metric on which the 1-bit model scores 0.53/GB, which they call more than 10×
the full-precision baseline. It ships with weights on Hugging Face, an MLX path for Apple silicon and
CUDA for NVIDIA, and speculative-decoding support for lossless draft-and-verify acceleration.
Read the numbers for what they are. Bonsai's **capability is Qwen3.6-27B's** — this is a compression
and kernels result, not a new model. Every score above is **vendor-reported on PrismML's own
15-benchmark suite in "thinking mode,"** so treat the suite and mode as chosen, not neutral. The
"~90–95% retained" headline is a real average that **masks much larger, uneven drops**: math barely
moves, but agentic tool-calling and vision lose ~18% at 1-bit — so the right variant depends entirely
on your workload. "Intelligence density," "first 27B on a phone," and "10×" are marketing framings
(intelligence-density is a coined score-per-GB metric), and the throughput figures are specific to an
RTX 5090 and an M5 Max. No independent evaluation exists yet.
## The takeaway
Bonsai is a bet that for a large slice of real use — on-device assistants, privacy-sensitive tasks,
hybrid deployments that route only the hard cases to a frontier API — a model that keeps 90% of a 27B's
quality while fitting in 3.9 GB beats a bigger model you can't run locally at all. That bet is strongest
where quality degrades gracefully (math, general reasoning) and weakest where it doesn't (agentic, vision).
The genuinely impressive engineering is the end-to-end part: getting one-bit embeddings and a one-bit LM
head to work is what turns "quantized MLPs" into a model that actually fits on the phone in your pocket.