Bonsai 2 27B: PrismML's sequel gets to 98.2% — on average

Two months after Bonsai 27B, PrismML has shipped Bonsai 2 27B — same idea, same rough footprint, a stronger base model, and a headline retention number that climbed from "~95%" to "98.2%." The size story barely moved: still a ternary {−1, 0, +1} weight format with an FP16 group scale, still around 5.9-6 GB depending on which file you count, still Apache 2.0. What changed is the base (Qwen3.6-27B → Qwen3.8-27B), the per-tensor precision map (very slightly, and in the opposite direction from what you'd expect), and — for the first time — real numbers on the long-horizon agentic benchmarks the first release's roadmap promised to chase. I pulled the safetensors/GGUF metadata for every repo in the Bonsai-2 collection, read the whitepaper end to end, and diffed its numbers against the model card's. They don't always agree with each other, which turns out to be the most useful fact in this release.

What actually changed from the first release

The first Bonsai 27B pushed its ternary and 1-bit encodings through every block — embeddings, attention, MLPs, the LM head — and made a selling point of having "no high-precision escape hatches." Bonsai 2 27B does something slightly different: it pulls back, by a fraction of a percent, and keeps a small set of tensors — the recurrent state path of the linear-attention layers (in_proj_a, in_proj_b, conv1d) plus the RMSNorm and Q/K-norm weights — in full precision. It's 26,238,464 parameters, 0.0976% of the language model, 52 MB at bf16. Toggle between parameter share and byte share to see why that tiny exception (and the separate vision tower) matter more than their parameter count suggests:

The honest read: Bonsai 2 27B is still about as end-to-end as a ternary model gets — 98%+ of parameters by count — but "end to end, no exceptions" was gen 1's pitch, not gen 2's. PrismML spent 0.01 bits/weight to buy back whatever quality that recurrent state path was costing at ternary. It's a small, specific concession, and the whitepaper is upfront that it's new.

The 5.9 GB number, audited

The brief version of the claim is "5.9 GB, down from 54 GB, roughly 9x." Here's what that collapses when you look at the actual bytes instead of the rounded headline. A 27B model at FP16 really is 53.81 GB — that's the exact size of Ternary-Bonsai-2-27B-F16.gguf, sitting in the same repo as a reference build. A ternary weight carries log2(3) ≈ 1.585 bits of pure information; on ~26.86B ternarizable parameters that's about 5.32 GB with no overhead at all. Add back the FP16 group scale (one per 128 weights) and PrismML's own "true ternary" figure is 1.72 bits/weight, 5.80 GB. Then add the packing overhead that an actual, loadable file needs — PrismML ships two: PTQ1_0 (dense trits, 5.95 GB on disk) and PQ2_0 (each trit in its own 2-bit slot, 7.21 GB). 5.95 GB is not a rounding error — it's the real file, three layers of overhead past the theoretical floor: group scales, then the 0.0976% full-precision exceptions, then the packing format's own cost.

You can see where that last layer goes without downloading a single weight. The MLX pack ships its own loader, and runtime/codec.py is the part that reads a packed GGUF block:

sizes = {"PQ2_0": 34, "PTQ1_0": 28}
if source not in sizes:
    raise ValueError(f"Unsupported format: {source}")
blocks = rows * width // 128
if len(raw) != blocks * sizes[source]:
    raise ValueError("Raw byte length does not match shape and format")
data = np.frombuffer(raw, dtype=np.uint8).reshape(blocks, sizes[source])
scale_bytes = data[:, :2] if source == "PQ2_0" else data[:, 26:28]
scales = scale_bytes.copy().view("<f2").reshape(rows, width // 128)
# ...
return words, np.ascontiguousarray(scales), np.ascontiguousarray(-scales)

That dictionary is the whole size story in two integers. A PTQ1_0 block is 28 bytes per 128 weights — 1.75 bits/weight, exactly. A PQ2_0 block is 34 bytes: 32 bytes of 2-bit slots plus the FP16 group scale, 2.125 bits/weight. Those are the model card's numbers (1.75 and 2.13). The whitepaper says 1.76 and 2.16 instead, because it is rating the shipped file rather than the format. The files settle it: 5,946,648,928 and 7,206,168,928 bytes on the Hub, which is 1.77 and 2.14 bits/weight over the 26.89B language-model parameters. The gap is small and it cuts both ways — the whitepaper's own storage table lists 5.93 GB and 7.25 GB, and neither matches what it uploaded, while the model card's 5.95 and 7.21 do.

The last line of that excerpt is also the neatest trick in the format. The bias handed to the kernel for every group is -scales: trits are stored as unsigned 2-bit codes 0, 1, 2 and reconstructed as (code − 1) × scale, which is how {−1, 0, +1} rides a stock affine 2-bit matmul with no ternary-specific dequantization path at all. The exotic part of this model is not the arithmetic.

None of those numbers include the vision tower, and "language weights" is the load-bearing phrase in every one of PrismML's own size claims. The tower ships as its own file — a Q8_0 container holding 4-bit HQQ weights, 0.63 GB, or a BF16 reference at 0.93 GB — over 460M parameters that were never ternarized and were never going to be. Add the smaller vision pack to the smaller language pack and the real download for a multimodal deployment is ~6.58 GB, not 5.9. That's still a large win over 54 GB. It just isn't the number in the highlights bullet.

98.2% of what, exactly

This is the number doing the most work in the launch post, and it's worth being precise about what it is an average of. PrismML publishes three different versions of "98.2%," and they disagree with each other on the absolute score:

• Launch post / whitepaper: 83.9 vs. 85.4, on a 20-benchmark suite (τ²-Bench, BFCL v3, HumanEval+, LiveCodeBench v6, MBPP+, BigCodeBench, IFEval, IFBench, MMLU-Redux, GPQA Diamond, AA-LCR, AIME25, AIME26, GSM8K, MATH-500, CharXiv, A-OKVQA, OmniDocBench v1.6, RealWorldQA, OCRBench v2), thinking mode, xhigh reasoning effort, against Qwen3.8-27B FP16.
• Model card: 84.78 vs. 86.32, on a different 14-benchmark suite that drops τ²-Bench, AA-LCR, BigCodeBench, and the four vision benchmarks above in favor of MuSR and MMMU-Pro, which don't appear in the 20-benchmark suite at all.

Both ratios round to 98.2%. Neither the suite nor the FP16 baseline score is the same number — 85.4 in one document, 86.32 in the other, for the identical Qwen3.8-27B FP16 model. That's not an error so much as a demonstration of the thing an aggregate is supposed to hide: pick a slightly different 15-20-benchmark bag and the same underlying model produces a different "% retained," and both bags happen to land on the same marketing number.

The full 20-benchmark table (thinking mode, xhigh) is the more complete of the two, and it's where the real spread lives:

Source: whitepaper Table 10, thinking mode, xhigh reasoning effort.

Math and instruction following barely move — AIME26 and IFBench actually score higher than Qwen3.8-27B FP16, and LiveCodeBench is a rounding error apart. The real losses cluster in exactly the categories a coarse "reasoning holds up" story tends to skip: OCR Bench v2 loses 6.7% of its FP16 score, BigCodeBench loses 5.6%, GPQA Diamond loses 5.2%. None of those individual drops is dramatic, but "98.2% aggregate" is quietly compatible with a 6.7% loss on any one benchmark that happens to get diluted by nineteen others that didn't move.

The bigger story isn't in this table at all. It's what the whitepaper reports in prose, a page away from the aggregate, and doesn't fold into either suite:

These are the two benchmarks PrismML says it evaluated "for the first time," specifically because the first Bonsai 27B's roadmap called out "long-horizon, tool-driven software engineering" as the next thing to fix. They're real evaluations, run through the Harbor framework with the Terminus-2 agent (89 tasks, single attempt) and the mini-swe-agent scaffold (500 instances) — not demo footage. And on the workload the release is most explicitly marketed at — "agentic coding," "long-horizon tool use" — the ternary model keeps roughly three-quarters of full precision, not 98.2%. Both numbers are true. They are not the same claim.

Memory vs. quality, against everything else at this size

Put Bonsai 2 27B next to the other ways to shrink a 27B model — full precision, a well-regarded conventional 4-bit GGUF, a conventional 2-bit GGUF, and the previous Bonsai release, all scored on PrismML's own 14-benchmark comparison table so the numbers are at least internally consistent:

Qwen3.8-27B IQ2_XXS is the useful control here: at 9.4 GB it is both bigger and worse than Bonsai 2 27B at 5.9 GB — a conventional low-bit build strictly dominated by a purpose-built one. (PrismML sizes that same third-party build two ways, too: 9.4 GB at 2.8 bpw on the model card, 7.3 GB at 2.2 bpw in the whitepaper. Bigger and worse either way, by less at 7.3.) UD-Q4_K_XL is the harder comparison and the one the highlights bullets skip: it beats Bonsai 2 27B by 0.4 points at three times the size, which is a real trade-off, not a rout — reach for it if that last half-point matters more to you than the footprint.

The intelligence-density chart, from the source

PrismML's own launch post makes the same point with a coined metric, "intelligence density" (-log2(1 - score/100) / size_GB), plotted against nine other models:

expand

Read the bars in order and the chart quietly undercuts its own headline: the tallest bar isn't today's release. It's 1-bit Bonsai 27B — the older, smaller, less capable model from the first launch — at 0.530/GB, ahead of the new "most capable model yet" at 0.444/GB. The formula divides by size before it multiplies by anything else, so a smaller, lower-quality model can win on "density" over a bigger, better one; PrismML's own chart is the proof. It's not a contradiction exactly — 1-bit Bonsai 27B really is smaller and the metric really does reward that — but it means "intelligence density" and "our most capable model" are two different superlatives, and the launch post's prose implies they're the same one.

The tok/s numbers, and the one I couldn't find

The launch post's headline throughput claims are 143 tokens/second on an RTX 5090 and 46.8 tokens/second on an Apple M5 Max, with no stated batch size or context length. The whitepaper's Appendix D fills that in: both are tg128 — token-generation throughput over 128 generated tokens, the memory-bandwidth-bound decode phase — at batch size 1, depth 0, with the vision tower excluded, on the PQ2_0 pack, measured on 2026-09-16. Prompt processing (pp512, the compute-bound phase) is a different and much larger number on the same hardware — 4121 tok/s on the RTX 5090 — which is the standard shape for batch-1 LLM inference and not specific to ternary weights, but it's worth knowing which phase "143 tok/s" describes before you compare it to anything.

It's also reproducible, which is the part I like about this release. The measurement is plain llama-bench, and the demo repo's own benchmark template gives the shape of the run — all layers offloaded, flash attention on — with Appendix D supplying the rest of the protocol (pp512/tg128, three runs after a warm-up pass):

BENCH=bin/cuda/llama-bench
$BENCH -m models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -p 512 -n 128 -r 3

Run it and you inherit PrismML's own problem, which is that its two documents disagree about what the answer should be. Whitepaper Table 5: 142.5 tok/s tg128 and 4121 tok/s pp512 for PQ2_0 on an RTX 5090. The model card's cross-platform table, same pack and same card: 129.9 and 3893 — about 9% and 6% lower — with energy quoted in J/token where the whitepaper quotes mWh/token. The PTQ1_0 row moves the same way (134.4 vs 120.5), and the M5 Pro row differs too (27.7 vs 28.1). These are clearly two different measurement runs — the hardware lists don't even match — but neither document dates the other or says which supersedes it, and the launch post promotes the higher pair. Treat "143 tok/s" as the top of a ~130-143 range on that card, not a figure to size a deployment against.

The brief for this piece asked about a third figure — 55 tok/s, WebGPU, M5 Max — and I could not find it in the launch post, the whitepaper, the model card, or the README of the WebGPU kernels demo space that PrismML links from the launch post's resources sidebar. That demo is real — it runs the model's ternary kernels in-browser over WebGPU — but its own README ships the Hub's generic template with no throughput number in it at all. The only published M5 Max figure I can verify is 46.8-47.0 tok/s, and it's measured through llama-bench on the native Metal backend, not a browser. If 55 tok/s exists somewhere, it isn't in any of PrismML's own primary sources for this release.

Ternary weights still need a runtime built for them

The memory win is real and it is not automatically a compute win. Two things make that concrete here. First, the weights aren't stored densely for cheap FP16 dequantization at inference time — they're stored in a rotated basis: each 1024-weight block is transformed by a fixed Walsh–Hadamard rotation before ternarization, and the runtime has to apply the matching transform to activations on every forward pass. At batch size 1 this rotation "lies on the critical path of every projection" (the whitepaper's words), and the fork's Metal and CUDA kernels spend real engineering fusing it into the load path specifically to keep it off that path.

That claim is checkable, because the MLX pack ships the loader that has to honor it. runtime/runtime.py puts the transform exactly where the whitepaper says it lives — in front of the matmul, every time:

def fwht(x, block, signs, inverse=False):
    shape, dtype = x.shape, x.dtype
    x = x.astype(mx.float32)
    if not inverse:
        x = x * signs
    x = mx.hadamard_transform(
        x.reshape(-1, block), scale=1 / math.sqrt(block)
    ).reshape(shape)
    if inverse:
        x = x * signs
    return x.astype(dtype)
 
 
class Packed(nn.Module):
    def __call__(self, x):
        # ... embedding path elided
        if self.block:
            x = fwht(x, self.block, self.signs)
        return mx.quantized_matmul(
            x, self.weight, self.scales, self.biases,
            transpose=True, group_size=128, bits=2,
        )

Sign flip, then a 1024-wide fast Walsh–Hadamard in float32, then the 2-bit matmul — per projection, per token. The embedding is the mirror image: it dequantizes the row first and applies the inverse transform to the result, which is why the GGUF ships a separate inverse_weight_names manifest containing exactly one tensor, token_embd.weight.

Second, the packed formats aren't something a stock runtime quietly mishandles — they're something it doesn't recognize at all. PQ2_0 and PTQ1_0 are PrismML's own ggml types, and mainline llama.cpp rejects both as unknown. The silent-corruption failure the model card warns about is narrower and more interesting: gen 1's ternary type, Q2_0, has landed in mainline, so a mainline build will cheerfully load a Q2_0 file, skip a Hadamard transform it knows nothing about, and produce garbage with no warning. MLX fails the same way from the other direction — the demo repo's own MLX script says stock MLX loaders "skip it and return wrong output rather than an error," which is why the model repo ships executable Python next to the weights. Bonsai 2's own GGUF at least defends itself: the rotation travels as metadata (prism.hadamard.block_size, sign_mode, weight_names), and the loader raises rather than guessing if any of it is missing. So: use the fork.

# prebuilt, pick the archive for your platform
# https://github.com/PrismML-Eng/llama.cpp/releases/latest
tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1
 
# or build it
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j    # drop -DGGML_CUDA=ON on macOS, Metal is default
 
./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -c 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 \
    -p "Explain quantum computing in simple terms." -n 256

Those are the model card's own quickstart commands — it prints ./bin/llama-cli for the prebuilt archive and ./build/bin/llama-cli if you built the fork yourself. That fork, the MLX fork, and the mlx-swift fork are all built around custom fused low-bit GEMM kernels that unpack trits and apply the group scale inside the matmul, with activations kept in higher precision throughout.

That both formats exist at all is the clearest evidence there's no free lunch here. PTQ1_0 packs trits densely and gets closer to the 1.72-bit floor, but unpacking costs arithmetic; PQ2_0 wastes bits on a 2-bit-per-trit slot but decodes cheaper. Which one wins is a hardware question, not a format question: PTQ1_0 is faster on Ada-class cards and the L4, PQ2_0 wins on Hopper, Ampere, Blackwell, and every Apple Silicon chip measured. A ternary weight format that needed no purpose-built kernel, and where one packing was uniformly faster than the other, would be the more surprising result.

Agentic claims: the demos and the evidence are two different things

The launch post's two video demos — a Cline coding-agent session and a "computer use" session, both on an RTX 5090 — are the most visible agentic evidence in the release, and they're exactly what they look like: unscored recordings of the model doing something, not a controlled comparison against anything. The real evidence for the agentic claims is the Terminal-Bench 2.1 and SWE-bench Verified numbers from two sections up — 75.8% and 75.4% retention, a genuine capability that a 5.9 GB model plausibly couldn't manage at all under naive quantization, but well short of "98.2% of full precision" for the workload the release is named after. Both things are true at once: this is a real improvement on the roadmap gen 1 set out, and the improvement is smaller than the release's own headline number for anything that resembles a coding agent running for more than a few turns.

License and the "9x" claim

Apache 2.0, unchanged from gen 1 — confirmed on every repo in the collection, and the actual LICENSE file matches. "9x smaller" checks out as a range rather than a single number: 9.0x for the PTQ1_0 file you'd actually deploy, 9.3x for PrismML's idealized "true ternary" rate that no packed file quite reaches, both against the real 53.81 GB FP16 reference sitting in the same repo. "More than 9x" is a fair thing to say about this model. It just isn't one number.

The takeaway

Bonsai 2 27B is a real, incremental win over the first release: a stronger base model, a genuinely improved (if still partial) answer to gen 1's agentic-coding roadmap item, and the same basic bet that a ternary 27B beats a bigger model you can't run locally. What changed between the two write-ups isn't the size claim — it's that this release comes with enough primary-source detail (a real per-benchmark table, a documented measurement protocol, an actual file manifest) to check the aggregate against, and the aggregate doesn't survive contact with its own footnotes intact. "98.2%" is a fair description of the reasoning core. It is not a fair description of what happens to a 5.9 GB model on a 500-instance SWE-bench run.