# Bonsai 2 27B: PrismML's sequel gets to 98.2% — on average

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/bonsai-2-27b
> date: 2026-09-18
> tags: quantization, inference-optimization, on-device, multimodal, benchmarks, explainer
Two months after [Bonsai 27B](/articles/bonsai-27b), PrismML has shipped **Bonsai 2 27B** —
same idea, same rough footprint, a stronger base model, and a headline retention number that
climbed from "~95%" to "98.2%." The size story barely moved: still a ternary `{−1, 0, +1}`
weight format with an FP16 group scale, still around 5.9-6 GB depending on which file you
count, still Apache 2.0. What changed is the base (Qwen3.6-27B → **Qwen3.8-27B**), the
per-tensor precision map (very slightly, and in the opposite direction from what you'd
expect), and — for the first time — real numbers on the long-horizon agentic benchmarks
the first release's roadmap promised to chase. I pulled the safetensors/GGUF metadata for
every repo in the [Bonsai-2 collection](https://huggingface.co/collections/prism-ml/bonsai-2),
read the [whitepaper](https://github.com/PrismML-Eng/Bonsai-demo/blob/main/bonsai-2-27b-whitepaper.pdf)
end to end, and diffed its numbers against the model card's. They don't always agree with
each other, which turns out to be the most useful fact in this release.

<ModelCard
  repo="prism-ml/Ternary-Bonsai-2-27B-gguf"
  claimed="27B, ~5.9 GB language weights"
  note="The 76.1 GB 'repo size' above is the whole repository, including the 53.81 GB F16 reference build used to produce the quantized packs -- nobody downloads that to run the model. The two deployable files are Ternary-Bonsai-2-27B-PTQ1_0.gguf (5.95 GB) and -PQ2_0.gguf (7.21 GB); the vision tower ships separately as a 0.63-0.93 GB mmproj file."
/>

## What actually changed from the first release

The first Bonsai 27B pushed its ternary and 1-bit encodings through *every* block —
embeddings, attention, MLPs, the LM head — and made a selling point of having "no high-precision
escape hatches." Bonsai 2 27B does something slightly different: it pulls back, by a fraction of
a percent, and keeps a small set of tensors — the recurrent state path of the linear-attention
layers (`in_proj_a`, `in_proj_b`, `conv1d`) plus the RMSNorm and Q/K-norm weights — in full
precision. It's 26,238,464 parameters, 0.0976% of the language model, 52 MB at bf16. Toggle
between parameter share and byte share to see why that tiny exception (and the separate vision
tower) matter more than their parameter count suggests:

<BitBudget />

The honest read: Bonsai 2 27B is still about as end-to-end as a ternary model gets — 98%+ of
parameters by count — but "end to end, no exceptions" was gen 1's pitch, not gen 2's. PrismML
spent 0.01 bits/weight to buy back whatever quality that recurrent state path was costing at
ternary. It's a small, specific concession, and the whitepaper is upfront that it's new.

## The 5.9 GB number, audited

The brief version of the claim is "5.9 GB, down from 54 GB, roughly 9x." Here's what that
collapses when you look at the actual bytes instead of the rounded headline. A 27B model at
FP16 really is 53.81 GB — that's the exact size of `Ternary-Bonsai-2-27B-F16.gguf`, sitting in
the same repo as a reference build. A ternary weight carries `log2(3) ≈ 1.585` bits of pure
information; on ~26.86B ternarizable parameters that's about 5.32 GB with no overhead at all.
Add back the FP16 group scale (one per 128 weights) and PrismML's own "true ternary" figure is
1.72 bits/weight, 5.80 GB. Then add the packing overhead that an actual, loadable file needs —
PrismML ships two: `PTQ1_0` (dense trits, **5.95 GB** on disk) and `PQ2_0` (each trit in its own
2-bit slot, 7.21 GB). **5.95 GB is not a rounding error — it's the real file**, three layers of
overhead past the theoretical floor: group scales, then the 0.0976% full-precision exceptions,
then the packing format's own cost.

You can see where that last layer goes without downloading a single weight. The MLX pack ships its
own loader, and `runtime/codec.py` is the part that reads a packed GGUF block:

```python
sizes = {"PQ2_0": 34, "PTQ1_0": 28}
if source not in sizes:
    raise ValueError(f"Unsupported format: {source}")
blocks = rows * width // 128
if len(raw) != blocks * sizes[source]:
    raise ValueError("Raw byte length does not match shape and format")
data = np.frombuffer(raw, dtype=np.uint8).reshape(blocks, sizes[source])
scale_bytes = data[:, :2] if source == "PQ2_0" else data[:, 26:28]
scales = scale_bytes.copy().view("<f2").reshape(rows, width // 128)
# ...
return words, np.ascontiguousarray(scales), np.ascontiguousarray(-scales)
```

That dictionary is the whole size story in two integers. A `PTQ1_0` block is **28 bytes per 128
weights** — 1.75 bits/weight, exactly. A `PQ2_0` block is **34 bytes**: 32 bytes of 2-bit slots plus
the FP16 group scale, 2.125 bits/weight. Those are the model card's numbers (1.75 and 2.13). The
whitepaper says 1.76 and 2.16 instead, because it is rating the shipped file rather than the format.
The files settle it: 5,946,648,928 and 7,206,168,928 bytes on the Hub, which is **1.77 and 2.14
bits/weight** over the 26.89B language-model parameters. The gap is small and it cuts both ways —
the whitepaper's own storage table lists 5.93 GB and 7.25 GB, and neither matches what it uploaded,
while the model card's 5.95 and 7.21 do.

The last line of that excerpt is also the neatest trick in the format. The bias handed to the kernel
for every group is `-scales`: trits are stored as unsigned 2-bit codes `0, 1, 2` and reconstructed as
`(code − 1) × scale`, which is how `{−1, 0, +1}` rides a stock *affine* 2-bit matmul with no
ternary-specific dequantization path at all. The exotic part of this model is not the arithmetic.

None of those numbers include the vision tower, and "language weights" is the load-bearing
phrase in every one of PrismML's own size claims. The tower ships as its own file — a Q8_0
container holding 4-bit HQQ weights, 0.63 GB, or a BF16 reference at 0.93 GB — over 460M
parameters that were never ternarized and were never going to be. Add the smaller vision pack to
the smaller language pack and the real download for a multimodal deployment is **~6.58 GB**, not
5.9. That's still a large win over 54 GB. It just isn't the number in the highlights bullet.

## 98.2% of what, exactly

This is the number doing the most work in the launch post, and it's worth being precise about
what it is an average *of*. PrismML publishes three different versions of "98.2%," and they
disagree with each other on the absolute score:

- **Launch post / whitepaper**: 83.9 vs. 85.4, on a **20-benchmark** suite (τ²-Bench, BFCL v3,
  HumanEval+, LiveCodeBench v6, MBPP+, BigCodeBench, IFEval, IFBench, MMLU-Redux, GPQA Diamond,
  AA-LCR, AIME25, AIME26, GSM8K, MATH-500, CharXiv, A-OKVQA, OmniDocBench v1.6, RealWorldQA,
  OCRBench v2), thinking mode, `xhigh` reasoning effort, against Qwen3.8-27B FP16.
- **Model card**: 84.78 vs. 86.32, on a **different 14-benchmark** suite that drops τ²-Bench,
  AA-LCR, BigCodeBench, and the four vision benchmarks above in favor of **MuSR** and
  **MMMU-Pro**, which don't appear in the 20-benchmark suite at all.

Both ratios round to 98.2%. Neither the suite nor the FP16 baseline score is the same number —
85.4 in one document, 86.32 in the other, for the identical Qwen3.8-27B FP16 model. That's not an
error so much as a demonstration of the thing an aggregate is supposed to hide: pick a slightly
different 15-20-benchmark bag and the same underlying model produces a different "%
retained," and both bags happen to land on the same marketing number.

The full 20-benchmark table (thinking mode, `xhigh`) is the more complete of the two, and it's
where the real spread lives:

| Category | Benchmark | Qwen3.6-27B FP16 | Qwen3.8-27B FP16 | Qwen3.8-27B IQ2_XXS | Bonsai 2 27B (ternary) |
| :--- | :--- | ---: | ---: | ---: | ---: |
| Knowledge & reasoning | MMLU-Redux | 93.26 | 91.46 | 85.79 | 89.09 |
| Knowledge & reasoning | GPQA Diamond | 86.87 | 90.51 | 65.45 | 85.76 |
| Knowledge & reasoning | AA-LCR | 74.00 | 78.00 | 64.00 | 77.00 |
| Math | GSM8K | 95.60 | 97.19 | 95.38 | 96.66 |
| Math | MATH-500 | 99.20 | 99.80 | 94.80 | 98.80 |
| Math | AIME25 | 90.42 | 96.67 | 82.50 | 95.00 |
| Math | AIME26 | 93.33 | 94.58 | 78.60 | 95.83 |
| Coding | HumanEval+ | 95.73 | 93.29 | 87.95 | 95.12 |
| Coding | MBPP+ | 83.07 | 83.86 | 78.31 | 83.07 |
| Coding | LiveCodeBench v6 | 89.10 | 90.05 | 70.05 | 90.07 |
| Coding | BigCodeBench | 62.37 | 61.49 | 49.81 | 58.07 |
| Instruction following | IFEval | 88.72 | 91.50 | 81.52 | 91.31 |
| Instruction following | IFBench | 60.33 | 71.00 | 52.33 | 74.00 |
| Agentic / tool calling | τ²-Bench | 82.90 | 82.73 | 69.43 | 80.22 |
| Agentic / tool calling | BFCL v3 | 77.19 | 76.74 | 66.10 | 74.92 |
| Vision | CharXiv | 81.60 | 82.02 | 78.76 | 80.00 |
| Vision | A-OKVQA | 89.87 | 89.34 | 86.20 | 86.81 |
| Vision | OmniDocBench v1.6 | 84.17 | 92.46 | 82.87 | 89.13 |
| Vision | RealWorldQA | 81.83 | 83.40 | 78.56 | 80.13 |
| Vision | OCR Bench v2 | 61.64 | 60.99 | 55.37 | 56.88 |
| **Overall (20)** | | **83.6** | **85.4** | **75.2** | **83.9** |

*Source: whitepaper Table 10, thinking mode, `xhigh` reasoning effort.*

Math and instruction following barely move — AIME26 and IFBench actually score *higher* than
Qwen3.8-27B FP16, and LiveCodeBench is a rounding error apart. The real losses cluster in exactly
the categories a coarse "reasoning holds up" story tends to skip: **OCR Bench v2** loses 6.7% of
its FP16 score, **BigCodeBench** loses 5.6%, **GPQA Diamond** loses 5.2%. None of those individual
drops is dramatic, but "98.2% aggregate" is quietly compatible with a 6.7% loss on any one
benchmark that happens to get diluted by nineteen others that didn't move.

The bigger story isn't in this table at all. It's what the whitepaper reports in prose, a page
away from the aggregate, and doesn't fold into either suite:

| Benchmark | Qwen3.8-27B FP16 | Bonsai 2 27B (ternary) | Retention |
| :--- | ---: | ---: | ---: |
| Terminal-Bench 2.1 | 69.7 | 52.8 | 75.8% |
| SWE-bench Verified | 80.6 | 60.8 | 75.4% |

These are the two benchmarks PrismML says it evaluated "for the first time," specifically because
the first Bonsai 27B's roadmap called out "long-horizon, tool-driven software engineering" as the
next thing to fix. They're real evaluations, run through the Harbor framework with the
Terminus-2 agent (89 tasks, single attempt) and the mini-swe-agent scaffold (500 instances) — not
demo footage. And on the workload the release is most explicitly marketed at — "agentic coding,"
"long-horizon tool use" — the ternary model keeps roughly **three-quarters** of full precision,
not 98.2%. Both numbers are true. They are not the same claim.

## Memory vs. quality, against everything else at this size

Put Bonsai 2 27B next to the other ways to shrink a 27B model — full precision, a well-regarded
conventional 4-bit GGUF, a conventional 2-bit GGUF, and the previous Bonsai release, all scored on
PrismML's own 14-benchmark comparison table so the numbers are at least internally consistent:

<Frontier />

Qwen3.8-27B IQ2_XXS is the useful control here: at 9.4 GB it is both **bigger and worse** than
Bonsai 2 27B at 5.9 GB — a conventional low-bit build strictly dominated by a purpose-built one.
(PrismML sizes that same third-party build two ways, too: 9.4 GB at 2.8 bpw on the model card,
7.3 GB at 2.2 bpw in the whitepaper. Bigger and worse either way, by less at 7.3.)
UD-Q4_K_XL is the harder comparison and the one the highlights bullets skip: it beats Bonsai 2 27B
by 0.4 points at three times the size, which is a real trade-off, not a rout — reach for it if
that last half-point matters more to you than the footprint.

## The intelligence-density chart, from the source

PrismML's own launch post makes the same point with a coined metric, "intelligence density"
(`-log2(1 - score/100) / size_GB`), plotted against nine other models:

<Figure
  src="/articles/bonsai-2-27b/fig1-intelligence-density.png"
  alt="Bar chart of intelligence density, in score per gigabyte, for eleven models. 1-bit Bonsai 27B leads at 0.530, ahead of Ternary Bonsai 2 27B at 0.444 and Ternary Bonsai 27B at 0.400; several conventional quantized and full-precision baselines trail far behind, from Qwen3.8-27B IQ2_XXS at 0.275 down to Gemma-4-31B FP16 at 0.044."
  caption="Intelligence density (score per GB) across model families, as published by PrismML (launch post, Figure II)."
/>

Read the bars in order and the chart quietly undercuts its own headline: the tallest bar isn't
today's release. It's **1-bit Bonsai 27B** — the older, smaller, *less capable* model from the
first launch — at 0.530/GB, ahead of the new "most capable model yet" at 0.444/GB. The formula
divides by size before it multiplies by anything else, so a smaller, lower-quality model can win
on "density" over a bigger, better one; PrismML's own chart is the proof. It's not a
contradiction exactly — 1-bit Bonsai 27B really is smaller and the metric really does reward that
— but it means "intelligence density" and "our most capable model" are two different superlatives,
and the launch post's prose implies they're the same one.

## The tok/s numbers, and the one I couldn't find

The launch post's headline throughput claims are **143 tokens/second on an RTX 5090** and **46.8
tokens/second on an Apple M5 Max**, with no stated batch size or context length. The whitepaper's
Appendix D fills that in: both are `tg128` — token-generation throughput over 128 generated
tokens, the memory-bandwidth-bound decode phase — at **batch size 1, depth 0, with the vision
tower excluded**, on the `PQ2_0` pack, measured on 2026-09-16. Prompt processing (`pp512`, the
compute-bound phase) is a different and much larger number on the same hardware — 4121 tok/s on
the RTX 5090 — which is the standard shape for batch-1 LLM inference and not specific to ternary
weights, but it's worth knowing which phase "143 tok/s" describes before you compare it to
anything.

It's also reproducible, which is the part I like about this release. The measurement is plain
`llama-bench`, and the demo repo's own benchmark template gives the shape of the run — all layers
offloaded, flash attention on — with Appendix D supplying the rest of the protocol (pp512/tg128,
three runs after a warm-up pass):

```bash
BENCH=bin/cuda/llama-bench
$BENCH -m models/bonsai2-gguf/27B/Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -p 512 -n 128 -r 3
```

Run it and you inherit PrismML's own problem, which is that its two documents disagree about what
the answer should be. Whitepaper Table 5: **142.5** tok/s `tg128` and **4121** tok/s `pp512` for
`PQ2_0` on an RTX 5090. The model card's cross-platform table, same pack and same card:
**129.9** and **3893** — about 9% and 6% lower — with energy quoted in J/token where
the whitepaper quotes mWh/token. The `PTQ1_0` row moves the same way (134.4 vs 120.5), and the M5
Pro row differs too (27.7 vs 28.1). These are clearly two different measurement runs — the hardware
lists don't even match — but neither document dates the other or says which supersedes it, and the
launch post promotes the higher pair. Treat "143 tok/s" as the top of a ~130-143 range on that card,
not a figure to size a deployment against.

The brief for this piece asked about a third figure — 55 tok/s, WebGPU, M5 Max — and I could not
find it in the launch post, the whitepaper, the model card, or the README of the
[WebGPU kernels demo space](https://huggingface.co/spaces/webml-community/ternary-bonsai-2-webgpu-kernels)
that PrismML links from the launch post's resources sidebar. That demo is real — it runs the
model's ternary kernels in-browser over WebGPU — but its own README ships the Hub's generic
template with no throughput number in it at all. The only *published* M5 Max figure I can verify
is 46.8-47.0 tok/s, and it's measured through `llama-bench` on the native Metal backend, not a
browser. If 55 tok/s exists somewhere, it isn't in any of PrismML's own primary sources for this
release.

## Ternary weights still need a runtime built for them

The memory win is real and it is not automatically a compute win. Two things make that
concrete here. First, the weights aren't stored densely for cheap FP16 dequantization at
inference time — they're stored in a **rotated basis**: each 1024-weight block is transformed by
a fixed Walsh–Hadamard rotation before ternarization, and the runtime has to apply the matching
transform to *activations* on every forward pass. At batch size 1 this rotation "lies on the
critical path of every projection" (the whitepaper's words), and the fork's Metal and CUDA kernels
spend real engineering fusing it into the load path specifically to keep it off that path.

That claim is checkable, because the MLX pack ships the loader that has to honor it. `runtime/runtime.py`
puts the transform exactly where the whitepaper says it lives — in front of the matmul, every time:

```python
def fwht(x, block, signs, inverse=False):
    shape, dtype = x.shape, x.dtype
    x = x.astype(mx.float32)
    if not inverse:
        x = x * signs
    x = mx.hadamard_transform(
        x.reshape(-1, block), scale=1 / math.sqrt(block)
    ).reshape(shape)
    if inverse:
        x = x * signs
    return x.astype(dtype)


class Packed(nn.Module):
    def __call__(self, x):
        # ... embedding path elided
        if self.block:
            x = fwht(x, self.block, self.signs)
        return mx.quantized_matmul(
            x, self.weight, self.scales, self.biases,
            transpose=True, group_size=128, bits=2,
        )
```

Sign flip, then a 1024-wide fast Walsh–Hadamard in float32, then the 2-bit matmul — per projection,
per token. The embedding is the mirror image: it dequantizes the row first and applies the *inverse*
transform to the result, which is why the GGUF ships a separate `inverse_weight_names` manifest
containing exactly one tensor, `token_embd.weight`.

Second, the packed formats aren't something a stock runtime quietly mishandles — they're something
it doesn't recognize at all. `PQ2_0` and `PTQ1_0` are PrismML's own ggml types, and mainline
llama.cpp **rejects both as unknown**. The silent-corruption failure the model card warns about is
narrower and more interesting: gen 1's ternary type, `Q2_0`, *has* landed in mainline, so a mainline
build will cheerfully load a `Q2_0` file, skip a Hadamard transform it knows nothing about, and
produce garbage with no warning. MLX fails the same way from the other direction — the demo repo's
own MLX script says stock MLX loaders "skip it and return wrong output rather than an error," which
is why the model repo ships executable Python next to the weights. Bonsai 2's own GGUF at least
defends itself: the rotation travels as metadata (`prism.hadamard.block_size`, `sign_mode`,
`weight_names`), and the loader raises rather than guessing if any of it is missing. So: use the
fork.

```bash
# prebuilt, pick the archive for your platform
# https://github.com/PrismML-Eng/llama.cpp/releases/latest
tar -xzf llama-<tag>-bin-<platform>.tar.gz -C bin --strip-components=1

# or build it
git clone https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j    # drop -DGGML_CUDA=ON on macOS, Metal is default

./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -c 32768 \
    --temp 1.0 --top-p 0.95 --top-k 20 \
    -p "Explain quantum computing in simple terms." -n 256
```

Those are the model card's own quickstart commands — it prints `./bin/llama-cli` for the prebuilt
archive and `./build/bin/llama-cli` if you built the fork yourself. That fork, the MLX fork, and
the mlx-swift fork are all built around custom fused low-bit GEMM kernels that unpack trits and
apply the group scale inside the matmul, with activations kept in higher precision throughout.

That both formats exist at all is the clearest evidence there's no free lunch here. `PTQ1_0`
packs trits densely and gets closer to the 1.72-bit floor, but unpacking costs arithmetic;
`PQ2_0` wastes bits on a 2-bit-per-trit slot but decodes cheaper. Which one wins is a hardware
question, not a format question: `PTQ1_0` is faster on Ada-class cards and the L4, `PQ2_0` wins on
Hopper, Ampere, Blackwell, and every Apple Silicon chip measured. A ternary weight format that
needed no purpose-built kernel, and where one packing was uniformly faster than the other, would
be the more surprising result.

## Agentic claims: the demos and the evidence are two different things

The launch post's two video demos — a Cline coding-agent session and a "computer use" session,
both on an RTX 5090 — are the most visible agentic evidence in the release, and they're exactly
what they look like: unscored recordings of the model doing something, not a controlled
comparison against anything. The real evidence for the agentic claims is the Terminal-Bench 2.1
and SWE-bench Verified numbers from two sections up — 75.8% and 75.4% retention, a genuine
capability that a 5.9 GB model plausibly couldn't manage at all under naive quantization, but well
short of "98.2% of full precision" for the workload the release is named after. Both things are
true at once: this is a real improvement on the roadmap gen 1 set out, and the improvement is
smaller than the release's own headline number for anything that resembles a coding agent running
for more than a few turns.

## License and the "9x" claim

Apache 2.0, unchanged from gen 1 — confirmed on every repo in the collection, and the actual
`LICENSE` file matches. "9x smaller" checks out as a *range* rather than a single number:
9.0x for the `PTQ1_0` file you'd actually deploy, 9.3x for PrismML's idealized "true ternary" rate
that no packed file quite reaches, both against the real 53.81 GB FP16 reference sitting in the
same repo. "More than 9x" is a fair thing to say about this model. It just isn't one number.

<Callout type="warn">
The size story holds up well: **5.95 GB really is 5.95 GB**, measured off the actual GGUF file,
and Bonsai 2 27B really is close to end-to-end ternary (98%+ of parameters). "**98.2% of full
precision**" is real but is an average over a benchmark suite that PrismML itself defines two
different ways across its own documents (20 vs. 14 benchmarks, different baselines, same rounded
percentage) — and it excludes the two agentic, long-horizon benchmarks evaluated for the first
time in this release, where retention drops to **~75%**. "**9x smaller**" is accurate as a range
(9.0-9.3x), not a single figure. The **55 tok/s WebGPU / M5 Max** claim does not appear in any
PrismML primary source I could find; the verifiable M5 Max number is **46.8-47.0 tok/s** on native
Metal, at batch size 1 over 128 tokens, without the vision tower. The Cline and computer-use RTX
5090 demos are unscored recordings, not benchmarks. The **RTX 5090 throughput** figure is itself two
numbers: 142.5 tok/s in the whitepaper, 129.9 tok/s in the model card, same pack and same card.
Ternary weights need a purpose-built runtime — mainline llama.cpp **rejects** `PQ2_0` and `PTQ1_0`
as unknown types; the file that loads silently and produces garbage is gen 1's now-upstreamed
`Q2_0`, because mainline has no Hadamard activation path.
</Callout>

## The takeaway

Bonsai 2 27B is a real, incremental win over [the first release](/articles/bonsai-27b): a
stronger base model, a genuinely improved (if still partial) answer to gen 1's agentic-coding
roadmap item, and the same basic bet that a ternary 27B beats a bigger model you can't run
locally. What changed between the two write-ups isn't the size claim — it's that this release
comes with enough primary-source detail (a real per-benchmark table, a documented measurement
protocol, an actual file manifest) to check the aggregate against, and the aggregate doesn't
survive contact with its own footnotes intact. "98.2%" is a fair description of the reasoning
core. It is not a fair description of what happens to a 5.9 GB model on a 500-instance SWE-bench
run.