Bonsai 2 2B, 4B and 9B: where the bytes go when the model is small

A month after Bonsai 2 27B, PrismML has shipped the rest of the family: Ternary Bonsai 2 2B, 4B and 9B, built on Qwen3.5-2B, 4B and 9B, Apache 2.0, with headline sizes of 0.43 GB, 0.95 GB and 2.0 GB. The format is the 27B's: every weight matrix stored as {−1, 0, +1} times an FP16 scale per 128 weights, in a basis rotated by a fixed Walsh–Hadamard transform that the runtime has to undo on the activations. I covered how that works, and why a stock runtime produces garbage without it, in the 27B write-up; the gen-1 lineage is in Bonsai 27B, and the from-scratch version of the idea is in Ternary15M. This piece is about what is different when the model is small, and about four claims that went around on release day:

• PrismML: "2B: 0.43GB, 4B: 0.95GB, 9B: 2.0GB", and in the launch post "more than 9× smaller than their full-precision counterparts."
• Atomic Chat, reposting: "PrismML shrank Qwen3.5 7x so the 9B goes from 17.9 GB to just 2.42 GB and it keeps up or outperforms the original model on benchmarks."
• PrismML: the 9B "uses roughly 40% fewer tokens on standard instruction-following and everyday math tasks."
• PrismML: "On iPhone 18 Pro Max, Bonsai 2 9B reaches up to 34 toks/sec, while the 2B model reaches up to 120 toks/sec."

I pulled the file listing of every repo in the Bonsai 2 Small collection, parsed the GGUF header of all nine language-model packs (the 2B locally, the 4B and 9B through HTTP range requests, never downloading their weights), read the whitepaper and the docs, and built PrismML's llama.cpp fork to run the 2B and 4B. Labels below: measured is something I computed from a file or a run, reported is PrismML's or Atomic Chat's figure, reasoned is my arithmetic on the other two.

The sizes are exact, and they are decimal gigabytes

Start with the easy part. The three PTQ1_0 files on the Hub are 429,183,456, 949,549,920 and 1,997,486,976 bytes (measured, from the Hub tree API; the 2B and 4B files I downloaded match the repos' SHA256SUMS). In decimal gigabytes that is 0.429, 0.950 and 1.997: the headline 0.43 / 0.95 / 2.0 GB, rounded honestly. In GiB, the unit your OS and llama-bench print, the same files are 0.400, 0.884 and 1.860 GiB. llama-bench reports the 2B as 398.81 MiB because it counts only the tensor data, without the 11 MB header (measured).

What the number covers is stated plainly in the model card and the docs: the language model only. The vision tower ships as a separate mmproj file, 0.36 GB (Q8_0) for the 2B and 4B and 0.62 GB for the 9B (measured: 364,664,352, 366,894,624 and 624,229,664 bytes). It is loaded only when an image arrives. Text-only, the headline is the whole download; with images, the 2B is 0.79 GB and the 9B is 2.62 GB (reasoned).

What is in the file

The tensor tables are where the small models stop looking like the 27B. Here is the 2B, measured from its GGUF header, every tensor shape times the ggml type size (PTQ1_0 is 28 bytes per 128 weights, 1.75 bits; PQ2_0 34 bytes per 128, 2.125 bits; Q2_0 18 bytes per 64, 2.25 bits; F32 4 bytes). The slices sum to the exact file size for all nine packs.

Three things fall out of it.

The vocabulary is a quarter of the model. Qwen3.5 has a 248,320-token vocabulary at every size. At hidden size 2048 that is a 508.6M-parameter table, 27.0% of the 2B's language model; at 4B (hidden 2560, also tied) it is 15.1%; at 9B, which unties the embedding and the head, the two tables together are 22.7% (measured). In the 27B the same pair is under a tenth. PrismML ternarizes the embedding and the head too, so this does not cost bits per weight directly. It costs in the next two ways.

The fixed costs do not shrink. The header is 11.0 MB at every size, nearly all of it the tokenizer, and the F32 tensors (RMSNorm weights, ssm_alpha, ssm_beta, ssm_conv1d, ssm_a, ssm_dt, the Gated DeltaNet layers' state path that PrismML keeps out of ternary) are 6.9 MB at 2B. Together they are 4.2% of the 2B file and 2.0% of the 9B's (measured). That is why the effective rate is 1.82, 1.81 and 1.78 bits per weight for the 2B, 4B and 9B PTQ1_0 files, not the format's 1.75 (measured: file bits over the GGUF's own parameter count; the model cards say ~1.83 / ~1.81 / ~1.78, the 2B's from dividing by a rounded 1.88B).

The decode cost is not the file size. A token reads every matrix it multiplies by. When the embedding is tied to the LM head, as in the 2B and 4B, the whole table is read every token as the head. When it is untied, as in the 9B, the embedding is only gathered one row at a time, so a 9B decode step streams about 1.76 GB in PTQ1_0 and 2.14 GB in PQ2_0, not 2.0 and 2.42 (measured from the tensor tables). That matters for the phone numbers below.

The widget carries all nine packs. With the 9B's vision tower added, the ratio to BF16 drops from 8.97x to 6.83x (reasoned): headline and baseline are both language-model only.

Reconciling 2.0 GB, 2.42 GB, "7x" and "more than 9x"

Atomic Chat's numbers are not wrong. They are a different file. The 9B ships in three packings of the same ternary weights:

The model card's banner reads "~2.42 GB language model (PQ2_0), down from ~17.9 GB at FP16", and PQ2_0 is what the demo repo's setup.sh downloads by default, because it is cheaper to unpack. Atomic Chat quoted that banner: 17.9 / 2.42 is 7.4x (reasoned), so "7x" is a fair rounding of the bigger pack. PrismML's "2.0 GB" is the smaller pack. Same weights, two layouts.

The 17.9 GB baseline checks out: 8,953,803,264 language-model parameters in the GGUF times 2 bytes is 17.91 GB (measured), which is the whitepaper's figure. It is not the size of the Qwen3.5-9B repo, which is 19.31 GB because it carries the vision tower and more (measured, Hub API).

"More than 9x smaller", in the launch post, is the one number that does not hold. Against the whitepaper's own BF16 baselines (3.76, 8.41, 17.91 GB) the PTQ1_0 ratios are 8.77x, 8.86x and 8.97x (reasoned). The whitepaper itself says "8.8–9.0× smaller" in Table 1, and its cover says "~9×". Both of those are right. The launch post rounded up past them.

The benchmark table, and "keeps up or outperforms"

PrismML published one table, the same numbers in the launch post and the whitepaper (Table 6): 17 benchmarks at 2B, 18 at 4B (adding τ²-Bench), 19 at 9B (adding AIME26), each against its own Qwen3.5 BF16 baseline, run with EvalScope on vLLM, thinking on, temperature 1.0. All reported:

Source: whitepaper Table 6; the launch post's Figure I has the same numbers.

The averages give 96.7%, 94.6% and 95.1% retention (reasoned). Counting benchmark by benchmark, Bonsai is ahead of its baseline on 5 of 17 at 2B, 4 of 18 plus one tie at 4B, and 2 of 19 at 9B: HumanEval+ and IFEval (reasoned). At 9B six more are within two points (MuSR, MATH-500, GSM8K, IFBench, BFCL v3, MBPP+), and eleven are behind by more. The biggest gaps are the same three at every size: GPQA Diamond (−18.9, −14.2, −12.6 points), OmniDocBench (−16.1, −15.3, −14.9), and τ²-Bench at 4B and 9B (−19.8, −12.2). So "keeps up or outperforms the original model on benchmarks" is true of instruction following and coding, roughly true of math, and not true of knowledge, document parsing or multi-turn tool use. PrismML's own text is more careful than the repost: it calls the τ²-Bench drop "the largest single loss at 4B" and says OmniDocBench "does not close with a larger model."

Two things about the above-100% rows are worth a closer look.

The 2B baseline is unusually weak where Bonsai wins. Qwen3.5-2B scores 65.96 on GSM8K but 85.40 on the harder MATH-500, and 34.33 on IFBench. Bonsai 2B scores 80.74 and 48.00. A full-precision model doing worse on grade-school math than on competition math is the signature of a run that failed to finish, not of a model that cannot add: with thinking on, a small model that loops or runs out of output budget scores zero on that item. That is a reading, not a measurement (the whitepaper does not report truncation rates), but it fits PrismML's stated focus. The release post says the work went into "avoiding repetitive loops, following instructions … and controlling verbosity." Those are post-training targets. A ternary model that beats its own BF16 parent got something besides compression, and the comparison that would separate the two, the same post-training at full precision, is not published.

The aggregate has no single definition across sizes. The 2B average is over 17 benchmarks, the 9B's over 19, so the three retention numbers are not on one scale. The whitepaper says so itself: "averages compare within a size only."

The more useful table for choosing a model is the whitepaper's Table 8, which compares each ternary model against a smaller model that needs more memory. The 9B (2 GB) against Qwen3.5-4B in Q3_K_M (2.3 GB) leads on five of six category averages, 77.83 to 73.99, and on AIME26 82.50 to 54.17 (reported). The 4-bit 4B wins on vision. At equal memory, a bigger ternary model beats a smaller conventional quant, which is the 27B's result again at a tenth of the scale.

"Roughly 40% fewer tokens": not published anywhere I could find

The launch post and the thread say the 9B "uses roughly 40% fewer tokens on standard instruction-following and everyday math tasks." I searched the whitepaper, the three model cards, the docs pages for each model, and every Markdown file in the demo repo. None of them contains a token count, a token-length comparison, or the figure 40%. There is no named benchmark, no baseline (Qwen3.5-9B with thinking on? off?), and no budget. The whitepaper's own methodology runs both models with thinking on and "the same output budgets", and reports accuracy only.

The direction is plausible: the model cards warn that "even simple prompts can use 3,000+ thinking tokens", and verbosity was a training target. But "40% fewer" has nothing published behind it, and it is a claim about post-training, not about ternary weights.

The phone numbers come from a different phone

expand

The launch post's figure has three device groups. The RTX 4090 bars, 604 / 369 / 249 tok/s for 2B / 4B / 9B, are the whitepaper's Table 3 numbers rounded (603.6, 368.6, 248.6, PTQ1_0, tg128, batch 1). The other two groups are not in the whitepaper at all.

expand

The whitepaper measured an iPhone 17 Pro Max (llama.cpp Metal, PQ2_0, tg128, batch 1): 89.9, 39.8 and 22.3 tok/s for the 2B, 4B and 9B, and its cover leads with "90 tok/s". The launch post and the X thread instead quote an iPhone 18 Pro Max at 120, 51 and 34, and an "M5 Pro-Max" at 310 / 245 / 179, where the whitepaper's MacBook Pro M5 Pro tops out at 276.3 / 145.2 / 91.7 with MLX Swift. No document gives the iPhone 18 runtime, pack, context, prompt or whether a drafter was on. "Up to" in the post is doing work: the whitepaper's Table 4 shows the same 9B at 89 tok/s on the iPhone 17 with speculative decoding on copy-heavy agent prompts, and "plain" (no speedup) on everyday prompts.

You can still sanity-check the numbers, because batch-1 decode is a bandwidth problem. Each token streams the weights it multiplies once, so tok/s times bytes per token is the bandwidth the run actually achieved. For the whitepaper's iPhone 17 rows, with the per-token bytes measured above (PQ2_0), that is 45.5 GB/s at 2B, 45.2 at 4B and 47.6 at 9B (reasoned): flat across sizes, which is what a phone that is streaming weights as fast as it can looks like. For the iPhone 18 figures it is 60.8, 57.9 and 72.6 GB/s. The 9B's is a fifth higher than the 2B's on the same phone, which a plain bandwidth-bound decode cannot do; something else (a drafter, a different pack, a shorter context) is likely in the 34. That is an inference, and I could not confirm it.

The same arithmetic on the RTX 4090's PTQ1_0 rows gives 252 / 346 / 438 GB/s (reasoned), well under the card's ~1 TB/s: at these sizes a desktop GPU is limited by kernel overheads, not by memory, which is also why the whitepaper's speedups over BF16 are largest on the slowest devices (5.2x on the phone for the 2B, 1.5x on a B200).

Running it on a CPU

PrismML's demo repo pins prism-b10777-e3de715 of its fork; I built the head of the prism branch (5ae2e16, 15 October) with GGML_NATIVE=ON, on a shared 4-core Intel Xeon cloud VM with AVX-512, CPU only. Building llama-bench, llama-completion and llama-simple with -j2 took about 8 minutes.

git clone --depth 1 -b prism https://github.com/PrismML-Eng/llama.cpp && cd llama.cpp
cmake -B build -DGGML_NATIVE=ON -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
cmake --build build -j2 --target llama-bench llama-completion llama-simple
./build/bin/llama-bench -m Ternary-Bonsai-2-2B-PTQ1_0.gguf -t 2 -p 512 -n 128 -r 3

All measured, -t 2, every command under nice -n 19 on a machine five other jobs were sharing (load average 7-9 on 4 cores during the runs), so read the error bars:

None of this is comparable to PrismML's charts. The point is the ordering, which the source explains. In the fork's ggml-cpu/arch/x86/quants.c, PQ2_0 has an AVX2 dot product against Q8_K activations, PTQ1_0 has only an SSSE3 path (unpacking five trits from a byte is arithmetic, not a shift and a mask), and Q2_0 has no x86 kernel at all and falls back to the generic C loop. So on this CPU the 2-bit-slot pack is 3.4x faster at prompt processing than the denser one (measured), the opposite of the RTX 4090, where PTQ1_0 is the fastest pack at every size (reported). Decode here runs at under 1 GB/s of effective weight traffic (reasoned: 2.20 tok/s times 0.418 GB), so it is compute-bound, not bandwidth-bound; the widget's bandwidth arithmetic does not apply to it. PrismML targets Metal and CUDA; the CPU backend works, but it is not where the tok/s figures come from.

Generation swung by more than 2x between identical runs (2.23 and 5.70 tok/s for the same 64-token decode, measured): the machine, not the model. Whether CPU weight repacking helps decode was lost in that noise.

What loads in RAM

A 0.52 GB file took about 1.1 GB of resident memory to run (measured, peak RSS 1,086-1,161 MB across the PQ2_0 runs). The fork's verbose log accounts for it:

• 483 MiB of the file, memory-mapped.
• 476 MiB more: on x86 the CPU backend repacks 151 tensors into an interleaved pq2_0_4x8 layout for its GEMM, a second copy of nearly the whole model.
• 501 MiB of compute buffer at the default 512-token micro-batch, about the size of a logits tensor for 512 positions over 248,320 vocabulary entries in F32 (reasoned: 512 × 248,320 × 4 bytes is 485 MiB). The vocabulary shows up again.
• 43 MiB of context state: the KV cache for the six full-attention layers and 19.27 MiB of recurrent state for the 18 linear-attention layers.

Run with -nr (no repack) and -ub 64 -b 64 and the same model peaks at 633.4 MB (measured). On a phone the Metal path has no CPU repack, but the logits buffer is the same arithmetic, so the micro-batch is worth setting deliberately on a memory-tight device.

What it said

Four prompts to the 2B PQ2_0 pack, with the model card's sampling settings, one sample each. This is a smoke test, not an evaluation.

Instruction following, thinking off, "List exactly three uses for a paperclip. Use a numbered list, at most six words per item, and no other text":

1. Holds things together
2. Measures distance
3. Cuts metal

The format is exactly right: three items, numbered, nothing else, 16 tokens. The content is half right. A paperclip does not cut metal.

A tool call, thinking off, with one set_alarm(time, label) function in the system prompt in the chat template's own format, and "Wake me up at 6:45 tomorrow, call it gym":

<tool_call>
<function=set_alarm>
<parameter=time>
06:45
</parameter>
<parameter=label>
Gym
</parameter>
</function>
</tool_call>

Well-formed, correct arguments, zero-padded to the HH:MM the schema asked for, 40 tokens.

A word problem, "muffins at 3 dollars, coffee at 2, 4 muffins and some coffees for 20 dollars, how many coffees": correct (4) with thinking off in 193 tokens, and correct with thinking on in 178, because with thinking on the model closed its <think> block immediately and wrote no reasoning at all. One prompt proves nothing about "40% fewer tokens", but it is the behaviour that claim describes.

What I could not check

• "Roughly 40% fewer tokens." No published measurement, benchmark or baseline.
• The iPhone 18 Pro Max and "M5 Pro-Max" figures. No runtime, pack, prompt or drafter setting is published for either; the whitepaper's measured phone is an iPhone 17 Pro Max.
• Whether the above-baseline 2B scores come from ternary training or from post-training, and whether the Qwen3.5-2B baseline runs were truncated. The whitepaper reports neither.
• The 9B on a CPU. I parsed its header but did not download its weights.
• The 2B MLX pack's size: 1.34 GB in the whitepaper and docs, 1,200,284,784 bytes on the Hub (measured).

The takeaway

The 27B made the case that a ternary format with a rotation and a purpose-built kernel keeps most of a model's ability at a ninth of the bytes. The small models make a different case: at 2B, the format is no longer the whole story. A 248,320-token vocabulary is a quarter of the weights and, on a CPU, most of the compute buffer; an 11 MB tokenizer header is 2.6% of the file; and the choice between PTQ1_0 and PQ2_0 moves the size by 20% and the speed by a factor that depends on which kernels your hardware has. The headline sizes are honest. The best number in the release is the one in the whitepaper's Table 8: a 2 GB ternary 9B beats a 2.3 GB conventional 4B on five of six categories. If memory is the constraint, a bigger ternary model is the better buy, and the 27B is the same argument at laptop scale.