Ternary Bonsai 2 27B Uncensored: the numbers behind a runtime abliteration

Continuum-AI-Corp/OrcaBonsai-27B-Uncensored — published as "OrcaRouter Ternary Bonsai 2 27B Uncensored" — is a refusal-ablation release with no modified checkpoint. It applies the same technique this site checked in OrcaRouter's GLM-5.3-Flash-Uncensored-FP8 to PrismML's Bonsai family, but on a base model that makes the usual approach expensive in a specific, checkable way: prism-ml/Ternary-Bonsai-2-27B-mlx-2bit is quantized to roughly 1.72 bits per weight by quantization-aware training (QAT), and editing weights on a QAT model means dequantizing, editing, and re-quantizing — a second lossy step stacked on top of the first. OrcaRouter's fix: don't touch the weights. Hook the residual stream instead, at inference, on every module that writes into it.

What abliteration is, briefly

Arditi et al., 2024 ("Refusal in Language Models Is Mediated by a Single Direction") found that across a range of open chat models, refusal concentrates along one direction in the residual-stream activation space, and that projecting that direction out of every module that writes the residual stream is usually enough to make a model stop refusing almost entirely. This site went through that mechanism and its limits in more detail when it checked OrcaRouter's abliteration of GLM-5.3-Flash; the short version repeats here only as far as it's needed to make sense of what this repo does differently.

"Almost entirely" is worth pinning to the paper's own numbers rather than to the phrase. Across 13 open chat models up to 72B, over 100 harmful instructions from JailbreakBench, ablation drops the refusal score — a substring match against a published list of refusal openers — to between 0.00 (Qwen 14B Chat) and 0.27 (Llama-2 70B Chat): near-total on most models, total on none. The safety score, which is Llama Guard 2's judgment that a completion is harmless, falls a good deal less far, to between 0.13 and 0.42. That gap is the paper saying plainly that removing the refusal phrasing is not the same thing as producing a compliant answer, and its Appendix D spends three worked examples on the cases where the two metrics disagree. OrcaRouter's own JailbreakBench figure for this pack — 4% residual refusal on the same 100-instruction set — lands inside that band, though with no harness published anywhere in the repo there's no way to tell whether it was scored the same way.

expand

The "differently" is the whole point of this release. Every abliteration this site has looked at so far — GLM-5.3-Flash-Uncensored-FP8, Qwen3.8-Flash-Next-Uncensored — ships as a new set of weights: a new Hugging Face repo, a new checkpoint, published alongside or instead of the base model. OrcaBonsai ships none of that. There is no orcarouter/*Bonsai* repo on the Hub — a quick check of OrcaRouter's own Hugging Face org turns up 28 public repos — 25 "Uncensored" releases across Qwen, GLM, DeepSeek, Nex, and Gemma, plus three plain MLX re-quantizations that carry no ablation at all — and not one of them is Bonsai. The GitHub repo is the whole release.

Why the usual approach is a bad fit here

Conventional directional ablation is a weight edit. For every matrix W that writes the residual stream, you orthogonalize it against a unit refusal direction r:

W ← W − r(rᵀW)

and save the result as a new checkpoint. That's exactly what happened, at native FP8, when this site checked GLM-5.3-Flash-Uncensored — the edited tensors kept the base model's own dtype and shape, byte for byte, everywhere except the values themselves.

Bonsai 2's weights don't have that option. Per bonsai_abliterate/ablation.py:

"""Apply a refusal direction to a Prism Hadamard ternary pack at run time.
 
Ordinary abliteration is a permanent weight edit: every matrix that writes the residual
stream is orthogonalised against the refusal direction, ``W <- W - r (r^T W)``. That is
not possible on Bonsai 2 27B. Its weights are ternary -- the affine container stores
``scale = s`` and ``bias = -s``, so the 2-bit codes ``{0,1,2}`` decode to exactly
``{-s, 0, +s}`` -- while the orthogonalised matrix is dense and full precision. Storing
it back would mean re-quantizing to ternary, and the model's quality at 1.72 bits/weight
comes from quantization-aware training, not from the format: re-quantizing without that
training is what destroys it.
"""

That's checkable independent of the repo's own framing. The affine container's bias == -scale identity — which is what makes {0,1,2} decode to exactly {-s, 0, +s} rather than three arbitrary levels — is something this site verified for the whole Bonsai family already, and scripts/prepare_ios_pack.py re-verifies it independently before it will drop a bias tensor, checking every one of the pack's 402 packed modules and refusing to proceed if max |bias + scale| isn't exactly zero. Orthogonalizing W mixes those three discrete levels into a dense matrix of arbitrary floats. Getting back to ternary from there is a second quantization pass, stacked on top of the QAT that produced the original weights — and it wasn't trained with that second pass in the loop.

Same operator, no weights touched

The runtime's fix follows directly from where the intervention actually needs to sit. Directional ablation doesn't require editing W — it requires removing a direction from W's output. For any linear map y = Wx, projecting the output and projecting the weight matrix first are the same operation, by nothing more than associativity of matrix multiplication:

(I − r·rᵀ)(Wx)  =  ((I − r·rᵀ)·W) x

Project y after the matmul, or edit W before it — the result is identical, because a projection matrix commutes trivially through a preceding linear map. bonsai_abliterate/ablation.py computes exactly the left-hand side, per module, per token:

class Ablated(nn.Module):
    def __call__(self, *args, **kwargs):
        y = self.inner(*args, **kwargs)
        yf = y.astype(mx.float32)
        component = mx.sum(yf * self._direction, axis=-1, keepdims=True)
        return (yf - self._alpha * component * self._direction).astype(y.dtype)

self.inner is the original, untouched down_proj / o_proj / out_proj module — packed ternary weights and all. Ablated never reads or writes a weight tensor; it wraps the module's output. Because the mathematics doesn't care which side of the matmul the projection happens on, the engineering gets to pick the side that's cheap: the activation side never forces a re-quantization, because there's no quantized thing on that side to re-quantize.

One detail the repo calls out that's easy to get wrong: Bonsai's projections run in a Hadamard-rotated basis on their input dimension only, and the runtime's own Packed loader un-rotates activations back to the plain hidden basis before they leave each module. Since the refusal direction is projected against outputs — which are already in the plain basis — no extra rotation handling belongs in ablation.py, and the repo is explicit that adding one "would project against the wrong basis." The direction itself is stored as nothing more exotic than 5,120 little-endian float32 values, matching hidden_size exactly.

129 sites, mapped

The repo's other easy mistake to make is wrapping too little. self_attn.o_proj is the obvious residual writer to reach for, but Bonsai 2 is a hybrid-attention model — most of its layers don't have an o_proj at all. Pulled straight from the pack's own config.json (fetched from the Hub, not guessed): 64 layers, hidden_size 5120, layer_types alternating three linear-attention layers to one full-attention layer — 48 linear-attention, 16 full-attention, exactly matching the repo's claim. Every layer writes the residual stream twice, once from its attention block's out-projection (linear_attn.out_proj on 48 layers, self_attn.o_proj on the other 16) and once from its MLP's down_proj — plus one more write from model.embed_tokens. q_proj/k_proj/v_proj and the MLP's gate_proj/up_proj don't get wrapped, because their outputs never touch the residual stream directly; neither does lm_head, which consumes the final residual state rather than writing to it.

64×2 + 1 = 129. That's not a number this article had to trust — it falls out of counting config.json's own 402-entry modules list by path suffix, which is the same count selfcheck.py expects and warns about if it doesn't see.

The strength knob, exactly

alpha isn't clamped anywhere in run.py's argument parser — it's a plain float, so the README's table (0, 0.5, 0.7, 0.9, 1.0, "over-projection" above 1.0) is a usage guide, not an enforced range. The formula makes the boundary behavior exact rather than approximate: decompose a write y into a component along r and a component perpendicular to it, and Ablated leaves the perpendicular part untouched while rescaling the parallel part by (1 − α). At α=1 the parallel part is zero — a full projection, matching a weight edit exactly. Past α=1, (1 − α) goes negative: the surviving component doesn't just keep shrinking, it flips to point against r. Nothing in the repo states that directly; it falls straight out of the formula the repo does state.

α=0 gets special treatment worth noting precisely: run.py checks if args.alpha != 0 before ever calling install(). At α=0, the wrapping never happens — not "the projection subtracts zero," but "the 129 Ablated modules never get constructed," so the forward pass at α=0 runs the exact same code path as the unmodified pack, not merely a mathematically-equivalent one.

Bit-identical, checked against the Hub

"Bit-identical" is a claim about a file this article can actually go check. Hugging Face's blob listing for the pack the runtime requires puts model.safetensors at 8,595,477,990 bytes — 8.005 GiB, which lines up exactly with the README's own "iPhone memory budget" breakdown (ternary codes, redundant biases, scales, norms, and a 0.858 GiB vision tower, summed to the same 8.005 GiB). That's the file bonsai_abliterate/pack.py loads via vision_artifact.load_vl_model, which in turn calls mx.load(directory / "model.safetensors") — a read. Every write call anywhere in bonsai_abliterate/, run.py, and scripts/selfcheck.py writes somewhere other than the --pack directory: nowhere in this repo's code does anything call mx.save_safetensors or open(..., "wb") on the path the user points --pack at. "Bit-identical" holds here because the code has no path that would make it otherwise — not because anything hashes the pack on load and confirms it.

That distinction matters for a second reason: this repo's own tooling never checks a hash. The pack itself ships files.json, a SHA-256 manifest for every file including model.safetensors, and reload-validation.json (reload_logits_exact: true, checked against 248,320 logits) — but both belong to prism-ml's pack, verifying prism-ml's own MLX serialization round-trip, and neither is ever read by bonsai_abliterate/ or run.py. If you point --pack at a directory that isn't actually the genuine pack, nothing in this repo will tell you.

The size check also catches something this article's brief assumed and had to unlearn: Ternary-Bonsai-27B covered on this site earlier is a 27B ternary model shipped in more than one container, and it's tempting to think "the original Bonsai pack" means whichever one you happen to have. It doesn't. prism-ml/Ternary-Bonsai-2-27B-gguf's own blob listing puts Ternary-Bonsai-2-27B-PTQ1_0.gguf at 5,946,648,928 bytes — a real file, a real 5.95 GB ternary pack, and the wrong one for this repo. run.py imports mlx.core, and the pack's quantized_matmul kernel exists only for Metal and CPU — there is no CUDA path, and there is no llama.cpp/GGUF path either. The GGUF file and the MLX pack hold the same underlying ternary codes — the MLX pack's own bundled runtime/codec.py is explicitly a "Lossless PQ2_0/PTQ1_0 to MLX affine 2-bit block transcoding," and PACK-RUNTIME.md notes the MLX pack's chat template "is copied from the source GGUF" — but they are not interchangeable files. "Bring the original Bonsai pack" means the 8.6 GB MLX directory, specifically, or the runtime raises an error before it gets anywhere near the ablation code.

The honest cost this repo doesn't measure

Per Ablated.__call__, each of the 129 hooks does one dot product and one scaled subtract over a 5120-dimensional vector — call it on the order of 4 × 5120 ≈ 20,000 floating-point operations, times 129 sites, times however many tokens: roughly 2.6 million extra FLOPs per generated token. Set against a ~27B-parameter forward pass (order 10^10–10^11 FLOPs per token), that back-of-envelope overhead is a rounding error — well under a tenth of a percent. That's not a number the repo states; it's this article's own estimate from the formula, and FLOP count is not the same thing as wall-clock cost.

What the repo does state is a single throughput figure, and it isn't the runtime's own. The README quotes "the ~47 tok/s figure quoted for an M5 Max laptop" from prism-ml's own Apple-platform table — and that table's own heading says it was "measured on the earlier pre-rotation build and reported pending re-measurement on the current stack (llama.cpp Metal backend)." That's three layers removed from what this article can actually check: it's a llama.cpp number, not an MLX number; it's from an earlier build the vendor itself flags as superseded; and it says nothing about what 129 additional hooks per forward pass do to it. Neither this repo nor the model card it borrows the number from publishes an alpha=0-versus-alpha=1 throughput comparison on the current MLX stack.

Source: prism-ml's own README for the pack, explicitly flagged there as a pre-rotation measurement pending re-run — not this repo's numbers, and not measured with the ablation hooks installed.

There's a real, unverified reason to expect the hooks to cost more than their FLOP count suggests: decode on this architecture is memory-bandwidth-bound (every token reads the whole 8+ GB weight set), and the pack's own GGUF card separately notes that batch-1 decode on several accelerators is "limited by instruction throughput and launch overhead" rather than raw compute. 129 additional small MLX ops per token, each a separate dispatch in the graph, sit in exactly that regime. Whether that shows up as measurable latency isn't something this article can determine without running the repo — which is also, notably, something the repo's own README never reports having done.

Weight edit vs. runtime hook

What runtime intervention is actually better at

The FLOP count is a wash and the throughput question is open, but one advantage here is real and doesn't need a benchmark to establish it: the same weights answer both --alpha 0 and --alpha 1. A weight-edited release either keeps the pre-edit checkpoint around separately or loses the ability to A/B against it; this one can't lose that ability, because there's only ever been one checkpoint. scripts/selfcheck.py measures the residual stream's component along r before and after install() — the repo's own bar for "correctly wrapped" is driving that component to roughly 1e-6 of the residual norm — and that check runs against the same loaded pack both times, which is a cleaner comparison than diffing two different checkpoints could ever be. The --layers flag extends the same idea: sweeping which layers get wrapped, or how strongly, is a runtime argument rather than a new file, which is precisely the affordance an interpretability researcher wants and a shipped, edited checkpoint can't offer.

The 5,120 numbers nobody checks

Everything above is about the operator. The other half of this release is the vector, and that half gets no verification at all.

Arditi et al.'s claim isn't only that ablating a direction stops refusal — it's that adding the same vector back causes refusal. Their §3.2 adds the difference-in-means vector to residual activations at the single layer it was extracted from, across all token positions, and generates over 100 harmless Alpaca instructions. Refusal scores go from 0.00–0.03 to 0.88–1.00 on twelve of the thirteen models (Llama-3 70B Instruct is the lone holdout, at 0.30). That bidirectional test is what upgrades "a direction whose removal happens to stop refusals" into "the direction that mediates refusal" — and it's the half of the paper that abliteration releases, this one included, never repeat.

expand

Getting such a vector is a search, not a measurement. The paper computes a difference-in-means vector for every (layer, post-instruction token position) pair, then selects the one candidate with the lowest bypass score subject to three filters: it has to induce refusal when added, it has to leave harmless prompts roughly alone in KL divergence, and it must not sit too close to an unembedding direction — otherwise you haven't found a feature, you've just suppressed the token the refusal happens to start with. Table 5 records the chosen layer and token position for each of the thirteen models. Figure 11 is why the search is needed at all: on Llama-3 8B Instruct the first six layers yield nothing, one of the five candidate positions (<|start_header_id|>) never works at any depth, and the band that does work is a few layers wide around layer 12.

expand

OrcaBonsai ships the vector and none of that. refusal_dir.safetensors carries five metadata keys — model, norm, basis, kind, hidden — and no layer, no token position, no dataset, no scores. scripts/selfcheck.py prints layer={meta.get('layer')} on the line where it announces the direction it just loaded: the slot is there in the reader, and the shipped file leaves it empty.

That matters because the self-check proves less than its PASS suggests. residual_components() computes |<h, d>| / ||h|| for whatever d it's handed, and install() wraps all 129 writers with Ablated(inner, d) using that same d. Hand it a random unit vector instead and the ratio still collapses, because with every residual writer projected the accumulated stream is orthogonal to d by construction, whatever d is — and the ~1e-6 floor the repo quotes is Ablated casting its float32 result back to the module's float16 dtype, not a fact about which vector was projected. The check establishes that the operator is installed. It cannot establish that the vector points at refusal. The experiment that would is the activation-addition half above, and it needs no judge model and no eval harness — only the refusal-substring list the paper publishes.

What isn't shown

The repo's own evals.jpg is the only refusal-rate evidence here, and it's presented as an image, not as data — no harness, no judge model, no decoding settings anywhere in the repo, and (per this article's callout above) not reproduced here beyond the numbers themselves. The direction file's own caveat is worth taking at face value rather than skipping past: directions/direction.json says the refusal direction "was estimated on the bf16 base model this pack was trained from," and that "how well it transfers across the quantization-aware training has not been measured yet." That's the repo being straightforwardly honest about a real gap — the math the runtime applies is exact regardless of quantization, but exactness of the projection doesn't by itself establish that the direction still points at the same behavioral feature after QAT. No independent replication of any of this exists that this article could find.

The ledger

Well supported. The mechanism: projecting an output and projecting the weight matrix that produced it are the same operation by simple associativity, so moving the intervention to activations is mathematically exact, not a workaround. The 129-site count, reconstructed independently from the pack's own config.json. The "no weight modification" claim, checked by reading every line in the repo that touches the --pack path and finding no write call anywhere in it. The distinct byte sizes of the MLX pack (8,595,477,990 bytes) and the GGUF release (5,946,648,928 bytes) that share the "Bonsai 2 27B" name but aren't interchangeable with this runtime.

Thin. The performance claim: the one throughput number in the README is inherited, from a different backend, from a build the vendor itself calls superseded, and says nothing about the cost of the 129 hooks it exists to describe. The refusal-rate table: real sample sizes, no harness, no judge, one unexplained asterisk, and no independent replication.

Not shown. Any alpha=0-vs-alpha=1 throughput comparison on the actual MLX runtime. Any measurement of how well a direction estimated on the bf16 base transfers through this model's own quantization-aware training — a gap the repo names itself rather than one this article had to find. Any provenance for the direction at all: the shipped safetensors records the model, the norm and the basis, but not the layer, the token position, the datasets or the selection scores that Arditi et al. treat as the entire difficulty of the method — and selfcheck.py's PASS is a statement about the operator, not about the vector it was handed.

Related on this site: Bonsai 27B for the ternary and 1-bit encodings, the group-wise scale/bias identity, and the honest, uneven cost of pushing low precision through an entire network; GLM-5.3-Flash-Uncensored-FP8 for the same refusal-ablation technique done as a weight edit, checked by tensor-inventory match instead of by reading for an absent write call; and GLM-5.3-Flash-MLX for what OrcaRouter's own re-quantization tradeoffs look like when the technique in question is a bit-width change rather than a refusal edit.