# Ternary Bonsai 2 27B Uncensored: the numbers behind a runtime abliteration

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/runtime-abliteration
> date: 2026-09-18
> tags: quantization, alignment, safety, on-device, apple-silicon, interpretability
[`Continuum-AI-Corp/OrcaBonsai-27B-Uncensored`](https://github.com/Continuum-AI-Corp/OrcaBonsai-27B-Uncensored) — published as **"OrcaRouter Ternary Bonsai 2 27B Uncensored"** — is a refusal-ablation release with no modified checkpoint. It applies the same technique this site checked in [OrcaRouter's GLM-5.3-Flash-Uncensored-FP8](/articles/glm-5-3-flash-uncensored) to [PrismML's Bonsai family](/articles/bonsai-27b), but on a base model that makes the usual approach expensive in a specific, checkable way: [prism-ml/Ternary-Bonsai-2-27B-mlx-2bit](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) is quantized to roughly 1.72 bits per weight by quantization-aware training (QAT), and editing weights on a QAT model means dequantizing, editing, and re-quantizing — a second lossy step stacked on top of the first. OrcaRouter's fix: don't touch the weights. Hook the residual stream instead, at inference, on every module that writes into it.

<Callout type="note">
This article checks claims against the cloned repository's code and against Hugging Face's own metadata for the packs involved. It names the technique and the public research behind it, but does not describe how to remove refusal training as a recipe, and it does not quote, paraphrase, or otherwise reproduce any of the model's generated outputs. What follows is what the code and the file metadata will and won't support.
</Callout>

| | |
|---|---|
| Repo | [`Continuum-AI-Corp/OrcaBonsai-27B-Uncensored`](https://github.com/Continuum-AI-Corp/OrcaBonsai-27B-Uncensored) — a runtime, not a model repo |
| Depends on | [`prism-ml/Ternary-Bonsai-2-27B-mlx-2bit`](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-mlx-2bit), supplied by the user, unmodified |
| Base model | `Qwen/Qwen3.8-27B`, per the pack's own `base_model` metadata |
| Technique | Refusal-direction ablation ([Arditi et al., 2024](https://arxiv.org/abs/2406.11717)), applied as a runtime forward hook |
| Residual writers wrapped | **129** — `mlp.down_proj`&times;64, `linear_attn.out_proj`&times;48, `self_attn.o_proj`&times;16, `model.embed_tokens`&times;1 |
| Weight modification | None — no `save_safetensors` or file-write call anywhere in the runtime's own code |
| License | Apache-2.0, matching the base pack |
| Platform | Apple Silicon (MLX); CPU-only on x86 Linux — no CUDA kernel for the pack's quantized matmul |

## What abliteration is, briefly

[Arditi et al., 2024](https://arxiv.org/abs/2406.11717) ("Refusal in Language Models Is Mediated by a Single Direction") found that across a range of open chat models, refusal concentrates along one direction in the residual-stream activation space, and that projecting that direction out of every module that writes the residual stream is usually enough to make a model stop refusing almost entirely. This site went through that mechanism and its limits in more detail when it checked [OrcaRouter's abliteration of GLM-5.3-Flash](/articles/glm-5-3-flash-uncensored); the short version repeats here only as far as it's needed to make sense of what this repo does differently.

"Almost entirely" is worth pinning to the paper's own numbers rather than to the phrase. Across 13 open chat models up to 72B, over 100 harmful instructions from JailbreakBench, ablation drops the refusal score — a substring match against a published list of refusal openers — to between 0.00 (Qwen 14B Chat) and 0.27 (Llama-2 70B Chat): near-total on most models, total on none. The safety score, which is Llama Guard 2's judgment that a completion is harmless, falls a good deal less far, to between 0.13 and 0.42. That gap is the paper saying plainly that removing the refusal *phrasing* is not the same thing as producing a compliant answer, and its Appendix D spends three worked examples on the cases where the two metrics disagree. OrcaRouter's own JailbreakBench figure for this pack — 4% residual refusal on the same 100-instruction set — lands inside that band, though with no harness published anywhere in the repo there's no way to tell whether it was scored the same way.

<Figure
  src="/articles/runtime-abliteration/fig1.png"
  alt="A grouped bar chart over 13 chat models from Qwen 1.8B to Llama-3 70B. For each model, solid orange and blue bars (refusal score and safety score with no intervention) sit between 0.62 and 0.99, while the hatched bars for directional ablation drop to between 0.00 and 0.27 for refusal score and between 0.13 and 0.42 for safety score."
  caption="Refusal score and safety score over 100 harmful JailbreakBench instructions, with no intervention (solid) and under directional ablation (hatched). Refusal collapses on every model; the safety score, which asks whether the completion is actually harmful, falls much less far (Arditi et al., Figure 1)."
/>

The "differently" is the whole point of this release. Every abliteration this site has looked at so far — GLM-5.3-Flash-Uncensored-FP8, [Qwen3.8-Flash-Next-Uncensored](/articles/qwen3-8-flash-next) — ships as a new set of weights: a new Hugging Face repo, a new checkpoint, published alongside or instead of the base model. OrcaBonsai ships none of that. There is no `orcarouter/*Bonsai*` repo on the Hub — a quick check of [OrcaRouter's own Hugging Face org](https://huggingface.co/orcarouter) turns up 28 public repos — 25 "Uncensored" releases across Qwen, GLM, DeepSeek, Nex, and Gemma, plus three plain MLX re-quantizations that carry no ablation at all — and not one of them is Bonsai. The GitHub repo is the whole release.

## Why the usual approach is a bad fit here

Conventional directional ablation is a weight edit. For every matrix `W` that writes the residual stream, you orthogonalize it against a unit refusal direction `r`:

```text
W ← W − r(rᵀW)
```

and save the result as a new checkpoint. That's exactly what happened, at native FP8, when this site checked GLM-5.3-Flash-Uncensored — the edited tensors kept the base model's own dtype and shape, byte for byte, everywhere except the values themselves.

Bonsai 2's weights don't have that option. Per [`bonsai_abliterate/ablation.py`](https://github.com/Continuum-AI-Corp/OrcaBonsai-27B-Uncensored/blob/main/bonsai_abliterate/ablation.py):

```python
"""Apply a refusal direction to a Prism Hadamard ternary pack at run time.

Ordinary abliteration is a permanent weight edit: every matrix that writes the residual
stream is orthogonalised against the refusal direction, ``W <- W - r (r^T W)``. That is
not possible on Bonsai 2 27B. Its weights are ternary -- the affine container stores
``scale = s`` and ``bias = -s``, so the 2-bit codes ``{0,1,2}`` decode to exactly
``{-s, 0, +s}`` -- while the orthogonalised matrix is dense and full precision. Storing
it back would mean re-quantizing to ternary, and the model's quality at 1.72 bits/weight
comes from quantization-aware training, not from the format: re-quantizing without that
training is what destroys it.
"""
```

That's checkable independent of the repo's own framing. The affine container's `bias == -scale` identity — which is what makes `{0,1,2}` decode to exactly `{-s, 0, +s}` rather than three arbitrary levels — is something [this site verified for the whole Bonsai family already](/articles/bonsai-27b), and `scripts/prepare_ios_pack.py` re-verifies it independently before it will drop a bias tensor, checking every one of the pack's 402 packed modules and refusing to proceed if `max |bias + scale|` isn't exactly zero. Orthogonalizing `W` mixes those three discrete levels into a dense matrix of arbitrary floats. Getting back to ternary from there is a second quantization pass, stacked on top of the QAT that produced the original weights — and it wasn't trained with that second pass in the loop.

## Same operator, no weights touched

The runtime's fix follows directly from where the intervention actually needs to sit. Directional ablation doesn't require editing `W` — it requires removing a direction from `W`'s *output*. For any linear map `y = Wx`, projecting the output and projecting the weight matrix first are the same operation, by nothing more than associativity of matrix multiplication:

```text
(I − r·rᵀ)(Wx)  =  ((I − r·rᵀ)·W) x
```

Project `y` after the matmul, or edit `W` before it — the result is identical, because a projection matrix commutes trivially through a preceding linear map. `bonsai_abliterate/ablation.py` computes exactly the left-hand side, per module, per token:

```python
class Ablated(nn.Module):
    def __call__(self, *args, **kwargs):
        y = self.inner(*args, **kwargs)
        yf = y.astype(mx.float32)
        component = mx.sum(yf * self._direction, axis=-1, keepdims=True)
        return (yf - self._alpha * component * self._direction).astype(y.dtype)
```

`self.inner` is the original, untouched `down_proj` / `o_proj` / `out_proj` module — packed ternary weights and all. `Ablated` never reads or writes a weight tensor; it wraps the module's *output*. Because the mathematics doesn't care which side of the matmul the projection happens on, the engineering gets to pick the side that's cheap: the activation side never forces a re-quantization, because there's no quantized thing on that side to re-quantize.

One detail the repo calls out that's easy to get wrong: Bonsai's projections run in a Hadamard-rotated basis on their *input* dimension only, and the runtime's own `Packed` loader un-rotates activations back to the plain hidden basis before they leave each module. Since the refusal direction is projected against *outputs* — which are already in the plain basis — no extra rotation handling belongs in `ablation.py`, and the repo is explicit that adding one "would project against the wrong basis." The direction itself is stored as nothing more exotic than 5,120 little-endian float32 values, matching `hidden_size` exactly.

## 129 sites, mapped

The repo's other easy mistake to make is wrapping too little. `self_attn.o_proj` is the obvious residual writer to reach for, but Bonsai 2 is a hybrid-attention model — most of its layers don't have an `o_proj` at all. Pulled straight from the pack's own `config.json` (fetched from the Hub, not guessed): 64 layers, `hidden_size` 5120, `layer_types` alternating three linear-attention layers to one full-attention layer — 48 linear-attention, 16 full-attention, exactly matching the repo's claim. Every layer writes the residual stream twice, once from its attention block's out-projection (`linear_attn.out_proj` on 48 layers, `self_attn.o_proj` on the other 16) and once from its MLP's `down_proj` — plus one more write from `model.embed_tokens`. `q_proj`/`k_proj`/`v_proj` and the MLP's `gate_proj`/`up_proj` don't get wrapped, because their outputs never touch the residual stream directly; neither does `lm_head`, which *consumes* the final residual state rather than writing to it.

<ResidualMap />

`64×2 + 1 = 129`. That's not a number this article had to trust — it falls out of counting `config.json`'s own 402-entry `modules` list by path suffix, which is the same count `selfcheck.py` expects and warns about if it doesn't see.

## The strength knob, exactly

`alpha` isn't clamped anywhere in `run.py`'s argument parser — it's a plain `float`, so the README's table (`0`, `0.5`, `0.7`, `0.9`, `1.0`, "over-projection" above `1.0`) is a usage guide, not an enforced range. The formula makes the boundary behavior exact rather than approximate: decompose a write `y` into a component along `r` and a component perpendicular to it, and `Ablated` leaves the perpendicular part untouched while rescaling the parallel part by `(1 − α)`. At `α=1` the parallel part is zero — a full projection, matching a weight edit exactly. Past `α=1`, `(1 − α)` goes negative: the surviving component doesn't just keep shrinking, it flips to point *against* `r`. Nothing in the repo states that directly; it falls straight out of the formula the repo does state.

`α=0` gets special treatment worth noting precisely: `run.py` checks `if args.alpha != 0` before ever calling `install()`. At `α=0`, the wrapping never happens — not "the projection subtracts zero," but "the 129 `Ablated` modules never get constructed," so the forward pass at `α=0` runs the exact same code path as the unmodified pack, not merely a mathematically-equivalent one.

## Bit-identical, checked against the Hub

<ModelCard repo="prism-ml/Ternary-Bonsai-2-27B-mlx-2bit" note="The pack this runtime depends on — the MLX-format directory, not the GGUF release with the same 'Bonsai 2 27B' name." />

"Bit-identical" is a claim about a file this article can actually go check. Hugging Face's blob listing for the pack the runtime requires puts `model.safetensors` at **8,595,477,990 bytes** — 8.005 GiB, which lines up exactly with the README's own "iPhone memory budget" breakdown (ternary codes, redundant biases, scales, norms, and a 0.858 GiB vision tower, summed to the same 8.005 GiB). That's the file `bonsai_abliterate/pack.py` loads via `vision_artifact.load_vl_model`, which in turn calls `mx.load(directory / "model.safetensors")` — a read. Every write call anywhere in `bonsai_abliterate/`, `run.py`, and `scripts/selfcheck.py` writes somewhere *other* than the `--pack` directory: nowhere in this repo's code does anything call `mx.save_safetensors` or `open(..., "wb")` on the path the user points `--pack` at. "Bit-identical" holds here because the code has no path that would make it otherwise — not because anything hashes the pack on load and confirms it.

That distinction matters for a second reason: this repo's own tooling never checks a hash. The pack itself ships `files.json`, a SHA-256 manifest for every file including `model.safetensors`, and `reload-validation.json` (`reload_logits_exact: true`, checked against 248,320 logits) — but both belong to *prism-ml's* pack, verifying prism-ml's own MLX serialization round-trip, and neither is ever read by `bonsai_abliterate/` or `run.py`. If you point `--pack` at a directory that isn't actually the genuine pack, nothing in this repo will tell you.

The size check also catches something this article's brief assumed and had to unlearn: [`Ternary-Bonsai-27B` covered on this site earlier](/articles/bonsai-27b) is a 27B ternary model shipped in more than one container, and it's tempting to think "the original Bonsai pack" means whichever one you happen to have. It doesn't. `prism-ml/Ternary-Bonsai-2-27B-gguf`'s own blob listing puts `Ternary-Bonsai-2-27B-PTQ1_0.gguf` at **5,946,648,928 bytes** — a real file, a real 5.95 GB ternary pack, and the wrong one for this repo. `run.py` imports `mlx.core`, and the pack's `quantized_matmul` kernel exists only for Metal and CPU — there is no CUDA path, and there is no llama.cpp/GGUF path either. The GGUF file and the MLX pack hold the same underlying ternary codes — the MLX pack's own bundled `runtime/codec.py` is explicitly a "Lossless PQ2_0/PTQ1_0 to MLX affine 2-bit block transcoding," and `PACK-RUNTIME.md` notes the MLX pack's chat template "is copied from the source GGUF" — but they are not interchangeable files. "Bring the original Bonsai pack" means the 8.6 GB MLX directory, specifically, or the runtime raises an error before it gets anywhere near the ablation code.

## The honest cost this repo doesn't measure

Per `Ablated.__call__`, each of the 129 hooks does one dot product and one scaled subtract over a 5120-dimensional vector — call it on the order of `4 × 5120 ≈ 20,000` floating-point operations, times 129 sites, times however many tokens: roughly 2.6 million extra FLOPs per generated token. Set against a ~27B-parameter forward pass (order `10^10`–`10^11` FLOPs per token), that back-of-envelope overhead is a rounding error — well under a tenth of a percent. That's not a number the repo states; it's this article's own estimate from the formula, and FLOP count is not the same thing as wall-clock cost.

What the repo does state is a single throughput figure, and it isn't the runtime's own. The README quotes "the ~47 tok/s figure quoted for an M5 Max laptop" from prism-ml's own Apple-platform table — and that table's own heading says it was "measured on the earlier pre-rotation build and reported pending re-measurement on the current stack (**llama.cpp Metal backend**)." That's three layers removed from what this article can actually check: it's a llama.cpp number, not an MLX number; it's from an earlier build the vendor itself flags as superseded; and it says nothing about what 129 additional hooks per forward pass do to it. Neither this repo nor the model card it borrows the number from publishes an alpha=0-versus-alpha=1 throughput comparison on the current MLX stack.

| Platform | Backend | Footprint | TG128 | PP512 |
|---|---|---|---|---|
| Apple M5 Max | llama.cpp, Metal | 7.2 GB | 47.0 tok/s | 765 tok/s |
| Apple M5 Pro | llama.cpp, Metal | 7.2 GB | 28.7 tok/s | 393 tok/s |
| Apple M4 Pro | llama.cpp, Metal | 7.2 GB | 18.0 tok/s | 125 tok/s |

*Source: prism-ml's own README for the pack, explicitly flagged there as a pre-rotation measurement pending re-run — not this repo's numbers, and not measured with the ablation hooks installed.*

There's a real, unverified reason to expect the hooks to cost more than their FLOP count suggests: decode on this architecture is memory-bandwidth-bound (every token reads the whole 8+ GB weight set), and the pack's own GGUF card separately notes that batch-1 decode on several accelerators is "limited by instruction throughput and launch overhead" rather than raw compute. 129 additional small MLX ops per token, each a separate dispatch in the graph, sit in exactly that regime. Whether that shows up as measurable latency isn't something this article can determine without running the repo — which is also, notably, something the repo's own README never reports having done.

## Weight edit vs. runtime hook

<WeightsVsRuntime />

## What runtime intervention is actually better at

The FLOP count is a wash and the throughput question is open, but one advantage here is real and doesn't need a benchmark to establish it: the same weights answer both `--alpha 0` and `--alpha 1`. A weight-edited release either keeps the pre-edit checkpoint around separately or loses the ability to A/B against it; this one can't lose that ability, because there's only ever been one checkpoint. `scripts/selfcheck.py` measures the residual stream's component along `r` before and after `install()` — the repo's own bar for "correctly wrapped" is driving that component to roughly `1e-6` of the residual norm — and that check runs against the *same* loaded pack both times, which is a cleaner comparison than diffing two different checkpoints could ever be. The `--layers` flag extends the same idea: sweeping which layers get wrapped, or how strongly, is a runtime argument rather than a new file, which is precisely the affordance an interpretability researcher wants and a shipped, edited checkpoint can't offer.

## The 5,120 numbers nobody checks

Everything above is about the *operator*. The other half of this release is the *vector*, and that half gets no verification at all.

Arditi et al.'s claim isn't only that ablating a direction stops refusal — it's that adding the same vector back causes refusal. Their §3.2 adds the difference-in-means vector to residual activations at the single layer it was extracted from, across all token positions, and generates over 100 harmless Alpaca instructions. Refusal scores go from 0.00–0.03 to 0.88–1.00 on twelve of the thirteen models (Llama-3 70B Instruct is the lone holdout, at 0.30). That bidirectional test is what upgrades "a direction whose removal happens to stop refusals" into "the direction that mediates refusal" — and it's the half of the paper that abliteration releases, this one included, never repeat.

<Figure
  src="/articles/runtime-abliteration/fig2.png"
  alt="A bar chart over the same 13 chat models. Orange bars for no intervention sit at or below 0.03 refusal score; blue bars for activation addition sit between 0.88 and 1.00 for twelve models, with Llama-3 70B the exception at about 0.30."
  caption="The other direction of the same claim: adding the refusal vector to residual activations at its source layer makes models refuse 100 harmless Alpaca instructions, from a 0.00–0.03 baseline to 0.88–1.00 on twelve of thirteen models (Arditi et al., Figure 3)."
/>

Getting such a vector is a search, not a measurement. The paper computes a difference-in-means vector for every (layer, post-instruction token position) pair, then selects the one candidate with the lowest bypass score subject to three filters: it has to induce refusal when added, it has to leave harmless prompts roughly alone in KL divergence, and it must not sit too close to an unembedding direction — otherwise you haven't found a feature, you've just suppressed the token the refusal happens to start with. Table 5 records the chosen layer and token position for each of the thirteen models. Figure 11 is why the search is needed at all: on Llama-3 8B Instruct the first six layers yield nothing, one of the five candidate positions (`<|start_header_id|>`) never works at any depth, and the band that does work is a few layers wide around layer 12.

<Figure
  src="/articles/runtime-abliteration/fig3.png"
  alt="A line plot titled Bypass score, with the refusal metric on the y-axis running from minus 10 to 10 and source layer 0 to 31 on the x-axis, for Llama-3 8B Instruct. Five coloured lines, one per candidate token position, all sit on the dashed no-intervention baseline near 8.7 through layer 5. Four of them plunge to between minus 7.5 and minus 10 around layer 12 and then climb back toward the baseline by layer 31; the fifth line, for the start-header token, never drops below about 6."
  caption="Bypass score for every candidate refusal direction on Llama-3 8B Instruct, by source layer and source token position; lower means the ablated model refuses less. Most candidates do nothing — the working band is a few layers wide around layer 12, and one token position never works at all (Arditi et al., Figure 11, left)."
/>

OrcaBonsai ships the vector and none of that. `refusal_dir.safetensors` carries five metadata keys — `model`, `norm`, `basis`, `kind`, `hidden` — and no layer, no token position, no dataset, no scores. `scripts/selfcheck.py` prints `layer={meta.get('layer')}` on the line where it announces the direction it just loaded: the slot is there in the reader, and the shipped file leaves it empty.

That matters because the self-check proves less than its `PASS` suggests. `residual_components()` computes `|<h, d>| / ||h||` for whatever `d` it's handed, and `install()` wraps all 129 writers with `Ablated(inner, d)` using that same `d`. Hand it a random unit vector instead and the ratio still collapses, because with every residual writer projected the accumulated stream is orthogonal to `d` by construction, whatever `d` is — and the `~1e-6` floor the repo quotes is `Ablated` casting its float32 result back to the module's float16 dtype, not a fact about which vector was projected. The check establishes that the operator is installed. It cannot establish that the vector points at refusal. The experiment that would is the activation-addition half above, and it needs no judge model and no eval harness — only the refusal-substring list the paper publishes.

## What isn't shown

<RefusalRates />

The repo's own `evals.jpg` is the only refusal-rate evidence here, and it's presented as an image, not as data — no harness, no judge model, no decoding settings anywhere in the repo, and (per this article's callout above) not reproduced here beyond the numbers themselves. The direction file's own caveat is worth taking at face value rather than skipping past: `directions/direction.json` says the refusal direction "was estimated on the bf16 base model this pack was trained from," and that "how well it transfers across the quantization-aware training has not been measured yet." That's the repo being straightforwardly honest about a real gap — the math the runtime applies is exact regardless of quantization, but exactness of the *projection* doesn't by itself establish that the *direction* still points at the same behavioral feature after QAT. No independent replication of any of this exists that this article could find.

## The ledger

**Well supported.** The mechanism: projecting an output and projecting the weight matrix that produced it are the same operation by simple associativity, so moving the intervention to activations is mathematically exact, not a workaround. The 129-site count, reconstructed independently from the pack's own `config.json`. The "no weight modification" claim, checked by reading every line in the repo that touches the `--pack` path and finding no write call anywhere in it. The distinct byte sizes of the MLX pack (8,595,477,990 bytes) and the GGUF release (5,946,648,928 bytes) that share the "Bonsai 2 27B" name but aren't interchangeable with this runtime.

**Thin.** The performance claim: the one throughput number in the README is inherited, from a different backend, from a build the vendor itself calls superseded, and says nothing about the cost of the 129 hooks it exists to describe. The refusal-rate table: real sample sizes, no harness, no judge, one unexplained asterisk, and no independent replication.

**Not shown.** Any alpha=0-vs-alpha=1 throughput comparison on the actual MLX runtime. Any measurement of how well a direction estimated on the bf16 base transfers through this model's own quantization-aware training — a gap the repo names itself rather than one this article had to find. Any provenance for the direction at all: the shipped safetensors records the model, the norm and the basis, but not the layer, the token position, the datasets or the selection scores that Arditi et al. treat as the entire difficulty of the method — and `selfcheck.py`'s `PASS` is a statement about the operator, not about the vector it was handed.

---

*Related on this site: [Bonsai 27B](/articles/bonsai-27b) for the ternary and 1-bit encodings, the group-wise scale/bias identity, and the honest, uneven cost of pushing low precision through an entire network; [GLM-5.3-Flash-Uncensored-FP8](/articles/glm-5-3-flash-uncensored) for the same refusal-ablation technique done as a weight edit, checked by tensor-inventory match instead of by reading for an absent write call; and [GLM-5.3-Flash-MLX](/articles/glm-5-3-flash-mlx) for what OrcaRouter's own re-quantization tradeoffs look like when the technique in question is a bit-width change rather than a refusal edit.*