Projecting out Whisper's hallucinations: abliteration, in reverse

Point a microphone at a rooster and Whisper large-v3 transcribes "i'm the best." Point it at a car horn and it writes "the train is coming up." Feed it a jackhammer and it writes "thank you." None of these clips contain speech. Abbasihafshejani and Jadliwala's paper, "Reducing Hallucinated Transcripts in Whisper via Hallucination Space Projection," fixes this without touching a single weight: it estimates a low-rank direction in Whisper's decoder activations from non-speech calibration audio, and subtracts it out at inference. Averaged across three model scales and three non-speech benchmarks, hallucination rate falls from 31.31% to 2.44%.

That is the same move this site covered in an abliterated GLM-5.3-Flash release: find a contrastive direction in activation space, project it out. Arditi et al.'s result was that a single direction mediates refusal in chat models, and ablating it turns off a safeguard. This paper runs the identical technique — estimate a subspace from paired contrastive activations, subtract it at inference, no gradient step anywhere — to turn off a failure mode instead. Same math, opposite target, and the same question follows it into a new domain: what does the model give up when you edit it this way?

Why this happens

Whisper (Radford et al., 2022) is a single auto-regressive decoder doing two jobs at once: generating transcript tokens, and estimating a no_speech_prob — the probability mass Whisper's own decoder assigns to a special <|nospeech|> token. The official implementation's decision rule is simple: if no_speech_prob clears a threshold $\tau$ (0.6 by default), discard whatever text got generated and return an empty transcript. The problem is that generation and rejection are two different computations sharing one decoder, and nothing forces them to agree. The decoder can assign high probability to a fluent token sequence — an acknowledgment, a subtitle-style sign-off, a repeated word — even when no_speech_prob never gets close to $\tau$.

Koenecke et al. (2024) found that roughly 1% of Whisper transcriptions overall contain unsupported content, over a third of it potentially harmful or misleading. Baranski et al. (2025) measured the non-speech case directly and found Whisper large-v3 emits non-empty text for 40.3% of pure non-speech inputs — acknowledgments, applause markers, animal-sound onomatopoeia, subtitle endings. A few of their examples, reproduced here exactly as Whisper transcribed them:

Table 4 from the paper — representative hallucinated transcripts on non-speech audio.

Existing fixes intervene somewhere else in the pipeline: an external voice-activity detector removes non-speech regions before Whisper ever sees them (WhisperX); a phrase-list filter removes known-bad strings after decoding (Bag of Hallucinations); a fine-tune recalibrates the decoder's "crazy heads" on non-speech examples with empty targets (Calm-Whisper). This paper's move is different in kind: reach directly into the decoder's own hidden states, mid-generation, and remove the direction that produces the hallucinated text — before a single token commits.

The method: find a subspace, subtract it

The estimation step (Section 3.1–3.2) needs paired contrastive data: non-speech clips where Whisper hallucinates, and non-speech clips where it correctly stays silent. Run unprojected Whisper over a calibration set, sort by whether it hallucinated, and collect decoder hidden states at a chosen layer $\ell$ from both groups:

Both groups are non-speech audio, so subtracting them cancels out whatever "this is a non-speech clip" looks like in activation space and isolates what's left over: the direction specific to hallucinating on non-speech rather than correctly rejecting it. Pair them up ($n = \min(N_h, N_f)$) and stack the differences into a matrix:

Take the SVD, $\Delta_{\ell} = U_{\ell}\Sigma_{\ell}V_{\ell}^{\top}$, and keep the top $r$ right singular vectors as a row-orthonormal basis $B_{\ell,r} = V_{\ell,1:r}^{\top} \in \mathbb{R}^{r \times d}$ — this is ordinary PCA on the difference vectors, the same construction Arditi et al. use for a rank-1 refusal direction, generalized to rank $r$. At inference, every decoder hidden state at layer $\ell$ gets the component along that basis removed:

$\alpha$ scales how much of that component comes out; $\alpha=1$ removes it completely, in the direction $B_{\ell,r}$ spans. Drag it below and watch what happens to a decoder state on either side of that subspace:

Applying this to every input is the always-on variant, and it works — but it also edits decoder states for genuine speech that never needed it. So the paper gates the intervention on Whisper's own judgment: run one pass unprojected, read off no_speech_prob, and only apply Eq. 1 if that probability already clears a (lower, separate) gate threshold $\gamma$. If the gate doesn't fire, decoding proceeds unmodified. If it does, Whisper decodes a second time with the projection active, and that run's no_speech_prob is what gets compared against $\tau$ for the final accept/reject call:

Algorithm 1 — Gated Low-Rank Decoder Projection
Input: model M, audio x, layer ℓ, basis B, strength α, gate γ, threshold τ
 
1. y_base, p_base ← M(x)                     # one unprojected pass
2. if p_base ≥ γ:
3.     attach projection hook at layer ℓ
4.     y, p ← M(x) with h'_ℓ = h_ℓ − α(h_ℓ B^T)B during decoding
5. else:
6.     y, p ← y_base, p_base                 # unchanged
7. return "" if p > τ else y

Two thresholds, two jobs: $\gamma$ decides whether the decoder gets edited at all; $\tau$ — Whisper's own, unchanged, default 0.6 — decides whether the (possibly edited) result gets kept. This is worth sitting with, because it explains the gate's entire value proposition, which Figure 3 of the paper shows directly:

expand

The non-speech distribution shifts hard to the right of $\tau$ — projection is doing exactly what it's supposed to. The speech distribution barely moves, because for real speech the gate mostly never fires in the first place: no_speech_prob starts near zero, stays under $\gamma$, and the projection hook never attaches. That's the mechanism behind gated projection's WER advantage over always-on, made concrete in a histogram rather than asserted.

Calibration: what it costs to build the subspace

The entire subspace comes from ESC-50 folds 1–3 — 1,200 environmental audio clips, a small, standard, freely downloadable benchmark. No paired human transcription is needed beyond what Whisper itself produces: run unprojected inference, split by hallucinated-vs-empty, take the SVD. No gradient computation anywhere in the pipeline. LibriSpeech validation-clean (2,703 clips) is used alongside it, but only to select $(\ell, r, \alpha, \gamma)$ by checking that a candidate setting doesn't wreck WER on real speech — it plays no role in defining the subspace itself.

The generalization result is the part worth taking seriously: a subspace built entirely from ESC-50 folds 1–3 is then evaluated, unchanged, on held-out ESC-50 folds 4–5, on UrbanSound8K (8,732 clips the subspace never saw), and on a filtered non-speech subset of FSD50K (8,621 clips). It transfers. That's evidence the projection is picking up a reusable decoder-level signature of "about to hallucinate" rather than overfitting to ESC-50's specific acoustic palette — though the paper is careful to flag, in its own limitations section, that broader calibration-to-deployment shifts (different domains entirely) remain untested.

The headline number, unpacked

Table 1 reports hallucination rate on the three non-speech test sets, at Whisper's default $\tau=0.6$, across all three model scales:

Table 1 — non-speech hallucination rate (%), τ=0.6.

The abstract's 31.31% and 2.44% are the mean of all nine Original cells and all nine Always-on cells in this table — not one dataset, not one model, the full 3×3 grid. It's a reasonable way to headline the result, but it also means the number papers over real spread: original hallucination rate on UrbanSound8K alone runs as high as 76.08% for large-v3, while FSD50K sits at 21.35%, a 3.5x gap in how bad the underlying problem is before any intervention runs. The gated column has its own oddity worth flagging directly — large-v3's gated hallucination rate (8.38% on ESC-50) is worse than both small's (1.53%) and medium's (2.75%), even though large-v3 uses the lowest gate threshold ($\gamma=0.05$, meaning it should fire the most often of the three). The paper doesn't explain this reversal, and neither can this article from the outside — it's a real data point that "bigger model, lower threshold" doesn't straightforwardly mean "more suppression," sitting right there in the paper's own table.

The honest core: what gated projection costs

Non-speech hallucination rate is the number that gets headlined. LibriSpeech word error rate is where the bill comes due — this is the paper's own honesty test, run on real speech with real reference transcripts, and it's the number to hold onto:

Table 2 — LibriSpeech WER (%).

Table 3 — LibriSpeech speech false-rejection rate (%): genuine speech incorrectly filtered as non-speech.

Subtract original from gated on each row and you get exactly the abstract's range: +0.33 points for small on test-clean, up to +4.39 points for medium on test-other. Nothing about rank, layer, or gate threshold changes within that range — it's produced entirely by which model scale and which split you happen to evaluate on, which is the kind of thing a single headline number cannot show and a table has to. Play with both axes below:

Rank and layer: why bigger models needed a wider subspace

The paper doesn't fix rank in advance — Figure 1 sweeps decoder layer $\ell$ against rank $r$ on the development split, holding $\alpha=1$ and no gating, before any gate parameters get chosen at all:

expand

Two things fall out of this grid. First, depth matters far more than width: early layers (4, 8, 12) barely move hallucination rate no matter the rank, while the effective range sits in the middle-to-late layers — a pattern echoed in Whisper small and medium's own sweeps (Appendix C). Second, rank saturates instead of scaling smoothly — going from $r=1$ to $r=8$ doesn't monotonically improve hallucination rate, and past a point it actively hurts WER, which is exactly what "unnecessary removal of speech-relevant information" looks like in a heatmap. The three final selected configurations end up small and different across scales:

Rank climbs from 1 to 4 as the model gets bigger — the same open question this site raised about GLM-5.3-Flash-Uncensored's residual refusal, inverted. There, an 11–18% residual refusal rate after single-direction ablation raised the question of whether refusal is really mediated by one direction in every architecture, or whether some models need more. Here the paper answers that question empirically, for hallucination rather than refusal: Whisper-small's failure mode collapses onto a single dominant direction ($r=1$ is already enough), but large-v3 needed four — a genuinely low-rank subspace either way, nothing close to a general-purpose edit, but not a single line either.

Two more ablations worth the honest read

Section 5.1's development-set search picked $\tau=0.6$ (Whisper's own unmodified default) without re-tuning it for the projected model. Appendix D.1 checks whether that default is still right once projection changes the underlying probability distribution, sweeping $\tau$ against gated projection on Whisper-medium (fixed at $\ell=24, r=2, \gamma=0.10$, but with $\alpha=1.0$ here rather than medium's main-text setting of 0.75 — the paper's own ablation, not this article's substitution):

Table 5 — no-speech threshold ablation, Whisper-medium.

The default holds up as the best balance point — pushing $\tau$ down to 0.4 buys hallucination rate down to 2.89% but pushes WER past 11% and FRR past 10%; pushing it up to 0.7 recovers WER and FRR but lets hallucination rate climb back to 7.85%. Worth reading the Original rows on their own, too: raising $\tau$ from 0.4 to 0.7 with no projection at all barely moves WER (5.77% → 5.46%) while hallucination rate nearly quadruples (13.08% → 42.75%) — threshold tuning by itself is a blunt, mostly-useless instrument here, which is the paper's own justification for doing something at the activation level instead.

Appendix D.2 asks whether stacking two projected layers helps, applying $(\ell=20, r=4, \alpha=0.5)$ and $(\ell=24, r=2, \alpha=1.0)$ together, gated at $\gamma=0.10$:

Table 6 — multi-layer projection ablation, Whisper-medium.

Stacking layers pushes hallucination rate down further on every non-speech set (ESC-50 goes from 2.75% to 0.13%), and roughly doubles WER and FRR on both LibriSpeech splits at the same time. This is the same trade-off surface the whole method lives on, at a different operating point — more suppression is available, it just keeps costing the same currency. The paper keeps single-layer gated projection as its recommendation for exactly this reason.

What a stock external VAD already buys

Section 5.4 compares against training-free baselines that don't touch the decoder at all: WhisperX's external voice-activity detector filtering audio before it reaches Whisper, and a reimplementation of Barański et al.'s Bag-of-Hallucinations phrase filter (no public BoH implementation exists, so the paper built one from ESC-50's own frequent hallucinated phrases — "thank you," "the end," "meow meow," eleven phrases in Appendix E). All numbers below are Whisper large-v3, averaged over the same three non-speech sets as Table 1:

expand

Phrase filtering barely dents the problem — it can only remove hallucinations that match a phrase already seen in its calibration list, and most of Whisper's non-speech vocabulary isn't that repetitive. The real tension is between the other two: gated projection's average hallucination rate (4.09%) actually beats WhisperX's (5.24%) — a stronger result than a purely training-free VAD pipeline, and one that needs no external model. But turn to speech quality and the ranking flips hard. At $\tau=0.5$, WhisperX reaches 2.67% / 4.74% WER on LibriSpeech test-clean/test-other with 0% FRR on both — genuinely better than Whisper's own unprojected baseline (4.15% / 5.87%), because removing non-speech regions before decoding gives the model cleaner context to work with. Gated projection's cost at that same operating region is 6.17% / 6.57% WER with real false rejection (2.86% / 1.40%). An external VAD is a dependency the projection method avoids; it also, on this paper's own numbers, costs almost nothing where projection costs several points of WER.

One more comparison point, transcribed rather than re-run: Calm-Whisper (Wang et al., 2025), a fine-tuning-based mitigation, reports 15.51% hallucination rate on UrbanSound8K — worse than either projection variant here (2.74% gated, 0.87% always-on) — but LibriSpeech WER of just 2.19% / 4.13%, well under gated projection's 6.17% / 6.57%. No public Calm-Whisper checkpoint exists, so the paper doesn't re-run it in its own pipeline, and this article can't either — but the shape of the comparison is legible from the paper's own reported numbers alone: a method willing to update weights buys a cheaper WER trade than one that only edits activations, at the price of weaker suppression and an actual training run.

Same math, opposite target

Return to where this piece started. Arditi et al. found that refusal in open chat models concentrates along a direction identifiable from a small contrastive prompt set, and that ablating it turns off the safeguard almost entirely. This paper's related-work section names its own closest precedent directly: Nullu (Yang et al., 2025), which projects vision-language activations out of a "HalluSpace" to suppress object hallucination — this paper's title borrows the construction and moves it to speech. All three do the identical thing mathematically: collect paired contrastive activations, take an SVD (or, in Arditi's case, the rank-1 special case of the same idea), and subtract the resulting subspace from live activations at inference. No weights change. No gradient step runs.

What differs is only what gets removed and what breaks when you remove it. Arditi's ablation is applied to a safety behavior — the cost, when the ablation is too aggressive or too narrow, is measured in how much harmful content gets through, or in this site's own read on a specific release, in an 11–18% residual refusal rate that a single direction didn't fully explain. This paper's projection is applied to a reliability behavior — the cost is measured in WER and FRR on the model's actual job. Both papers describe their method as offering a "controllable trade-off," and both are right to. But they're not the same trade: a hallucination-suppression method that costs WER is failing at the thing Whisper is for, in a way that's directly measurable on every speech benchmark that exists. A refusal-suppression method that fails is failing at something layered on top of the model's core competence — harder to measure, easier to hand-wave. Reading these two papers side by side is a reminder that "we found a subspace and projected it out" is a description of a mechanism, not a verdict on whether doing so was a good idea. That verdict depends entirely on what sat in the subspace.

What isn't shown

The paper's own limitations section is unusually direct, and worth repeating rather than softening. The method is scoped to one specific failure mode — a non-speech input producing a non-empty transcript — and explicitly does not address hallucination in long-form transcription, multilingual audio, or acoustically ambiguous speech-adjacent inputs, where hallucination looks different and this subspace was never estimated to catch it. The gated variant still increases WER and FRR at every tested operating point; it reduces always-on's damage, it doesn't eliminate the trade-off. The subspace transfers from ESC-50 to two other benchmarks, but broader calibration-to-deployment domain shifts (a call-center corpus, a different language, a different microphone chain) remain untested. And the gate itself leans on Whisper's own no_speech_prob being a trustworthy signal in the first place — for the identical reason Whisper hallucinates in the first place, that estimate is not independent of the same decoder doing the generating.

None of that erases the headline result. A rank-1-to-4 subspace, estimated once from 1,200 clips of a public dataset with no gradient step, cuts non-speech hallucination by a factor of roughly ten to fifty depending on model scale and dataset — that holds up. It just isn't free, the paper says so in its own tables, and a method's honesty about its own cost is worth exactly as much attention as its headline.

Related on this site: GLM-5.3-Flash-Uncensored for the abliteration side of this exact technique, including the residual-refusal question this piece answers from the other direction; speech-to-speech for a voice pipeline built around the external-VAD approach this paper benchmarks against; and Audex for a different bet on ASR reliability — bolting speech onto a strong text decoder rather than editing an existing one's activations.