Auditable AI text detectors: a sparse autoencoder under the verdict

mdjsonmcpcopy↗ai

2026-10-27 · 20 min · interpretability · sparse-autoencoders · ai-detection · evals · small-models

Why read this

Solid· top 85%

Reads the DetectorScope head line by line, then rebuilds the pipeline on a 14M model and shows a dataset shortcut and a false-alarm domain from its own audit.

• Original analysis
• Runs on a laptop CPU
• Concrete numbers to act on

·Interpretability·Apache-2.0·Practitioner paper

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 59 of 100, ranked 387 of 621 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

A paper from Google DeepMind and the University of Pennsylvania, Building Auditable AI Text Detectors with Sparse Autoencoders (arXiv 2610.30006), was shared on X this week. Its claim is narrow and useful. Take the strongest kind of AI-text detector, a fine-tuned language model, and replace the part that decides with something a reviewer can read. Every verdict becomes a sum of named features, and each feature has a location in the text.

This article has three parts. First, what the paper and the detectorscope code do, with the head's arithmetic read from source. Second, a toy rebuild I ran on a CPU: a 14M-parameter model, a sparse autoencoder I trained in numpy, and an L1 probe on top. Third, what the toy's audit found, which is the point of the whole idea: a dataset shortcut that dominates the verdict, a human-text domain it flags 81% of the time, and a length effect.

Every claim is labelled. Reported means the paper or the repository says it. Measured means I ran it. Reasoned means it is my inference and I did not test it.

The pipeline in four moves

The paper's setting is an AI detector that is accurate but opaque, and a stylometric detector that is transparent but weaker. It wants both. The route is a concept bottleneck (reported).

1. Fine-tune a language model to classify text as human or AI. The paper uses Gemma 2 2B.
2. Train sparse autoencoders on its residual stream at four depths (layers 7, 13, 17 and 22 in the paper's Figure 2). A sparse autoencoder, or SAE, rewrites a dense activation vector as a wide, mostly-zero vector of features that are meant to mean one thing each.
3. Label every feature with a short natural-language description and score the label. The score is a simulation test: a model is shown only the description and a set of documents and must guess which activation quantile each document came from. The Spearman correlation with the truth is the feature's score.
4. Keep only features whose score is at least a threshold tau, max-pool them over the document's tokens, and fit an L1-regularised logistic regression on what is left.

expand

The mask in step 4 is the paper's contribution beyond earlier work. Prior SAE probes fed every feature to the classifier, including dense and polysemantic ones, which makes the explanation of the probe less faithful than it looks. Here a feature that cannot be described and verified never reaches the classifier (reported).

What the code does

I cloned detectorscope at commit 52ecc87 (shallow, no large blobs) and read the parts that decide. The package is Apache-2.0, and the model class tags exported repositories cc-by-4.0 (read from source).

AuditableDetector in detectorscope/model/detectors/auditable.py holds a language model with hooks at the inputs of chosen blocks, one JumpReLUSAE per hook, and a DetectorHead. Its defaults are d_model=2304 and d_sae=16384, the Gemma 2 2B sizes, so four hooks give a 65,536-wide feature vector per document. The pooling is two lines:

# auditable.py
def pool(sae_acts):
    return torch.concat([a.max(dim=1)[0] for a in sae_acts], dim=1)

Max over the sequence, then concatenate layers. That one choice defines what "a feature fired in this document" means everywhere downstream, including in the caches the heads are trained on.

The head is the part worth reading slowly. In heads/logreg.py:

# LogRegHead.get_importance_scores
return (self._mask_input(x) * self._weight).cpu()

For a linear model the logit is the sum of w_i * x_i plus a bias. So that product is simultaneously each feature's additive contribution and its exact zero-ablation effect. The docstring says so: no extra forward passes, no approximation. Detection.residual is defined as logit - bias - sum(importances) and is zero for the logistic head. The XGBoost head is different. A tree ensemble is piecewise constant, so zeroing an active feature often changes nothing and its importance is exactly zero (read from source).

Three smaller things from the code:

• The mask is folded into the weights when a head is saved (_weight * interp_mask), so a released logreg_tauX.XX.safetensors file has no separate mask, and the threshold sits in the file header. A detector repository carries one head per (type, tau) pair, named by that tau.
• A verdict is "ai" if the logit exceeds the head's stored threshold. Highlights come from a sparse per-token matrix kept next to the pooled vector, so you can see which token each feature peaked on (read from source).
• The description generator bins documents into 4 activation quantiles plus an inactive bin and highlights tokens above 10% of the feature's maximum (n_quantiles=4, HIGHLIGHT_THRESHOLD = 0.1). The paper uses a Gemini model to write and score descriptions (reported).

The README's usage is one call; the printed Detection lists the top three features with signed importances and descriptions.

from detectorscope import AuditableDetector
detector = AuditableDetector.from_pretrained("liamdugan/detector-scope")
print(detector("Like I said, this is about accuracy, about speed, and about cost."))

I did not run this. It needs the Gemma 2 2B weights and the released SAEs, which do not fit the disk and CPU budget of this session.

What the paper reports

All numbers in this section are the paper's, reported, not mine.

On the COLING shared-task data (English split of 610k training documents from 41 generators, tested on a development set and two out-of-distribution test sets) and on the RAID benchmark, three systems are compared. Base-SAE uses the pretrained Gemma Scope SAEs on the original model, so only the logistic weights are trained. FT-SAE fine-tunes the SAEs on the task. FT-Model+SAE fine-tunes the detector first and then its SAEs.

• FT-Model+SAE recovers the black-box detector. On the in-distribution development set, with the mask applied, it scores 99.2 AUC and 96.8 F1, against 99.8 and 98.0 for fine-tuned gemma-2-2b.
• When the black box fails to generalise, so does FT-Model+SAE. On the devtest split the black box has 86.8 F1 and the masked FT-Model+SAE 84.8.
• The surprise is the base-model systems. Without any detector fine-tuning, masked Base-SAE gets 88.6 F1 and masked FT-SAE 94.2 on devtest, against 81.6 for Binoculars and 75.5 for Fast-DetectGPT. The authors attribute it to linearly separable "AI text" features already present after pre-training, and say they leave the cause to future work.
• On the peer-review benchmark PRDB, every trained classifier is near 100 AUC. The authors read that as a dataset that is too easy, which makes it a good place to audit.

expand

Two causal checks matter more than the accuracy tables. In the feature-level paraphrase ablation (Table 3), a model is asked to rewrite a document so that one named feature disappears, and the change in the detector's logit is correlated with the feature's importance. The correlation is positive in all twelve configurations, 0.69 to 0.81 on PRDB and 0.36 to 0.62 on COLING, and the mask raises it. In the document-level ablation the authors give a paraphrasing model the detector's own active-feature list and report that it flips 60% to 100% of predictions, against at most 8.7% for a generic request to sound more human. They take that as evidence the lists are high-recall. It is also a plain statement about publication: a feature list that is good enough to audit by is good enough to rewrite by. I do not go further into it here.

The qualitative section is the one I wanted to reproduce in miniature. The top PRDB features are specific and mostly formatting: double colons after bold markdown headers (11 human documents against over 54,000 GPT-4o ones), an opening square bracket (7 against over 22,000 Llama ones). The human-side features are stylistic: sentence-initial "I", "seem", "see". On COLING the Base-SAE's top features are topical and licence-boilerplate signals, which the authors read as domain shortcuts. They note that whether a formatting quirk counts as spurious is a policy question, and that the system's value is that a provider can decide case by case (reported).

Their stated limits (reported): auto-interp descriptions are biased to recall over precision, which an informal regex test found affects roughly half the features; and only Gemma Scope on Gemma 2 2B was tried.

A toy rebuild

The question I can answer on a CPU is not whether the paper's numbers hold. It is whether the audit mechanism does what the paper says it does, on something small enough to look at in full. I built the same four moves at a scale where every piece is inspectable.

Measured setup. The model is pythia-14m (6 layers, width 128, Apache-2.0), run through a numpy forward pass I wrote for GPT-NeoX. The check I have is that it gives 5.14 nats per token on one English document, against 10.83 for a uniform guess. I have no reference implementation to compare to, so that is a sanity check and not a proof of equivalence. I read the residual stream entering block 3 and used the first 128 tokens of each document.

The data is HC3 (CC BY-SA 4.0): questions with a human answer and a ChatGPT answer. I trained on two domains, reddit_eli5 and finance (1,400 documents per class pair, so 2,800 in all, split 2,240 to fit and 560 to validate), and tested on three the model never saw: open_qa, medicine and wiki_csai, 800 documents each. The test documents average 104 tokens.

The SAE has 1,024 features, a ReLU and an L1 penalty, a unit-norm decoder, and 7,000 Adam steps on 247,450 token activations. On 20,000 tokens it leaves 2.4% of the variance unexplained at a mean of 43 active features per token, with no dead features. I wrote the backward pass by hand. The probe is an L1 logistic regression on max-pooled features, with the penalty chosen on the validation split. The whole run takes about 14 minutes on 2 threads of a shared box; the script is at public/articles/sae-text-detectors/sae_probe.py, seeded, and a re-run agrees with the figures below to about three decimals, not to the last digit.

There is no LLM here, so there is no description and no simulation score. My stand-in for the score is concentration: the share of a feature's activation mass (over tokens where it is above 10% of its own maximum) that sits on its single most common token. A feature with concentration 0.97 is a "comma" feature. One with 0.43 is a loose family: its top token is " just" and the next are " only", " simply", " really". It is a crude proxy and it favours token detectors, which is the kind of feature a one-layer 128-wide SAE finds. It is not the paper's score.

Reading a verdict

At tau 0.4, 148 of 1,024 features are kept and 59 get a non-zero weight. The 12 documents below come from the held-out domains: for each one, the highest-scoring human document, a high-scoring AI document, a low-scoring human document and the lowest-scoring AI document.

What the 12 explanations show, all measured:

• Feature 154 fires on " the" and "," in every one of the 12 documents and contributes between -2.8 and -3.5 each time. It says almost nothing about a document. My reading, reasoned and not tested, is that it stands in for the intercept: scikit-learn's liblinear solver penalises the bias, and the fitted bias came out as exactly zero. An always-on, high-density feature of this kind is what the paper's mask is meant to keep out. My crude score let it through, since it is in the kept set.
• Feature 684 fires on a newline (top token \n, concentration 0.99) and supplies up to +4.3 of a document's logit. Feature 820 fires on a space before a period and pushes the other way (about -2.9 where it appears). Feature 432 fires on double spaces and gives -4.7.
• Most of the rest are small and legible: . after a sentence (+1.5 to +1.8), , (about +1), " or" (+0.9), " as" in "such as" (+1.2); " I" (-1.3 to -1.5) and " don" (-0.8) toward human.
• The lowest-scoring AI document is a refusal, "I'm sorry, but I don't understand what you're asking." The first-person and contraction features pull it to a logit of -4.4 and the detector passes it. A refusal has a very recognisable pattern, and this detector misses it because its features are about tokens, not about the move being made.

That is the whole idea: the verdict has no hidden remainder. Each bar is w * x, the bars plus "others" sum to the logit, and a reviewer can disagree with a specific line.

Where the shortcut comes from

Features 684, 820 and 432 are formatting. I went back to the raw text and counted, in exactly the documents I used.

Measured, from the jsonl files. In the training data, a newline is a nearly perfect marker of ChatGPT and a space before a period a nearly perfect marker of the human side. Reasoned: the human spacing is residue from how the human corpora were tokenised before they were packaged, not something people type. The detector learned the packaging. That is the same shape of problem the paper finds with Base-SAE features that track licence boilerplate, and the audit surfaces it in one reading. It also fails in a specific place: the wiki_csai humans have newlines, because Wikipedia paragraphs do.

The threshold sweep

I swept tau from 0 (every feature with 20 or more firing tokens, 1,024 in all) to 0.7 (67 kept), refitting the probe each time.

Measured. Accuracy in the training domains barely moves: validation AUC is 0.997 with everything and 0.992 at tau 0.7. The paper's Figure 4 shows the same flat shelf, then a cliff at the top of the range. My cliff is absent because my score is cruder, and I did not push tau beyond 0.7. On the held-out domains the picture is mixed. medicine stays at 0.99 AUC. open_qa drops from 0.94 AUC at tau 0 to about 0.89 and stays there. wiki_csai is poor throughout, at 0.74 to 0.77.

One comparison I did not expect. A dense probe on the 128 raw max-pooled activations, with no SAE, gets 0.946 AUC in the training domains and 0.77, 0.86 and 0.69 on the three held-out ones. The sparse probe beats it everywhere. Reasoned, with no ablation to back it: the 1,024-wide feature space is simply a bigger, sparser basis for a linear readout, and the paper's own claim is that the SAE's benefit is interpretability, not accuracy. This toy does not argue against that.

The false-alarm side

This is the part I would not skip, because it is the one an auditor needs. The threshold is the logit that gives 5% false alarms on held-out human documents from the training domains (-0.20). At that threshold, on the three held-out domains, measured:

• open_qa: 20% of human answers are flagged, 81% of AI answers are caught.
• medicine: 19% and 99%.
• wiki_csai: 81% of human documents are flagged, and 99% of AI documents are caught.

The median human logit in wiki_csai is +2.36. The median AI logit in open_qa is +1.15. Human encyclopedia prose scores as more AI-like than ChatGPT's answers in another domain. The explanations say why for the top case: the highest-scoring human document, about electronic circuits, gets +4.2 from a newline plus +1.7, +1.2 and +1.0 from a full stop, "such as" and a comma, which is the shape of an expository definition.

Does removing the formatting fix it? Mostly no, which is a result in itself. Dropping the one newline feature (684) moves wiki_csai humans flagged from 81% to 61%. Dropping all 31 features whose top token is only whitespace or punctuation moves it to 64%, and costs accuracy where the shortcut was real: medicine AUC falls from 0.99 to 0.94 and validation AUC from 0.995 to 0.973. So about a quarter of that domain's false alarms were the newline shortcut, and the remainder is the register: formal, definitional, even-paced writing. I did not test that last claim beyond the examples above, so it is reasoned.

The audit did what an audit should. It does not make the detector safe on Wikipedia. It names why it is unsafe there and what removing that cause costs.

Short texts

The model is causal, so the first k tokens of a document give exactly the activations a k-token document would. I pooled and scored prefixes of 8 to 128 tokens with the same head and threshold, over the 2,400 held-out documents. Measured:

Two things happen at once. With few tokens, few features fire and the logit stays near its baseline, below the threshold, so the detector says "human" to almost everything and detects almost nothing. With more tokens, max-pooling keeps the largest value each feature ever reached, so the logit of every class only goes up with length. Human text drifts from a mean logit of -2.35 to -0.81 as it grows from 8 to 128 tokens, closer to the threshold, and the pooled false alarm rate climbs with it. A threshold chosen for 128-token documents is the wrong number for 32-token ones, and the sign of the error flips with length. Reasoned: this is a property of max-pooling and a fixed threshold, so any detector built this way should be audited at the lengths it will see, not only at the length it was calibrated on. The paper reports no length analysis that I found.

Writers who are not native English speakers

I have no data for this and I did not simulate it. The finding that matters comes from outside the paper. Liang and colleagues (GPT detectors are biased against non-native English writers, 2023) tested several detectors on TOEFL essays and reported that, on average, more than half were flagged as AI, because the detectors read low perplexity and a plain, regular register as machine-like (reported by them; I did not re-run it).

What the toy adds is a mechanism, reasoned and not tested. Features of the type I found, " or", " as", " important", full stops after short regular sentences, are exactly what a careful writer working in a second language uses a lot. A detector built on such features would have little to separate that writer from the model, and the wiki_csai result shows what happens when human text is formal and regular. This is also where the paper's design helps most. A decision that comes with "feature X, token Y" can be contested by the person it affects. A black-box score cannot. That does not make the score correct, and it does not substitute for measuring the false positive rate on the actual population of writers, which neither the paper nor I did.

What I would take from it

• Use the head as a reading aid first. The exact-sum property (w * x, no residual) is real and checkable, and it is worth the engineering for the review process alone.
• Audit on text from outside the training distribution before anything else. My detector had 0.995 AUC on its own domains and flagged four in five human texts in one other. The paper's own warning on formatting features, and on dataset shortcuts, is borne out at toy scale.
• A mask by description score removes undescribable features and not wrong ones. My density feature, and the newline feature, would both survive a score based on how well a token explains them. Whether a feature is valid to use is a policy decision, as the paper says.
• Report errors by length and by domain, not in one number. Pooled figures hid a 0.3% to 40% range.

What I did not do

• I did not run detectorscope, Gemma 2 2B, Gemma Scope or any of the paper's tables. Every figure in the reported section is quoted.
• The toy has one layer, one 14M model, ReLU in place of JumpReLU, 1,024 features, one seed, and no language model to label features. The concentration score is mine. Do not read my sweep as a replication of the paper's Figure 4.
• HC3 answers come from an older ChatGPT and carry the formatting artefacts above. The shortcut results are about this dataset and say nothing about the paper's data, which has its own artefacts in the audit tables.
• No non-native writer data, no per-document uncertainty, no confidence intervals. With 400 documents per class and domain, a rate such as 81% carries a binomial interval of roughly four points either way, reasoned from the sample size.

The paper's idea holds up where I could test it: a verdict that is a sum of named parts can be argued with, and in my case it was argued with within an hour. What it does not do, by itself, is make a detector fair or correct. That still needs measuring on the people and the texts it will be used on.