Finding the neuron you asked for: neuron-finder's two screens, and a CPU re-run

mdjsonmcpcopy↗ai

2026-10-30 · 18 min · interpretability · qwen · small-models · transformers

Why read this

Solid· top 85%

Reads neuron-finder's search, recomputes its 46 results, and re-runs it small on a CPU with random-neuron controls: topics found, language needs a lower dose.

• Original analysis
• Explained from first principles

·Interpretability·Runs on a consumer GPU·MIT·Practitioner teardown

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

1 of 3: General advice

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 50 of 100, ranked 546 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 27 October @ChangJonathanC posted: "Claude and I developed a method that can quickly find the neuron you want! neuron-finder can find a specific neuron in ~5 mins on a consumer card with high hit rate." That is a claim about a search, not about a neuron, and it is a testable one. A week of posts had shown that one MLP neuron of Qwen3.5-4B can move a model's persona. The next question is the practical one: if you can name a behaviour, how do you find the neuron that writes it without trying all 294,912?

This article reads the tool's search code, recomputes its headline numbers from the result files it publishes, and re-implements the core idea on a model small enough for a CPU, with random neurons as the control. It then puts the tool next to the two other methods that surfaced in the same week. It is about benign behaviours: languages, topics, styles and how a model describes itself. Our earlier article, neuron steering, explains what pushing a neuron does and why the dose is scaled by the residual norm; I do not repeat that here.

Every claim carries one of three labels. Measured means I ran it or recomputed it from files. Reported means the repository or a post says so and I did not re-run it. Reasoned means arithmetic or inference from the two.

The detector is not the writer

The reason a search is needed at all is a gap our earlier article measured on Qwen3-0.6B. Ranking neurons by how differently they fire on French and English text gave a clean list: the top 40 kept their sign on held-out prompts. But pushing the ten best-ranked neurons toward French changed the language in two of ten, and the neuron ranked third by that contrast was 2,158th out of 86,016 by gradient attribution (measured, in that article). Firing on French and writing French are different properties, and a search that only reads activations finds the first.

neuron-finder is built around that gap. It scores every neuron twice, once as a detector and once as a writer, and then checks the shortlist by actually pushing.

What the repository is

changjonathanc/neuron-finder is MIT licensed, with the copyright line "Claude Opus 5.5 and Jonathan Chang". I read it at commit 38ae049 (29 October 2026). The Python is 519 lines: neuron_finder.py 415, stats.py 79, steer.py 25 (measured, by counting). The default model is Qwen/Qwen3.5-4B, 32 layers of 9,216 MLP neurons, so 294,912 neurons in all. The README says it needs Linux and an NVIDIA GPU (about 17 GB at the default batch), and any OpenAI-compatible API as the judge (reported). It ships 46 targets, a per-neuron baseline for Qwen3.5-4B, and the ranked results of one run per target.

I did not run it, even on a second attempt. The tool is not GPU-only in principle but it is in code, and the gap is small; see "What a second pass found" below. Everything below about its behaviour is read from its code or recomputed from its published results; the re-implementation later is my own.

The search, step by step

1. Pairs. The model answers 24 everyday prompts twice, once under a persona's system prompt ("You are God…") and once without. The model writes both sides, so the pair is on-policy.

2. Read score. For each pair, take every neuron's mean activation at the input of down_proj over the reply's tokens, styled minus plain. Average over pairs and divide by the neuron's natural spread on ordinary chat, which stats.py measures once over 1M tokens of chat. This finds neurons that fire more on the persona.

3. Write score. Take the gradient of $\log p(\text{styled reply}) - \log p(\text{plain reply})$ with respect to each layer's MLP output, summed over every position, because the push is applied at every position. Project it on the neuron's down_proj column and multiply by the size of a dose-1 push, so the score is the first-order change in that log-ratio if the neuron were pushed by one dose:

Here $\Delta_p$ is the log-ratio of pair $p$, $m_{l,t}$ the MLP output of layer $l$ at position $t$, and $r_l$ the residual stream leaving layer $l$. A second write list divides $w$ by the neuron's effect on ordinary chat, so neurons that change all text do not take the top places (all read from code). In the author's 46 runs the winner came from the read list alone 19 times, from the write lists alone 23 times and from both 4 times (measured, from the published files; the README states the same counts).

4. Shortlist. The top 30 of each list, plus the top 2 of each list in every layer from 15% to 95% of the depth, plus 16 random neurons as a null. The comment in the code says why: late layers have larger scores and would crowd out the early, topical layers. The median run has 165.5 distinct candidates (measured).

5. Steer and judge. Each candidate is pushed at three doses (0.8, 1.5, 2.5) with the sign its screen gave, using the recipe from the earlier article: $c = \text{dose}\times\operatorname{med}\lVert r_l\rVert / \lVert W_{\text{down}}[:,n]\rVert$, added to the neuron's activation at every position. Every setting gets 8 sampled replies at temperature 0.8 and one call to an LLM judge asking whether the trait is clearly present and the text is not gibberish. Looping, very short or mostly non-ASCII replies count as misses without a judge call.

6. Confirm. The five best settings get 32 fresh replies under a new seed. The winner is found if at least 3 in 8 of its replies are hits and its rate beats the 95th percentile of the random neurons, partial if it only beats them, not found otherwise.

It searches single neurons only. A pair or a triple would need the tool's batched steering to be pointed at combinations, which the main loop does not do (read from code).

What its 46 runs show

I recomputed the headline numbers from results/*/ranked.json. They match the README.

• 36 targets found, 5 partial, 5 not found (measured from the files; reported by the README).
• Median run: 315 s end to end, minimum 230 s, maximum 714 s. The README says the time is mostly the wait for the judge.
• Confirmed hit rate of the winner: median 0.75.
• The random neurons set a median 95th-percentile bar of 0.00. One target, "robot", has a bar of 1.00: the judge already rates the plain assistant voice as a robot, so no neuron can beat the control, and the tool says so.
• Winners cluster early: 12 of 46 at layers 6 and 7. The signed dose of the winner is negative for 27 of 46, and the dose is 0.8 for 22, 1.5 for 21 and 2.5 for only 3.
• Two targets, anime and cartoon mascot, are won by the same neuron, layer 13 neuron 4810.

The README also lists independent checks (reported): its alien neuron, layer 22 neuron 2742, is the one the neuron-explorer reports; its cookie neuron, layer 24 neuron 3102, is the one in a post that combined it with a cartoon-mascot neuron. The public source of the Neuron Studio Space lists the same cookie neuron in one of its presets (read from the page, not run).

Two misses are worth reading. "Golden Gate" is not found: no candidate gets a hit in 32 confirm replies. The bridge neuron, layer 6 neuron 7448, gets 19 of 32 for the broader "bridge" target but names the Golden Gate in only 2 of 32 replies. And a target can fail because the tool cannot tell, not because there is nothing there: the README says that for the targets it misses, "we cannot tell if the tool missed the neuron or if the model has no single neuron for it."

A CPU re-run on Qwen3-0.6B

To see what survives scaling down, I wrote my own version of the search from the code above and ran it on Qwen3-0.6B (28 layers of 3,072 neurons, so 86,016) on a shared 4-vCPU cloud VM, CPU only, 2 threads, float32. The script is finder_toy.py, 280 lines, and it does not run or copy the tool's code. What changed:

• 12 prompt pairs instead of 24, and a baseline spread taken from the plain replies (about 500 tokens) instead of a million tokens of chat.
• Two score lists, read and write; no third list normalised by the neuron's effect on ordinary text.
• Layers 5 to 24 (20% to 90% of the depth), the top 8 of each list plus the top 1 per layer per list, so 43 to 46 distinct candidates, and 6 random neurons as the null.
• Two doses instead of three, 8 sampled replies of 16 tokens per setting, and the top 3 settings confirmed on 32 fresh replies. The verdict rule is the tool's.
• A keyword rule in place of the LLM judge. A reply is a hit if it is not degenerate (the same kinds of check as the tool's: too short, looping, mostly non-ASCII) and contains at least 3 French function words, or at least one astronomy word, or at least one cooking word. That is a proxy, and the French one is weak: a string of "de la que" counts.

I ran three benign targets: French, space and cooking. Because the tool's own README says a new model family can need different doses, I also re-ran French at doses 0.3 and 0.6, which is where our earlier article found its working window on this model.

All measured. Each count is one seed, so read them as a demonstration, not a rate.

What I take from it:

• Topics are easy to find; this is the part that worked. The space winner turns "Who are you?" into an encyclopedia page ("### Who Are You? Our Solar System … Neptune is a planet"), and the cooking winner into a recipe ("# Step 1: Prepare the ingredients"). Both beat their random-neuron bar by a wide margin. A single neuron in the lower third of the network writes a whole topic.
• Language is harder, and the proxy hides it. At the default doses the French winner was partial, and its replies are Romance-flavoured fragments rather than French. At 0.3 the tool found a neuron whose replies drift between French, German and English ("Who sind Sie?"). The keyword rule cannot tell either from French, so the hit rate there is an upper bound.
• The dose grid is a real constraint. The same neuron, layer 8 neuron 122, scored 0.125 at +0.8 and 0.41 at +0.3. And the two neurons our earlier article found by contrast ranking were not both recovered. Layer 16 neuron 477 was on the shortlist but scored 0.00 at 0.8 and 1.5, 0.25 at 0.3 and 0.00 at 0.6; layer 17 neuron 649 was not shortlisted at all. A search that screens well and then tests at the wrong dose misses neurons that exist.
• The two screens disagreed with the 4B result. Pooled over my four runs, a read-listed neuron reached the 0.375 bar in 18 of 168 settings (10.7%), a write-listed one in 2 of 172 (1.2%), a neuron both lists chose in 3 of 16 (18.8%) and a random one in 1 of 48 (2.1%). On the tool's own 4B runs the read and write lists did equally well, at 2.6% and 2.5%, with 8.9% for both lists and 1.3% for random. What agrees is that a neuron both screens like is the best bet. Why my write list did so little is a guess (reasoned): its gradient predicts the effect of a small push while these doses are large for a 0.6B model, and I left out the list that divides it by the neuron's effect on ordinary text, which the author added so that late layers do not crowd the top.
• The control earned its keep. In the space run one random neuron scored 3 of 8, so the bar was 0.375 and the winner's 30 of 32 had to clear it. Without a null, a keyword rule that fires on ordinary text would have called a lot of neurons found.

What the screens save

Everything before the steering stage, the pairs, the baseline and both screens over all 86,016 neurons, took about 72 s on a quiet machine (the read screen 11 s, the write screen 24 s). Steering took 335 to 405 s for 784 to 832 sampled replies, about 2.3 replies a second, and the confirm stage 40 s; a run was 448 to 473 s once the model was cached (the first took 854 s on a busier box). Trying every neuron at two doses with 8 replies would be 1,376,256 replies, which at the same rate is 6.8 days, about 1,500 times the replies of the shortlist run (reasoned, from the measured rate).

What a second pass found

On 31 October I tried again, with the site owner's explicit go-ahead to run the repository. The plan was to run the tool's own neuron_finder.py on Qwen3-0.6B on the CPU for three of its own targets (space, food and french; it has no cooking target, food is the nearest) and compare with my re-implementation. That run did not happen. The permission layer on this machine refused to execute the downloaded code, a CPU copy of it, and I did not look for a way around the refusal. So every number above is still from my re-implementation, and nothing in this article is newly measured against the real tool. What I could check without executing it, by reading it again and recounting its published result files (measured, by counting and by parsing the JSON, not by running the search):

• The headline holds. Parsing all 46 results/*/ranked.json gives 36 found, 5 partial, 5 not found, a median runtime of 315.5 s (230 to 714 s) and a median confirmed hit rate of 0.75, as stated above.
• A cloud judge is not required. The judge is one HTTP POST to JUDGE_BASE_URL (default api.openai.com, model gpt-6.1-terra, strict JSON schema). A local server that answers with {"grades":[{"trait_clear":…,"gibberish":…}]} would stand in for it, so a keyword scorer can replace the LLM without editing the search. I wrote such a stand-in and did not get to use it. Earlier I said the tool needs a judge API; it needs an endpoint, not a vendor.
• The GPU is hard-coded, not structural. neuron_finder.py names CUDA on 8 lines (.cuda(), device="cuda", .to("cuda")) and loads in bfloat16. Moving it to a CPU is a text substitution, but the 40-line patch I prepared was never run, so I cannot tell you how fast it would be. My own CPU rate (about 2.3 replies a second at 16 tokens) is not a prediction for it: the tool generates 48 tokens a reply, and its default shortlist, about 165 neurons at 3 doses with 8 replies each, is roughly 4,000 screening replies per target (reasoned).
• Two differences from my toy that I had not stated. The tool screens each setting on only 4 prompts (2 samples each), and it generates 48 new tokens, three times my 16. It also counts keyword hits in its report but does not use them to rank; only the judge decides. Longer replies give a keyword rule more chances to fire, so my hit rates are, if anything, a harsher test than the tool's would be at equal doses (reasoned).

What this changes: nothing in the measured results on this page, and one thing in how I described the tool. What it leaves open is the comparison I wanted: whether the tool's own algorithm, with the same scorer, finds the same space and cooking neurons on this model, and how many forward passes it spends doing so. Those remain untested.

Two other ways the same week found neurons

Gradient attribution, then switch off. @treetowntree and @_a9lim reported that most of "I'm an AI" in Qwen3-8B before chat training rests on 16 neurons. Their figure's caption says the neurons were chosen by gradient attribution of "I am an AI." against "I am a human.", and that "switched off" means held at the neuron's usual level. With 16 of 442,368 neurons held, the share of 200 written replies that say the model is an AI fell from 68% to 20%, then 9% at 64 and 0% at 256, while holding 1,024 random neurons changed nothing; the model's most likely answer to "What kind of being are you?" flipped at 8 neurons (all reported, read from the posted figure). I checked one number: Qwen3-8B has 36 layers of 12,288 MLP neurons, which is 442,368 (measured, from its config). A companion figure reports that a model trained on text from before 1930 says it is an AI in 0% of replies and one trained on today's web in 16% (second post, reported).

This is the write score's cousin: a gradient of a contrast of two target sentences. The differences are the intervention (hold, not add), the selection by top-$N$ rather than a steering sweep, a dose-response in the number of neurons with a matched random control, and a judge that scores questions not used to choose the neurons. I did not try to reproduce it, and I read it as a result about where a model's self-description is stored, not as something to use.

Brute force. A library of persona neurons pushes every one of Qwen3-8B's 442,368 neurons alone, up and then down, asks "Who are you?", and keeps the replies: 884,736 of them, of which 34,220 say the model is "someone or something other than an AI" (reported, from the page). The post calls these 34,220 neurons; the page counts replies, and a neuron can appear in both directions, so the number of distinct neurons is at most that (reasoned).

What this does not show

The README's own caveats apply and I repeat the ones that matter here (reported). One LLM judge decides, and judges can be wrong, so read the samples. High doses make odd text: a hit means the trait is clear, not that the reply is useful. Architecture matters: Qwen has no norm after the MLP, so one neuron can push the residual far, whereas Gemma-2 and 3 have a post_feedforward_layernorm that bounds each layer's MLP output, so a single neuron saturates. And "a neuron that makes the model say something is not necessarily a 'concept neuron' in a deeper sense."

On my side: the model is 0.6B and the keyword score is a proxy for the judge. I ran three targets, one seed. The numbers support the shape of the comparison, not a rate.

Sources

• changjonathanc/neuron-finder, commit 38ae049, README, neuron_finder.py, stats.py, results/*/ranked.json; the announcement.
• treetowntree and a9lim, part 2 and part 1; the persona neuron library announced by @jfonsecarivera.
• The neuron-explorer repository and the Neuron Studio Space, as in our earlier article.
• My run: Qwen3-0.6B, float32, CPU, 2 threads, a keyword rule in place of the judge. Only my own prompts and outputs are shown.