Needle 3: one checkpoint, five models, and a file that weighs more than the page says

Cactus Needle 3 is one 20-layer, 121M-parameter checkpoint trained so that any depth from 2 to 20 layers — sliced out of the same weights — is a deployable model in its own right. That is the real headline, and it is a genuinely different claim from the two prior Needle pieces on this site: Needle 2 shipped one 45M model, and needle-environments audited the schemas you fine-tune it on. Needle 3 ships nineteen models for the storage cost of one.

The sentence Cactus put on its own banner

The tweet-length version circulating is "Needle 3 can match DeepSeek V4 Flash." Read the live page and Cactus is more careful than that themselves: the banner at the top says "Needle 3 fine-tuned on the Cactus Platform passes DeepSeek V4 Flash from 4 layers up," and the body text narrows it further — "4L can match DeepSeek V4 Flash when tuned on downstream tasks for one epoch." That is three qualifiers the shorter claim drops: fine-tuned, not base; from 4 layers, not every layer; and on the benchmark it was tuned for, not in general. Credit where due — Cactus states this plainly on its own page, in a TIP banner nobody could miss. The looser claim is something the announcement acquired on the way out the door, not something the source says.

There's a smaller, sillier inconsistency on the same page. The hero paragraph says the whole model is "a single 8-29 MB binary." Scroll to the intelligence-laddering section and the same page says "Intelligence laddering produces 9 to 29 MB CQ2-bit binaries." Eight or nine — it's a one-word difference, it doesn't change any conclusion, and it's the kind of thing that survives when a number gets typed twice by two different people. I flag it because the bigger size claim below is not that kind of typo.

What "sliceable" means, and why you can't just cut a transformer in half

Training one network so that a prefix of it is also a valid, independently useful network is not a new idea — it's the shape shared by LayerDrop (randomly drop transformer layers during training so any subset survives at inference), early-exit networks like BranchyNet and DeeBERT (attach a classifier at intermediate depths and stop once confidence is high enough), and Matryoshka Representation Learning (train an embedding so its first k dimensions, for every k, are themselves a usable lower-dimensional embedding). Needle 3 applies the same instinct to whole transformer blocks rather than embedding dimensions, and its specific recipe is worth separating from the family.

Why you can't just truncate a normal transformer. Train a 20-layer model the ordinary way and every block's function is defined relative to what the blocks before and after it do — layer 14 can specialize on the assumption that layer 6 already resolved local disambiguation and layer 18 will clean up what it leaves rough. Cut the stack at layer 8 and you don't get a smaller version of the model; you get a model whose blocks were never trained to work without the ones you removed. Blocks become load-bearing for specific other blocks, and naive post-hoc truncation breaks the load path. Needle's own training write-up names this directly: the ladder objective is a regularizer specifically because "no single block can become load-bearing, and the early blocks are pushed to do real work rather than defer it."

How the depths are chosen. A 20-layer Needle is blocks $f_0, \dots, f_{19}$. A depth-$d$ subnetwork keeps a subset $S_d$ of $d$ blocks, run in their original order, with the rest passed through as the identity — literally skipped, not zeroed. Two properties matter: the subsets must nest ($S_2 \subset S_3 \subset \cdots \subset S_{20}$) so one training run serves every depth, and each depth must stay spread through the stack so a shallow subnetwork still sees early and late computation rather than, say, only the first eight layers. Both properties fall out of one rule: start from the two endpoints, then repeatedly split the widest remaining gap at its midpoint. Twenty blocks give one fixed sequence —

0, 19, 9, 14, 4, 6, 11, 16, 2, 7, 12, 17, 1, 3, 5, 8, 10, 13, 15, 18

— and subnetwork $d$ is that sequence's first $d$ entries. $S_2 = \{0, 19\}$: every depth keeps the first and last block, which is why the architecture diagram below labels them explicitly. $S_8$ is $\{0, 4, 6, 9, 11, 14, 16, 19\}$, and so on up to the full stack.

How one run trains all nineteen depths. The naive approach — one loss term per depth, every step — is $19\times$ the forward passes. Needle instead samples one path per optimizer step. With probability 0.8 it's an ordinary full-depth language-modeling step. Otherwise, a depth $d$ is drawn uniformly from $\{2, \dots, 19\}$, only the blocks in $S_d$ run, and the loss is

— cross-entropy at that depth, plus a KL term pulling $p_d$ toward the frozen full-model prediction $p_L$ (stop-gradient, so no extra backward pass beyond the teacher's own). That's self-distillation, and it's doing real work: cross-entropy alone gives a 2-layer step a fraction of a percent of the gradient budget a 20-layer step gets, and the teacher signal is what keeps the shallow rungs from being trained on noise. Because the sampling is unbiased, the expected objective over many steps is exactly the mixture a joint 19-loss-term objective would compute — at the cost of one forward pass.

The training report's own control experiment is the cleanest evidence that this changes what the blocks learn, not just what you can extract afterward: a ladder-trained network and an identically-configured non-ladder control, trained for the same number of steps, reached full-depth validation losses of 1.958 versus 2.037 — the ladder-trained model is better at being a 20-layer model, not just better at also being smaller ones. And its 16-layer exit matched the control's full 18 layers. Asking every prefix to be useful is a regularizer that redistributes work toward the early blocks; that redistribution is precisely what naive truncation of an ordinarily-trained model never gets, because nothing during its training ever asked block 8 to be able to stand on its own.

Where it actually degrades. Cactus's own base-model chart (confidence-gated, the config you'd actually ship) shows four of the five depths: 11.7 → 36.8 → 80.7 → 86.0 on Mobile Actions as depth goes 4 → 8 → 16 → 20 layers. That is not a smooth curve — the jump from 8 to 16 layers is +43.9 points, more than triple the +5.3 gained going from 16 to 20. DroidCall degrades more gently: 21.0 → 36.5 → 40.0 → 47.0. Different benchmarks, different knees; the ladder doesn't buy uniform graceful degradation, it buys a curve you get to choose a point on.

And the 2-layer slice is missing from that chart entirely, "for an honest reason" in Cactus's own words: at 25M parameters, its confidence head withholds almost every call, so its gated score on both benchmarks is effectively zero. Forced to answer regardless — no confidence gate — it manages 0.9% on Mobile Actions and 20.5% on DroidCall. That's the direct answer to whether the smallest slice is "genuinely useful or just technically loadable": out of the box, it's the latter. It loads, it runs, and it does almost nothing. Fine-tuned on one product's tools, the same 25M slice reaches 66.7 on Mobile Actions and 56.5 on DroidCall — usable, though still short of DeepSeek's 60.5 on DroidCall specifically. Two layers of a generalist is a curiosity; two layers tuned on your twelve tools is a product, and that gap is the whole argument for shipping the ladder rather than just the 121M model.

expand

Simple Attention Networks: deleting the feed-forward layer and measuring what's lost

The block diagram names a Monarch Hadamard FFN, which sounds like Needle still has a conventional feed-forward sublayer — it's cheaper, not absent. The architecture's real premise is a separate, checkable research result Cactus published alongside the ladder: A Controlled Study of Attention-Only Transformers. A Simple Attention Network (SAN) is a decoder transformer with the feed-forward sublayer deleted from every block — nothing else changes: pre-norm, GQA, RoPE, a scalar residual gate, tied embeddings. For a fixed attention pattern, a SAN layer is linear in its inputs, and each head's output at a position sits inside the convex hull of the value vectors it attends to. A SAN layer can only select and transport content already present in context — it cannot synthesize a representation the context doesn't support.

Cactus ran the "delete the FFN and measure what's lost" experiment three ways, because deleting the FFN perturbs parameters, compute, and nonlinear depth all at once, and each control isolates one:

Delete the FFN with nothing else changed and you lose 0.47 nats — you deleted 72% of the model's parameters, so of course it's worse. Hand that parameter budget back to attention as depth instead, and the gap collapses to 0.006 nats, 0.27% of loss, reproducing to one part in ten thousand across seed pairs, on a 105B-token training run. The FFN's parameters matter enormously; its specific functional form, once attention gets an equal budget, mostly doesn't.

The 0.006 nats isn't spread evenly, either. Every training document has a query, a reasoning trace, and an answer region; decomposed that way, the attention-only model is ahead on traces and every answer region by 105B tokens, and behind only on query tokens — the positions with the least context to route from, where only knowledge baked into weights (rather than present in the prompt) can help. That's the one place a purely content-transporting layer structurally can't compete, and it's a small slice of total loss (8% of tokens, a per-token gap five times the aggregate) precisely because most real text is mostly routable.

This is exactly the deficit Needle's engram — hashed n-gram memory, read by gather rather than matmul — exists to buy back. Cactus's own framing: "Needle 2 was a Simple Attention Network with an engram memory bolted on for the facts that are not in context; Needle 3 keeps the attention-only stack and adds a ladder." And it's exactly why the architecture fits tool calling specifically rather than being a general free lunch: the function name, the argument values, and the schema are all already in the prompt. Tool calling and structured extraction are routing problems almost by definition — select the right span, transport it into a call — which is the one regime a context-grounded, FFN-free model was never expected to be worse at.

CQ2-bit, and a file that doesn't weigh what the page says

Cactus Quants (CQ) is not Q2_K or a ternary scheme, though it lands in similar bit-width territory. The mechanism: split a tensor's input dimension into groups of 128 weights, and multiply each group by a normalized Walsh-Hadamard matrix — an orthogonal rotation. Whatever a group's original weight distribution looked like, after that rotation its coordinates resemble a sample from a Gaussian, because a Hadamard transform spreads any one weight's contribution across all 128 positions in the group. That's the trick: instead of calibrating a codebook per group to whatever distribution that group happens to have (which is what schemes like Q2_K's per-block scale-and-minimum do), CQ needs only one fixed Lloyd-Max codebook, computed once, that works for every rotated group in the model. Activations get the same rotation at inference, which cancels against the weight-side rotation because the Hadamard matrix is orthogonal — so the matmul result is unchanged; only the storage format is.

The storage cost per weight is $b + \tfrac{1}{8}$ bits: $b$ bits per index, plus a shared 16-bit norm amortized over the 128-weight group ($16/128 = 0.125$). For CQ2, $b=2$, so 2.125 bits/weight — not the 1.58-bit-per-weight, ternary $\{-1, 0, 1\}$ representation that schemes like BitNet use; CQ trades a slightly larger nominal budget for the codebook-sharing trick, and — per the earlier Needle 2 piece — it's trained against these quantization levels from pretraining onward rather than applied post-hoc, so the deployed 2-bit model is the one that was actually optimized, not a lossy compression of a different one.

That last part is checkable without downloading the model. A safetensors file keeps a JSON header at its front, and Needle 3's carries a __metadata__ block describing the run that produced it. Two range requests — eight bytes for the header length, then the header — are enough to read the entire quantization-aware schedule out of a 242 MB file.

The anneal is exactly what the page describes: start at 4 bits, end at 2, with the transition spread across the first half of training and a final phase that pins the embedding and mHC tensors at 4 bits — the same embedding=4,mhc=4,default=2 scheme config.json ships with. Publishing the seed and the dataset revision alongside it is more than most releases bother with. The one thing the metadata contradicts is small and probably an oversight: 31.15% of the training mixture is speech data, and config.json lists the model's modalities as text only.

Here's where the checkable claim breaks. config.json states the model has 121,021,910 parameters. At a flat CQ2-bit, that's $121{,}021{,}910 \times 2.125 \div 8 = 32{,}146{,}445$ bytes — 32.15 MB, already past the page's 29 MB ceiling using nothing but Cactus's own published bit-cost formula and parameter count. And that's the generous estimate: config.json's own quantization.scheme field reads "embedding=4,mhc=4,default=2" — the tied embedding table and the multi-lane hyper-connection gates are stored at 4 bits, not the 2-bit default, because they're more precision-sensitive. So I downloaded the actual file.

Cactus-Compute/needle3's needle3.cact on Hugging Face resolves, via its real CDN redirect (not a git-lfs pointer — the HTTP response after the redirect carries the real bytes), to a Content-Length of 35,335,380 bytes: 35.34 MB, or 33.70 MiB. That's roughly 22% over the page's stated ceiling in decimal-MB terms, and it's consistent with the config's own mixed-precision scheme rather than contradicting it — the embedding and mHC tensors at 4 bits explain why the real file lands above even the honest all-2-bit estimate. As one more cross-check: the companion checkpoints/needle3.safetensors (the fp16 checkpoint you fine-tune from) is 242,047,978 bytes, and that number resolves exactly rather than approximately. Strip the 8-byte length prefix and the 8,248-byte JSON header and 242,039,722 bytes of tensor payload remain; sum the 64 tensors that header describes and you get 242,039,722 — the same figure to the byte, every one of them F16. Those tensors hold 121,019,861 parameters, which is 2,049 fewer than config.json's total_parameters. An overcount of 0.0017% is far too small to move anything in this piece; I record it because I measured it. The 121M figure is solid even where the file-size claim isn't.

None of this touches the smaller slices directly — Cactus doesn't ship pre-sliced files, you generate them yourself with needle build --layers N, so there's no 4-layer or 8-layer .cact sitting on the Hub to download and check the same way. The published "8 MB at 4 layers to 29 MB at 20" range may undercount those too, by a similar mechanism, but I didn't measure them and won't claim a number I didn't check.

"Beats DeepSeek V4 Flash" — on what, exactly

Cactus's own benchmark suite names six evaluations, split into two families reported as means: Tools (Mobile Actions, 961 rows, phone commands to Android intents, exact call; DroidCall, 200 rows, some requiring two calls in order, exact calls in order; BFCL v4, 3,641 rows, AST match, no call permitted on irrelevant requests) and Extractions (DSTC8, 1,813 dialogue turns, field F1; SNIPS gold, 700 rows, schema given; SNIPS 7-way, 700 rows, pick the schema among seven first). Base checkpoints run through the shipped CQ2-bit binary with the confidence gate on; baselines run at f16 under vLLM; DeepSeek V4 Flash runs through its cloud API. That asymmetry favors the baselines, not Needle — post-hoc 2-bit quantization would collapse models never trained for it, so keeping baselines at full precision is, if anything, generous to them. It's the same asymmetry Needle 2's benchmarks disclosed, and Needle 3 states it again.

expand

On this specific chart, "beats models 10x its size on mobile tool calls" holds up exactly: Needle3-20L scores 86.0 against LFM2.5 1.2B's 82.4 — a real win, at roughly a tenth the parameters and two bits against sixteen. The fuller claim narrows to something more interesting and more defensible: matching a frontier model requires fine-tuning, and only shows up from four layers up.

expand

That figure is genuinely the strongest result in the release, and it holds up exactly as stated: on DroidCall, fine-tuning lifts every depth 18 to 36 points, and 4 layers (62.5) through 20 layers (70.0) all clear DeepSeek's 60.5 — the "from 4 layers up" language is precise, not rounded.

It does not hold on Mobile Actions, which the same Figure 3 shows as its second panel. There, fine-tuning lifts the smaller slices enormously — the 2-layer model goes from 0.9 to 66.7, a 65.8-point swing — but no tuned subnetwork reaches DeepSeek's 88.4, including the full 20-layer model, which actually scores a hair below its own untuned base (84.5 tuned vs. 86.7 base, both forced-call). Fine-tuning on a benchmark can very slightly hurt a slice that was already close to that benchmark's ceiling; more training data doesn't help what's already saturated, and here it cost 2.2 points. The banner claim is real, sourced, and true — on DroidCall. The same figure, one click away on the same page, shows it isn't true on Mobile Actions, and nothing on the page says so.

The extraction side gets less attention and deserves a sentence: on DSTC8 specifically, the base Needle3-20L (40.7) is beaten outright by LFM2.5 230M (53.0) — a smaller model — while on SNIPS gold and SNIPS 7-way it does land in the "matches 2-3x bigger models" range Cactus claims. "Extractions" is reported as a mean of three suites; the mean can be true while one of the three components isn't, and DSTC8 is the one that isn't.

Why any of this is plausible at all is a point this site has now made three separate times, about three unrelated products. Parallel Constrained Decoding works because restricting a full vocabulary distribution to a handful of legal JSON values is arithmetic you already have, once the candidate set is small. The tiny browser models — a 41,321-parameter syntax highlighter, a 38,745-parameter date parser — reach 83–99% on their own narrow tasks at four to five orders of magnitude fewer parameters than a general chat model, because the output space is a handful of classes or a short structured value, not open-ended prose. Needle's whole scoping argument, going back to Needle 2, is the same shape at a different point on the curve: a fixed, small tool vocabulary needs no world knowledge, so a 121M model — or a 29M one, fine-tuned — can compete with a model a hundred times larger precisely because most of that larger model's capacity is spent on a problem Needle was never asked to solve. Three unrelated codebases, three unrelated size classes, the same underlying reason.

"Empty list, not a guess" — what's actually doing the refusing

This is a real design commitment worth taking seriously, and it turns out to be enforced by three different mechanisms doing three different jobs, not one learned "abstain" class.

The grammar itself. "A request no declared tool can serve returns the empty call [], which is the whole contract for refusal" — Cactus's own confidence guide is explicit that off-topic detection needs no score at all. The byte-level grammar compiled from your tool schemas apparently permits an empty function_calls array as one of its valid completions, so "nothing matches" isn't a confidence judgment, it's a parse that was always available.

The confidence floor. Every response separately carries a confidence — the minimum of a calibrated head scored over the finished call, and the raw decode probability of the call's own tokens — and the engine applies a hard floor of 0.1 beneath which a call is redirected into suppressed_calls rather than executed. This is the number for "is this specific call, which the model was willing to produce, trustworthy," not for "was there a tool here at all."

Rule-based grounding gates, sitting on top of both and independent of the score: a call is withheld outright if the request negates the tool's verb, quotes a command attributed to someone else, fills a required enum with a value it never named, fills a required slot with a control word, states no quantity for a required number with no default, or reverses a source/destination pair. This is close kin to what needle-environments found from the other direction — every one of that repo's nine critical test cases turned out to be a refusal category (missing, negation, invalid), and the degenerate always-refuse model passed all of them. The grounding gates are the shipped, general-purpose version of exactly the guard those hand-written test cases were checking for.

One exception that swallows all three: a tool declared with triggers — regex patterns matched against the raw request — always produces a call for a matching request, skipping the confidence floor and the guess gates. Only the contradiction-style grounding checks still apply. That's a deliberate escape hatch for intents that must always reach a tool, and it means the refusal contract is not uniform across your own tool surface — it depends on whether you opted a given tool into pattern-matched routing.

from typing import Literal
 
@needle.tool(triggers=[r"\b(turn|switch|power|flip)\b.*\b(on|off)\b", r"\btoggle\b"])
def control_device(device: str, action: Literal["on", "off", "toggle"]):
    "Switch or toggle any named smart-home device."
    return {"device": device, "action": action}
 
agent = needle.Needle(tools=[control_device, get_weather])
agent.complete("toggle the garage door")
# function_calls [{"name": "control_device", "arguments": {"device": "garage door", "action": "toggle"}}]

What isn't published: a false-positive rate. The confidence guide's own worked illustration — a scatter of "right" and "wrong" calls against a 0.70 threshold, showing 19 acted on with 1 wrong and 18 sent to confirmation with 6 actually right — is captioned "illustrative calls." It is a diagram of what a calibrated score should look like, not a reliability curve measured on a held-out set. BFCL v4's "no call on irrelevant" scoring is the closest thing to a real measurement, and it's folded into one aggregate accuracy number (50.2 for the base 20-layer model, below several smaller baselines — the same enterprise-API blind spot Needle 2's BFCL score had) rather than broken out as its own false-positive rate. The guide adds one more honest caveat in passing: "correct Spanish calls have been measured at 0.0," which is exactly the kind of number that should make you distrust any fixed threshold on a deployment that isn't English-only. Without a published reliability diagram or an isolated false-positive rate on genuinely out-of-scope requests, "calibrated" is a design intent I can confirm the mechanism for and not a number I can confirm the value of.

There's a concrete reason to want that number before trusting the mechanism completely. GitHub issue #117 against Needle 2 (engine 2.0.4, cactus-needle 2.0.13, not Needle 3 — I want to be precise about which generation this is) reported a LoRA-tuned .cact where JAX greedy decoding on the exported weights reproduced the trained refusal (<tool_call>[]</tool_call>) on 8 held-out rows, while the native engine, given bit-identical weights and near-identical first-token logits (mean absolute difference 0.04 on the base model, 0.54 tuned), emitted an actual tool call with plausible-sounding reasoning on 6 of those 8 — "what's the capital of France?" triggered a real search_site call in the engine and a correct refusal in JAX. All 393 prefix tokens matched between the two paths. The issue is closed on GitHub with no visible comment thread, so I can't tell you whether it was root-caused or went stale — only that it exists, that it is exactly the failure mode "empty list, not a guess" is supposed to prevent, and that it showed up specifically after fine-tuning, which is the workflow this whole release is built to encourage.

The ledger

Real and checkable. The ladder mechanism — bisected block order, stochastic-depth sampling, self-distillation, all traced to the actual training report rather than taken on faith. The Simple Attention Network result underneath it, an honest ablation with three separate controls and a result (0.006 nats at matched parameters) that's small enough to be believable and large enough on query tokens to explain exactly why engrams exist. The DroidCall fine-tuning result: every tuned depth from 4 layers up genuinely clears DeepSeek V4 Flash, verified against Cactus's own figure.

Narrower than advertised. "Passes DeepSeek V4 Flash" is true on DroidCall and false on Mobile Actions, in the very figure that's cited for the claim. "Beats models 10x its size" checks out cleanly on Mobile Actions and doesn't extend to BFCL v4, where several smaller baselines beat the base 121M model — the same enterprise-surface gap Needle 2 had. The 2-layer slice is technically loadable and functionally inert until you fine-tune it, which "any depth from 2 to 20" doesn't advertise on its own.

Doesn't hold up. The 20-layer needle3.cact on Hugging Face is 35.34 MB, not the 29 MB the page states twice (once as "8-29 MB," once as "9-29 MB" — they can't even agree with themselves on the floor). No per-slice Raspberry Pi throughput table exists anywhere I could find, so "up to 4k tokens/sec" is an unattributed endpoint of a five-slice range, not a number you can assign to a specific depth. And no reliability diagram or false-positive rate backs "calibrated confidence" — the one figure that gestures at it says "illustrative" in its own caption.

None of that makes the ladder fake. The mechanism is real, published in enough detail to check, and the DroidCall result is a genuine frontier-matching claim from a 29M-parameter model. It's the file size and the throughput number — the two claims a reader could check without reading a single equation — that don't survive being checked.