2026-08-23 · 10 min · edge-inference · tool-calling · quantization · fine-tuning · explainer
Cactus Needle 2 is a 45M-parameter model for tool calling, device use and structured extraction, shipped as a single 14 MB binary that runs in 28 MB of RAM. It does 500 tokens/sec decode on a Raspberry Pi 5, 300–700 on a sub-$200 phone, and it fits on an ESP32-S3.
The base-model benchmarks are respectable and mixed — it trades wins with FunctionGemma 270M, LFM2.5 230M and Apple FM, at 2 bits against their f16. That is the part most coverage stops at, and it is the less interesting half.
The other half is a sentence near the bottom of the page: fine-tuning lifts accuracy by 21 to 58 points and puts Needle 2 ahead of DeepSeek V4 Flash — a frontier cloud model — on three of the four benchmarks. Not because the small model got smarter, but because your product exposes a fixed, limited set of tools, and a specialist trained on exactly those beats a generalist that has to be ready for anything.
| Model | Cactus Needle 2 · 45M params · Apache 2.0 |
| Size | 14 MB binary · 28 MB peak session RAM, deterministic |
| Precision | CQ2-bit, trained against Cactus Quants from pretraining through post-training |
| Architecture | Simple Attention Network · 27 layers × 512 wide · Hadamard MLP, engram tables, multi-lane residuals |
| Data | 115B-token proprietary pretraining corpus + 38B post-training |
| Speed | 500+ tok/s decode on a Pi 5 · 400–1,500 on Quest 3S / Vision Pro |
| Fine-tuning | on your own laptop, minutes to hours · needle finetune → a .cact file |
Why 45M is the right number
The framing is the strongest part of the release, and it is a scoping argument rather than a modelling one.
Turning on a light does not need a frontier model. Smartwatches, home assistants and robots already expose their abilities as functions with typed parameters, so the only hard part is mapping a messy sentence onto them: which function, and which arguments. Framed that way, the problem needs no world knowledge and no open-ended prose.
The market argument behind it is worth repeating: there are more than 21 billion IoT devices against roughly 1.5 billion PCs, most phones in emerging markets ship under $200, and roughly four in five edge devices cost under $200. "Edge AI" has come to mean Macs and gaming desktops. The actual edge has no GPU, no NPU, and a few dozen megabytes of RAM.
The row to compare against is the third one, not the last. Anyone can beat a 3B model on arithmetic by being sixty times smaller. Beating a conventional Transformer at the same parameter count by 24% takes an architecture that declines to run some of its own parameters through a matmul — which is what the engram does: eight million parameters living in hashed n-gram tables, read a few rows at a time by gather, contributing capacity at zero arithmetic cost.
And arithmetic is only half the bill. On device silicon, moving a byte out of flash costs orders of magnitude more than a multiply-accumulate, so the engine is built around never rematerializing anything: 2-bit codes expand inside vector registers and fuse into integer dot products, so resident memory stays at blob size and decoding a token reads at most the 14 MB file once. On structural tokens the grammar matcher knows which tokens are legal before the logits exist, so up to 98% of the vocabulary projection is skipped — and skipped entirely on steps whose output is already forced.
All of it is an energy argument. On device silicon, moving a byte out of flash or DRAM costs orders of magnitude more than a multiply-accumulate, so the budget that matters is FLOPs per token and bytes per token together — and the architecture attacks the first while the engine attacks the second.
The row worth staring at is not Apple FM. It is transformer at matched params: a conventional Transformer given Needle's own parameter count still spends 87 MFLOPs per token against Needle's 70, because every parameter it owns has to be exercised through a matmul. The gap is the engram — 8M parameters held in hashed n-gram tables and read a few rows per token by gather, contributing capacity at zero arithmetic cost.
The rest of the Simple Attention Network is doing the same kind of work, one block at a time.
Every block here is a different answer to the same question: how do you add capability without adding bytes read per token? The Hadamard MLP mixes channels with a matrix that has no weights. The engram stores knowledge in tables that are gathered rather than multiplied. The lanes buy routing flexibility with dot products instead of width. The grammar head declines to compute most of the vocabulary projection at all.
The memory system is designed backwards from the same constraint. A 256-token sliding attention window means the KV cache is bounded no matter how long a session runs — and the system prompt and tool declarations are pinned as permanent sinks, so the one thing a tool-calling model must never forget is structurally unable to be evicted. That is what turns a RAM budget into a deterministic 28 MB ceiling rather than a curve.
Cactus publish the whole block as one diagram, and it is worth seeing in their notation rather than only in mine — every stage carries its own update rule, including the two engram sites and the Sinkhorn-normalised routing matrix that mixes the four residual lanes.
The bounded window is why microcontrollers are reachable at all: the RAM ceiling is a deterministic 28 MB rather than a curve that grows with conversation length. ESP32-P4 with 32 MB of PSRAM, STM32H7, NXP i.MX RT. The engine compiles single-threaded for bare metal and ships as a static library for Cortex-M4, M7 and M55.
And the quantization is not post-hoc. Needle trains against Cactus Quants from pretraining through post-training — weights, activations and KV cache alike — so the 2-bit model you deploy is the model that was trained. Small models break under post-hoc quantization; this sidesteps the question rather than surviving it.
What the base model actually does
Click through all five and a shape appears that the individual tables do not show. Needle wins by twelve points where entire tool domains are held out, and loses by nineteen where the tool surface is enterprise Java SDKs. The margin tracks distance from its training distribution, not model size — it beats a 270M model on one benchmark and loses to it on another, at 2 bits against f16 in both.
Which is the whole case for fine-tuning stated as an evaluation result. A generic 45M checkpoint is competitive on the tool vocabulary it was trained for and mediocre outside it. Your product does not have a generic tool vocabulary; it has a fixed, small, specific one — and a model this size is cheap enough to retrain on exactly that.
Cactus state both asymmetries in the comparison upfront, which is more than most releases manage, and they point in opposite directions. The baselines stay at f16 deliberately, because post-hoc 2-bit quantization collapses models never trained for it — that favours the baselines. Needle is trained only for agentic tool calling while every baseline carries chat, prose and world knowledge — that favours Needle. There is no clean way to level both, so they do not try.
Click through the five benchmarks and a pattern emerges that no single table shows. Needle wins by twelve points on Seal-Tools out-of-domain, the test built specifically to hold entire tool domains out of training. It loses by nineteen on BFCL v4, whose enterprise Java and JavaScript SDK surfaces sit completely outside its consumer-device corpus. The margin tracks distance from the training distribution, not model size — it beats a 270M model on one benchmark and loses to it on another, at 2 bits against f16 in both cases.
One number I would not skip past: on Mobile Actions, Needle is third of four on strict accuracy at 63.7 and first by five points on picking the right function at all, at 98.3% name accuracy against LFM2.5's 93.0%. It knows what to call and loses rows on argument values. That is a specific, addressable failure mode, and it is exactly the one that fine-tuning on your own schemas is best placed to fix.
Cactus's own framing of the base result is this plot, and it is an honest one to lead with — the point being made is about the x-axis, not the y.
The fine-tuning argument
The claim on the page is the interesting one and I want to state exactly what it does and does not say. It says fine-tuning lifts accuracy by 21 to 58 points and puts a 45 MB-of-RAM model ahead of DeepSeek V4 Flash on three of four benchmarks. It does not say which benchmark got which lift, and the per-benchmark values live in a chart rather than a table — so the bands above are the stated range drawn over the measured base scores, not points I have.
The mechanism, though, is not mysterious and does not need a scaling law. A general model has to allocate capacity across every function and argument schema it might ever see. Your device exposes twelve. Turning a light on does not need world knowledge or open-ended prose — it needs a mapping from a messy sentence to one of twelve typed signatures, and that is a small problem that happens to be the whole problem. The interesting part is not that a specialist wins; it is that the specialist is now small enough to train on the laptop you are reading this on.
Here is what the page claims and what it does not. It claims a lift of 21 to 58 points and a result ahead of DeepSeek V4 Flash on three of four benchmarks. It does not break the lift down per benchmark in text — those values live in a chart — so the bands above are the stated range drawn over the measured base scores rather than points I have.
That is a large range and I would want it decomposed. A 21-point lift on Mobile Actions (63.7 → 84.7) and a 58-point lift on DroidCall (17.0 → 75.0) are very different claims about very different things, and the second would be remarkable enough to lead with.
But the mechanism does not need a scaling law to be believable. A general tool-calling model has to allocate capacity across every function signature and argument schema it might ever encounter. Your smart lamp exposes twelve. Forty-five million parameters spread across twelve typed signatures is a great deal of model per signature, and the base-model benchmark pattern above — winning inside its distribution, losing outside it — is that same fact measured from the other side.
The genuinely new part is not that a specialist beats a generalist on a narrow task. That has always been true and has always been impractical, because the specialist cost a training run. The new part is that the specialist is small enough to train on the laptop you are reading this on, in minutes to a few hours, and export as a single file.
What I would want before shipping it
The fine-tuning result needs a table. Four benchmarks, base and tuned, with the training set size for each. "21 to 58 points" with a chart is a marketing shape; the same information as five rows is a reproducible claim. It also matters how much data each lift needed, because "fine-tune on your own tools" is only a real product story if the answer is hundreds of examples rather than tens of thousands.
Comparing a fine-tuned specialist to a zero-shot frontier model is not a fair fight, and the page does not say so. DeepSeek V4 Flash has not seen your tool schemas either; if it had a few hundred of your traces it would presumably also improve. The honest version of the claim is that a fine-tuned 14 MB model beats a zero-shot frontier model on your narrow domain — which is still a good and useful claim, and still the right engineering decision for a device with no network.
DroidCall is a benchmark where everyone fails. Nobody clears 18% and every model scores zero on the two-call rows. That is worth flagging in either direction: either the benchmark is measuring something the whole size class cannot do, or it is scoring something other than what it means to.
The 115B-token pretraining corpus is proprietary. The model is Apache 2.0 and the engine ships as source, which is more open than most. But "pretrained on a proprietary 115B-token corpus and post-trained on 38B tokens with compact reasoning traces and careful dataset distribution design" is the sentence doing the most work in the whole architecture section, and it is the one nobody can check.
The bit that generalizes
The comparison Cactus draws is the one to keep: LFM2.5-230M was pretrained on 19 trillion tokens, roughly 120× Needle's total, and the two trade wins on function calling.
That is not a statement about anyone's training being wasteful. It is a statement about what tool calling is. Most of those 19 trillion tokens bought general language competence, and general language competence is not what maps "turn the lamp down a bit" onto set_brightness(device_id, level). Narrowing the problem until it fits in 45M parameters, and then handing people the means to narrow it further onto their own twelve tools, is a more interesting bet than making the small model bigger.
Whether the fine-tuned numbers survive being written out as a table is the open question. The architecture and the engine, at least, are checkable today.