# Cactus Needle 2: the interesting number is what happens after you fine-tune it

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/needle-finetune
> date: 2026-08-23
> tags: edge-inference, tool-calling, quantization, fine-tuning, explainer
[Cactus Needle 2](https://cactuscompute.com/needle) is a 45M-parameter model for tool calling, device use and structured extraction, shipped as a **single 14 MB binary that runs in 28 MB of RAM**. It does 500 tokens/sec decode on a Raspberry Pi 5, 300–700 on a sub-\$200 phone, and it fits on an ESP32-S3.

The base-model benchmarks are respectable and mixed — it trades wins with FunctionGemma 270M, LFM2.5 230M and Apple FM, at 2 bits against their f16. That is the part most coverage stops at, and it is the less interesting half.

The other half is a sentence near the bottom of the page: **fine-tuning lifts accuracy by 21 to 58 points and puts Needle 2 ahead of DeepSeek V4 Flash — a frontier cloud model — on three of the four benchmarks.** Not because the small model got smarter, but because your product exposes a fixed, limited set of tools, and a specialist trained on exactly those beats a generalist that has to be ready for anything.

| | |
|---|---|
| Model | [Cactus Needle 2](https://cactuscompute.com/needle) · 45M params · Apache 2.0 |
| Size | **14 MB** binary · **28 MB** peak session RAM, deterministic |
| Precision | CQ2-bit, trained against Cactus Quants from pretraining through post-training |
| Architecture | Simple Attention Network · 27 layers × 512 wide · Hadamard MLP, engram tables, multi-lane residuals |
| Data | 115B-token proprietary pretraining corpus + 38B post-training |
| Speed | 500+ tok/s decode on a Pi 5 · 400–1,500 on Quest 3S / Vision Pro |
| Fine-tuning | on your own laptop, minutes to hours · `needle finetune` → a `.cact` file |

<Callout type="note">
I wrote about [MimiModel](/articles/mimimodel) recently — an independent C99 reimplementation of Needle's engine for the ESP32-S3, which reached 69.6% on Mobile Actions against the official engine's 69.2% on byte-identical weights. This piece is about the model and the official engine; that one is about what happens when someone rebuilds the decode loop from scratch and audits it.
</Callout>

## Why 45M is the right number

The framing is the strongest part of the release, and it is a scoping argument rather than a modelling one.

> Turning on a light does not need a frontier model. Smartwatches, home assistants and robots already expose their abilities as functions with typed parameters, so the only hard part is mapping a messy sentence onto them: which function, and which arguments. Framed that way, the problem needs no world knowledge and no open-ended prose.

The market argument behind it is worth repeating: there are more than 21 billion IoT devices against roughly 1.5 billion PCs, most phones in emerging markets ship under \$200, and roughly four in five edge devices cost under \$200. "Edge AI" has come to mean Macs and gaming desktops. The actual edge has no GPU, no NPU, and a few dozen megabytes of RAM.

<EnergyBudget />

All of it is an energy argument. On device silicon, moving a byte out of flash or DRAM costs orders of magnitude more than a multiply-accumulate, so the budget that matters is FLOPs per token *and* bytes per token together — and the architecture attacks the first while the engine attacks the second.

The row worth staring at is not Apple FM. It is *transformer at matched params*: a conventional Transformer given Needle's own parameter count still spends 87 MFLOPs per token against Needle's 70, because every parameter it owns has to be exercised through a matmul. The gap is the **engram** — 8M parameters held in hashed n-gram tables and read a few rows per token by gather, contributing capacity at zero arithmetic cost.

The rest of the Simple Attention Network is doing the same kind of work, one block at a time.

<AttentionNetwork />

Cactus publish the whole block as one diagram, and it is worth seeing in their notation rather than only in mine — every stage carries its own update rule, including the two engram sites and the Sinkhorn-normalised routing matrix that mixes the four residual lanes.

<Figure
  src="/articles/needle-finetune/fig1.png"
  alt="The Simple Attention Network block: tokens enter a tied 8192 x 512 embedding, then 27 layers of mHC lane read, engram fusion over hashed n-gram memory, GQA attention with RoPE over a 256-token window plus pinned tool sinks, a Hadamard MLP, and an mHC lane write; the lanes are averaged, normalised, and unembedded, and decoding is constrained by a byte-level grammar into an exact function call."
  caption="The Simple Attention Network, one block. Note the last row: the byte-level grammar is part of the architecture diagram, not an afterthought bolted onto sampling (Cactus-Compute/needle2 model card, architecture figure)."
/>

The bounded window is why microcontrollers are reachable at all: the RAM ceiling is a deterministic 28 MB rather than a curve that grows with conversation length. ESP32-P4 with 32 MB of PSRAM, STM32H7, NXP i.MX RT. The engine compiles single-threaded for bare metal and ships as a static library for Cortex-M4, M7 and M55.

And the quantization is not post-hoc. Needle trains against Cactus Quants from pretraining through post-training — weights, activations and KV cache alike — so the 2-bit model you deploy *is* the model that was trained. Small models break under post-hoc quantization; this sidesteps the question rather than surviving it.

## What the base model actually does

<BenchmarkMatrix />

Cactus state both asymmetries in the comparison upfront, which is more than most releases manage, and they point in opposite directions. The baselines stay at f16 deliberately, because post-hoc 2-bit quantization collapses models never trained for it — that favours the baselines. Needle is trained only for agentic tool calling while every baseline carries chat, prose and world knowledge — that favours Needle. There is no clean way to level both, so they do not try.

Click through the five benchmarks and a pattern emerges that no single table shows. Needle wins by twelve points on Seal-Tools out-of-domain, the test built specifically to hold entire tool domains out of training. It loses by nineteen on BFCL v4, whose enterprise Java and JavaScript SDK surfaces sit completely outside its consumer-device corpus. **The margin tracks distance from the training distribution, not model size** — it beats a 270M model on one benchmark and loses to it on another, at 2 bits against f16 in both cases.

One number I would not skip past: on Mobile Actions, Needle is third of four on strict accuracy at 63.7 and **first by five points on picking the right function at all**, at 98.3% name accuracy against LFM2.5's 93.0%. It knows what to call and loses rows on argument values. That is a specific, addressable failure mode, and it is exactly the one that fine-tuning on your own schemas is best placed to fix.

Cactus's own framing of the base result is this plot, and it is an honest one to lead with — the point being made is about the x-axis, not the y.

<Figure
  src="/articles/needle-finetune/fig2.png"
  alt="Mobile-Actions accuracy against total parameters. Needle 2 at 45M parameters and CQ2-bit sits just under 64, marked in orange at the far left. LFM2.5 230M is highest at about 69, FunctionGemma 270M about 64, and Apple FM at 3B about 57, joined by a curve that declines gently with size."
  caption="The size–quality frontier below mobile class. Needle is not the most accurate point; it is roughly five times smaller than the nearest one and sits at 2 bits against their f16 — and the curve through the baselines is flat enough that the parameter axis is doing the arguing (Cactus-Compute/needle2 model card, frontier figure)."
/>

## The fine-tuning argument

<ScopeArgument />

Here is what the page claims and what it does not. It claims a lift of 21 to 58 points and a result ahead of DeepSeek V4 Flash on three of four benchmarks. It does not break the lift down per benchmark in text — those values live in a chart — so the bands above are the stated range drawn over the measured base scores rather than points I have.

That is a large range and I would want it decomposed. A 21-point lift on Mobile Actions (63.7 → 84.7) and a 58-point lift on DroidCall (17.0 → 75.0) are very different claims about very different things, and the second would be remarkable enough to lead with.

But the mechanism does not need a scaling law to be believable. A general tool-calling model has to allocate capacity across every function signature and argument schema it might ever encounter. Your smart lamp exposes twelve. Forty-five million parameters spread across twelve typed signatures is a great deal of model per signature, and the base-model benchmark pattern above — winning inside its distribution, losing outside it — is that same fact measured from the other side.

The genuinely new part is not that a specialist beats a generalist on a narrow task. That has always been true and has always been impractical, because the specialist cost a training run. **The new part is that the specialist is small enough to train on the laptop you are reading this on**, in minutes to a few hours, and export as a single file.

## What I would want before shipping it

**The fine-tuning result needs a table.** Four benchmarks, base and tuned, with the training set size for each. "21 to 58 points" with a chart is a marketing shape; the same information as five rows is a reproducible claim. It also matters *how much* data each lift needed, because "fine-tune on your own tools" is only a real product story if the answer is hundreds of examples rather than tens of thousands.

**Comparing a fine-tuned specialist to a zero-shot frontier model is not a fair fight, and the page does not say so.** DeepSeek V4 Flash has not seen your tool schemas either; if it had a few hundred of your traces it would presumably also improve. The honest version of the claim is that a fine-tuned 14 MB model beats a *zero-shot* frontier model on your narrow domain — which is still a good and useful claim, and still the right engineering decision for a device with no network.

**DroidCall is a benchmark where everyone fails.** Nobody clears 18% and every model scores zero on the two-call rows. That is worth flagging in either direction: either the benchmark is measuring something the whole size class cannot do, or it is scoring something other than what it means to.

**The 115B-token pretraining corpus is proprietary.** The model is Apache 2.0 and the engine ships as source, which is more open than most. But "pretrained on a proprietary 115B-token corpus and post-trained on 38B tokens with compact reasoning traces and careful dataset distribution design" is the sentence doing the most work in the whole architecture section, and it is the one nobody can check.

## The bit that generalizes

The comparison Cactus draws is the one to keep: LFM2.5-230M was pretrained on 19 trillion tokens, roughly 120× Needle's total, and the two trade wins on function calling.

That is not a statement about anyone's training being wasteful. It is a statement about what tool calling *is*. Most of those 19 trillion tokens bought general language competence, and general language competence is not what maps "turn the lamp down a bit" onto `set_brightness(device_id, level)`. Narrowing the problem until it fits in 45M parameters, and then handing people the means to narrow it further onto their own twelve tools, is a more interesting bet than making the small model bigger.

Whether the fine-tuned numbers survive being written out as a table is the open question. The architecture and the engine, at least, are checkable today.

<Callout type="note">
Cactus have since published [needle-environments](/articles/needle-environments), six hand-curated tool schemas meant to be the starting point for exactly the fine-tune described above. It does not supply the table asked for here — the "90%+" that came with it is an acceptance threshold rather than a score — but the schemas themselves encode eight genuinely useful rules about designing tool surfaces for a constrained decoder, and I pulled them apart in a separate piece.
</Callout>