# Needle Environments: the 0.9 is a gate, not a score
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/needle-environments
> date: 2026-08-26
> tags: edge-inference, tool-calling, evaluation, constrained-decoding, open-source
[needle-environments](https://github.com/cactus-compute/needle-environments) is six Python files. Each one declares five tools for a product surface — a smart home, a music player, a smartwatch — and carries thirty-two test cases at the bottom. It exists so that you can copy one, swap the enum values for your product's, fine-tune [Needle 2](/articles/needle-finetune) on it, and ship a 14 MB tool-caller that runs in 28 MB of RAM.
The release went out with a line worth checking: *watch a 14 MB LLM score 90%+ on held-out production tasks.*
I read all 902 lines, parsed every test case, and pulled apart the engine binary the environments load. Three things about that sentence do not survive. **90%+ is not a score anywhere in the repository — it is an acceptance threshold in an `if` statement.** **Nothing is held out**: the 192 cases ship in the same six files as the schemas they test, and the README tells you to tune the schemas until they pass. And a model that refuses every request in the world passes 12 of 32 while failing **zero** of the nine cases marked critical, because all three critical categories are refusal categories.
That is the audit. The reason to read past it is that the repository is much better than its own pitch. Stripped of the benchmark framing, these six files are a **schema style guide for constrained decoding on a very small model**, and every rule in it is the residue of something that went wrong in someone's evening. The best line in the whole project is a parenthesis in a module docstring explaining why the demo house has a study instead of an office.
| | |
|---|---|
| Repo | [cactus-compute/needle-environments](https://github.com/cactus-compute/needle-environments) · Apache 2.0 · 6 files, 902 lines |
| Model | [Needle 2](https://huggingface.co/Cactus-Compute/needle2) · 45M params · 14 MB binary · Apache 2.0 |
| Engine | `libneedle.so`, fetched at runtime · byte-level grammar compiled from your schemas |
| Suite | **192** cases = 6 × 32, category mix identical in all six: 18/4/3/3/2/2 |
| Critical | 9 per file — `missing`, `negation`, `invalid`, all of which expect **no call** |
| The gate | `passed >= round(0.9 * len(TEST_CASES))` → **29/32** and zero critical failures |
| Percent signs in the repo | **zero** |
## The number that isn't there
Every one of the six files ends the same way. Not similarly — identically: the whole `run_tests` function is byte-for-byte the same in all six (`md5 392062ff…`), and its last line is
```python
return passed >= round(0.9 * len(TEST_CASES)) and not critical_failures
```
`round(0.9 * 32)` is 29. So an environment "passes" at 29 of 32 with no critical failures. That is a bar the author set, not a result anyone measured. Nothing in the repo records a score; `grep -c '%'` across all six files and the README returns zero, and the Needle 2 model card carries no figure for these suites either.
The category design is where it gets uncomfortable. Twelve of the thirty-two cases expect an empty call list, and the nine flagged `critical` are exactly the refusal categories — `missing` (four), `negation` (three), `invalid` (two). So the degenerate baseline is not hypothetical:
```python
def complete(query):
return {"function_calls": []}
```
Twelve of thirty-two. Zero critical failures. It is nowhere near the gate, which is the system working as intended. But it clears the check that was supposed to be the *strict* one, which means the strict check certifies nothing about the capability the model exists for. Severity and capability point in opposite directions here, and only severity is enforced with a hard zero.
Look at panel (a) for a second longer than it deserves. Six rows, and you cannot tell them apart: same eighteen positives, same four `missing`, same three `irrelevant`, same three `negation`, same two `invalid`, same two `parallel`, in the same order. This is a template, filled six times. That is a perfectly reasonable way to build acceptance suites for six product surfaces — it makes them comparable — but it means "192 test cases" is 32 test cases and six vocabularies, and it should not be read as breadth.
## What "held out" would have to mean
The word doing the most work in the claim is *held-out*, and it cannot be true here in any of the three senses it might mean.
It is not held out **from the suite author**: the 192 cases live in the same files as the schemas they test, immediately below them. The README's advice for adapting an environment is to "swap the `Literal` values … and keep the shapes" — so the shapes were arrived at by iterating against these cases.
It is not held out **from the model**, and the repo says so out loud. `wearable.py`'s docstring: *"Workout types use the gerund forms the model was trained on."* The enum strings were chosen to match Needle 2's training distribution. That is good engineering and it is the precise opposite of a held-out evaluation.
And it is not held out **from production**, because none of it came from production. Every query is written in the clean register of a demo — *turn on the kitchen lights*, *set the thermostat to 22 degrees*. Nothing has a false start, a filler word, a speech-to-text mangling, or two requests fused with an "and, uh". If you want to know how a 45M model does on real dictation, this suite is silent about it.
None of this makes the environments bad. It makes them **acceptance tests**, which is what the code says they are — a gate that decides an exit status, run by `sys.exit(0 if run_tests() else 1)`. The mislabelling happened somewhere between the repository and the announcement, and the repository is the honest artefact.
## The rules, which are the actual product
Read the six files as a style guide and they are dense with earned knowledge. I pulled out eight rules, each traceable to a specific line.
Two of them are worth carrying to any codebase, at any model size.
**Delete the optional argument the model likes to guess.** `kitchen_appliance.py` says it plainly: *"Optional settings the model tends to guess (oven modes, cup sizes, default cycles) are deliberately absent."* `set_oven` takes a temperature and nothing else — no mode, no rack, no timer. Four of the thirty-two cases in every file are `missing` (critical), and they are all the same failure: the user under-specified and the model filled in the blank. You can attack that with a better prompt, or you can attack it by not having the field. The second one is a proof rather than a nudge.
**Bounds are not validation, they are the grammar.** `temperature: Annotated[int, needle.Field(ge=50, le=250)]` does not get checked after decoding; it is compiled into the byte-level grammar that constrains every token, so *set the oven to 400* has no representation to emit. The kitchen docstring's phrasing is exactly right — bounds make unsafe requests **unrepresentable**. That is a different guarantee from making them unlikely, and it is available to anyone whose serving stack supports constrained decoding.
There is a nice, quiet reason all six environments have exactly five tools, too. The Needle 2 model card explains that a built-in retrieval head "renders only the top five tools per turn". Five is the page size. A five-tool environment is exactly one page, so tool retrieval can never be the component that failed.
## A room named office
The single best line in the repository is a parenthesis in `smart_home.py`'s module docstring:
> One learned rule: avoid enum values that hide inside likely query words (a room named office poisons an off action, so this home has a study).
`off` occurs inside `office`. The decoder is a native library, so you cannot read the selection rule in the Python — but the shipped `libneedle.so` carries the debug format string that names it:
```
[debug] enum select: start=%d acc='%s' grounded=%zu best=%s
```
`acc` is the accumulated bytes, `best` is the winner, and **`grounded` is a count** — enum candidates are scored by occurrence in the query text, not by a parse. So a home with a room called *office* grounds the `off` action in every sentence that names the room, including *turn on the office lights*. The fix was to rename the room.
I ran that check exhaustively: every enum value in each file against all thirty-two of its queries, substring hit versus word-boundary hit. The vocabularies have been swept almost clean — `smart_home.py` carries twenty enum values across thirty-two queries with **zero** substring-only groundings, which does not happen by accident. Two survive across the whole repo: `brew` inside *brewing* in the kitchen, which lands on the right answer by luck, and `low` inside *sunflower42* in `productivity.py`, where the expected call is a note with no priority at all.
This is the finding worth taking away, and it is not a benchmark result. **On a small model with a byte-level grammar, the names you give your enum values are part of the decoder.** It never comes up with a cloud model, whose tokenizer and context are large enough to swallow the distinction. It comes up immediately at 45M parameters, and nobody documents it, because the people who hit it fix it in their own vocabulary and move on. Here someone wrote it down.
## The suite dies on your weights
One more thing, and it is the kind that only shows up if you actually try to run the workflow the repo is selling.
The environments exist so you can fine-tune on them. `cactus-needle` is explicit that fine-tuning does not update the confidence head, so it warns you and then sets the field to `None`:
```python
if self._weights:
response["confidence"] = None
```
And `run_tests` does this:
```python
if got and response.get("confidence", 0.0) < min_confidence:
got = []
```
`dict.get(key, default)` returns the default only when the key is **absent**. It is present, holding `None`. So the comparison is `None < 0.0`, which raises `TypeError`, on the first test case that produces a call — case 1 of 32 in `smart_home.py`, *turn on the kitchen lights*. The suite that exists to validate your fine-tune cannot be run against a fine-tune.
Both problems are one-line fixes, and neither has been hit. Taken together that says something specific: the workflow this repository exists to support — adapt an environment, fine-tune on it, re-run `run_tests` — has not been executed end to end, by anyone, including whoever wrote it. The dead branch is only visible if you pass the argument the docstring recommends; the `TypeError` is only visible if you point the suite at tuned weights. Doing either is the first thing a user does.
There is a third snag in the same neighbourhood, for anyone actually building on this. The engine holds one set of weights globally and cannot unload them, so `Needle.__init__` raises if you construct a base-model agent after a tuned one — and every environment builds its agent at *import* time. Importing two environments in one process after loading tuned weights does not do what you would expect.
## Eight undocumented knobs
While I had the engine open: `libneedle.so` reads eight `NEEDLE_*` environment variables.
```
NEEDLE_CONFIDENCE NEEDLE_CONF_RESCORE NEEDLE_DEBUG NEEDLE_KV_BITS
NEEDLE_KV_WINDOW NEEDLE_NO_REBASE NEEDLE_STRICT_VALIDATE NEEDLE_THREADS
```
None of them appear in the model card, the README, or the Python package — the two the Python side reads (`NEEDLE_HF_REPO`, `NEEDLE_LIB_PATH`) are a disjoint pair. And every one of the six environment files sets one of them, above the import, before the engine ever loads:
```python
os.environ.setdefault("NEEDLE_STRICT_VALIDATE", "1")
import needle
```
So a "curated environment" is not only a set of schemas. It is a **runtime configuration**, and the one line that supplies it is the one line a reader copying the schema style will not copy. If you follow the README's advice and write your own environment from scratch, you get a differently-configured decoder and no indication that you did.
## The ledger
**What is genuinely valuable.** The eight schema rules, and above all the two that generalise: delete the optional field the model wants to guess, and choose enum strings that cannot hide inside the words your users will say. The enum-grounding observation is a real, specific, reproducible property of constrained decoding on small models, and I have not seen it written down elsewhere. The five-tools-per-environment discipline matched to the engine's five-tool retrieval page is the kind of detail that only comes from having shipped the thing.
**What does not hold.** "90%+ on held-out production tasks" is three claims and all three fail: 90% is a threshold rather than a measurement, nothing is held out in any of the three available senses, and none of it came from production. The 192 cases are 32 cases and six vocabularies. And the strictness that *is* enforced — nine critical cases, hard zero — sits entirely on refusals, so it is satisfied completely by a model that never does anything.
**What I would fix, in an afternoon.** Change `.get("confidence", 0.0)` to `(response.get("confidence") or 0.0)` and the suite runs against tuned weights. Make `min_confidence=0.4` the default so the gate matches the contract. Print the score rather than a boolean, so there is a number to quote. Split the test cases into a `cases/` directory the schemas do not sit in, and add ten dictated-sounding queries per environment — the ones with a false start and a fused request — and the suite starts being about production instead of resembling it.
**What I would watch.** Whether anyone reports a number. Needle 2 is a genuinely interesting artefact — 45M parameters, [2-bit from pretraining onward](/articles/needle-finetune), [running off flash on a \$5 chip](/articles/mimimodel) — and the fine-tuning result it rests on is real and load-bearing. It deserves an evaluation that could have failed. A repository of six acceptance suites, all templates of one another, whose pass condition is a threshold nobody has published a score against, is not that yet. The pieces to make it that are already in the files.