Needle Environments: the 0.9 is a gate, not a score

needle-environments is six Python files. Each one declares five tools for a product surface — a smart home, a music player, a smartwatch — and carries thirty-two test cases at the bottom. It exists so that you can copy one, swap the enum values for your product's, fine-tune Needle 2 on it, and ship a 14 MB tool-caller that runs in 28 MB of RAM.

The release went out with a line worth checking: watch a 14 MB LLM score 90%+ on held-out production tasks.

I read all 902 lines, parsed every test case, and pulled apart the engine binary the environments load. Three things about that sentence do not survive. 90%+ is not a score anywhere in the repository — it is an acceptance threshold in an if statement. Nothing is held out: the 192 cases ship in the same six files as the schemas they test, and the README tells you to tune the schemas until they pass. And a model that refuses every request in the world passes 12 of 32 while failing zero of the nine cases marked critical, because all three critical categories are refusal categories.

That is the audit. The reason to read past it is that the repository is much better than its own pitch. Stripped of the benchmark framing, these six files are a schema style guide for constrained decoding on a very small model, and every rule in it is the residue of something that went wrong in someone's evening. The best line in the whole project is a parenthesis in a module docstring explaining why the demo house has a study instead of an office.

The number that isn't there

Every one of the six files ends the same way. Not similarly — identically: the whole run_tests function is byte-for-byte the same in all six (md5 392062ff…), and its last line is

return passed >= round(0.9 * len(TEST_CASES)) and not critical_failures

round(0.9 * 32) is 29. So an environment "passes" at 29 of 32 with no critical failures. That is a bar the author set, not a result anyone measured. Nothing in the repo records a score; grep -c '%' across all six files and the README returns zero, and the Needle 2 model card carries no figure for these suites either.

The category design is where it gets uncomfortable. Twelve of the thirty-two cases expect an empty call list, and the nine flagged critical are exactly the refusal categories — missing (four), negation (three), invalid (two). So the degenerate baseline is not hypothetical:

def complete(query):
    return {"function_calls": []}

Twelve of thirty-two. Zero critical failures. It is nowhere near the gate, which is the system working as intended. But it clears the check that was supposed to be the strict one, which means the strict check certifies nothing about the capability the model exists for. Severity and capability point in opposite directions here, and only severity is enforced with a hard zero.

expand

Look at panel (a) for a second longer than it deserves. Six rows, and you cannot tell them apart: same eighteen positives, same four missing, same three irrelevant, same three negation, same two invalid, same two parallel, in the same order. This is a template, filled six times. That is a perfectly reasonable way to build acceptance suites for six product surfaces — it makes them comparable — but it means "192 test cases" is 32 test cases and six vocabularies, and it should not be read as breadth.

What "held out" would have to mean

The word doing the most work in the claim is held-out, and it cannot be true here in any of the three senses it might mean.

It is not held out from the suite author: the 192 cases live in the same files as the schemas they test, immediately below them. The README's advice for adapting an environment is to "swap the Literal values … and keep the shapes" — so the shapes were arrived at by iterating against these cases.

It is not held out from the model, and the repo says so out loud. wearable.py's docstring: "Workout types use the gerund forms the model was trained on." The enum strings were chosen to match Needle 2's training distribution. That is good engineering and it is the precise opposite of a held-out evaluation.

And it is not held out from production, because none of it came from production. Every query is written in the clean register of a demo — turn on the kitchen lights, set the thermostat to 22 degrees. Nothing has a false start, a filler word, a speech-to-text mangling, or two requests fused with an "and, uh". If you want to know how a 45M model does on real dictation, this suite is silent about it.

The rules, which are the actual product

Read the six files as a style guide and they are dense with earned knowledge. I pulled out eight rules, each traceable to a specific line.

Room = Literal["kitchen", "living_room", "bedroom", "study"]
#                                                   ^^^^^^^ never "office"

Two of them are worth carrying to any codebase, at any model size.

Delete the optional argument the model likes to guess. kitchen_appliance.py says it plainly: "Optional settings the model tends to guess (oven modes, cup sizes, default cycles) are deliberately absent." set_oven takes a temperature and nothing else — no mode, no rack, no timer. Four of the thirty-two cases in every file are missing (critical), and they are all the same failure: the user under-specified and the model filled in the blank. You can attack that with a better prompt, or you can attack it by not having the field. The second one is a proof rather than a nudge.

Bounds are not validation, they are the grammar. temperature: Annotated[int, needle.Field(ge=50, le=250)] does not get checked after decoding; it is compiled into the byte-level grammar that constrains every token, so set the oven to 400 has no representation to emit. The kitchen docstring's phrasing is exactly right — bounds make unsafe requests unrepresentable. That is a different guarantee from making them unlikely, and it is available to anyone whose serving stack supports constrained decoding.

There is a nice, quiet reason all six environments have exactly five tools, too. The Needle 2 model card explains that a built-in retrieval head "renders only the top five tools per turn". Five is the page size. A five-tool environment is exactly one page, so tool retrieval can never be the component that failed.

A room named office

The single best line in the repository is a parenthesis in smart_home.py's module docstring:

One learned rule: avoid enum values that hide inside likely query words (a room named office poisons an off action, so this home has a study).

off occurs inside office. The decoder is a native library, so you cannot read the selection rule in the Python — but the shipped libneedle.so carries the debug format string that names it:

[debug] enum select: start=%d acc='%s' grounded=%zu best=%s

acc is the accumulated bytes, best is the winner, and grounded is a count — enum candidates are scored by occurrence in the query text, not by a parse. So a home with a room called office grounds the off action in every sentence that names the room, including turn on the office lights. The fix was to rename the room.

I ran that check exhaustively: every enum value in each file against all thirty-two of its queries, substring hit versus word-boundary hit. The vocabularies have been swept almost clean — smart_home.py carries twenty enum values across thirty-two queries with zero substring-only groundings, which does not happen by accident. Two survive across the whole repo: brew inside brewing in the kitchen, which lands on the right answer by luck, and low inside sunflower42 in productivity.py, where the expected call is a note with no priority at all.

This is the finding worth taking away, and it is not a benchmark result. On a small model with a byte-level grammar, the names you give your enum values are part of the decoder. It never comes up with a cloud model, whose tokenizer and context are large enough to swallow the distinction. It comes up immediately at 45M parameters, and nobody documents it, because the people who hit it fix it in their own vocabulary and move on. Here someone wrote it down.

The suite dies on your weights

One more thing, and it is the kind that only shows up if you actually try to run the workflow the repo is selling.

The environments exist so you can fine-tune on them. cactus-needle is explicit that fine-tuning does not update the confidence head, so it warns you and then sets the field to None:

if self._weights:
    response["confidence"] = None

And run_tests does this:

if got and response.get("confidence", 0.0) < min_confidence:
    got = []

dict.get(key, default) returns the default only when the key is absent. It is present, holding None. So the comparison is None < 0.0, which raises TypeError, on the first test case that produces a call — case 1 of 32 in smart_home.py, turn on the kitchen lights. The suite that exists to validate your fine-tune cannot be run against a fine-tune.

response = agent.complete(case["query"])
got      = response.get("function_calls") or []          # ["control_lights", ...]
if got and response.get("confidence", 0.0) < min_confidence:
    got = []

# response["confidence"] is None  — needle sets it to None for tuned weights, and .get()
#                              returns its default only when the key is ABSENT

Traceback (most recent call last):
  File "smart_home.py", line 160, in run_tests
    if got and response.get("confidence", 0.0) < min_confidence:
                                                 ^
TypeError: '<' not supported between instances of 'NoneType' and 'float'
# case 1 of 32 — "turn on the kitchen lights". The first one that returns a call.

Both problems are one-line fixes, and neither has been hit. Taken together that says something specific: the workflow this repository exists to support — adapt an environment, fine-tune on it, re-run run_tests — has not been executed end to end, by anyone, including whoever wrote it. The dead branch is only visible if you pass the argument the docstring recommends; the TypeError is only visible if you point the suite at tuned weights. Doing either is the first thing a user does.

There is a third snag in the same neighbourhood, for anyone actually building on this. The engine holds one set of weights globally and cannot unload them, so Needle.__init__ raises if you construct a base-model agent after a tuned one — and every environment builds its agent at import time. Importing two environments in one process after loading tuned weights does not do what you would expect.

Eight undocumented knobs

While I had the engine open: libneedle.so reads eight NEEDLE_* environment variables.

NEEDLE_CONFIDENCE   NEEDLE_CONF_RESCORE   NEEDLE_DEBUG      NEEDLE_KV_BITS
NEEDLE_KV_WINDOW    NEEDLE_NO_REBASE      NEEDLE_STRICT_VALIDATE   NEEDLE_THREADS

None of them appear in the model card, the README, or the Python package — the two the Python side reads (NEEDLE_HF_REPO, NEEDLE_LIB_PATH) are a disjoint pair. And every one of the six environment files sets one of them, above the import, before the engine ever loads:

os.environ.setdefault("NEEDLE_STRICT_VALIDATE", "1")
import needle

So a "curated environment" is not only a set of schemas. It is a runtime configuration, and the one line that supplies it is the one line a reader copying the schema style will not copy. If you follow the README's advice and write your own environment from scratch, you get a differently-configured decoder and no indication that you did.

The ledger

What is genuinely valuable. The eight schema rules, and above all the two that generalise: delete the optional field the model wants to guess, and choose enum strings that cannot hide inside the words your users will say. The enum-grounding observation is a real, specific, reproducible property of constrained decoding on small models, and I have not seen it written down elsewhere. The five-tools-per-environment discipline matched to the engine's five-tool retrieval page is the kind of detail that only comes from having shipped the thing.

What does not hold. "90%+ on held-out production tasks" is three claims and all three fail: 90% is a threshold rather than a measurement, nothing is held out in any of the three available senses, and none of it came from production. The 192 cases are 32 cases and six vocabularies. And the strictness that is enforced — nine critical cases, hard zero — sits entirely on refusals, so it is satisfied completely by a model that never does anything.

What I would fix, in an afternoon. Change .get("confidence", 0.0) to (response.get("confidence") or 0.0) and the suite runs against tuned weights. Make min_confidence=0.4 the default so the gate matches the contract. Print the score rather than a boolean, so there is a number to quote. Split the test cases into a cases/ directory the schemas do not sit in, and add ten dictated-sounding queries per environment — the ones with a false start and a fused request — and the suite starts being about production instead of resembling it.

What I would watch. Whether anyone reports a number. Needle 2 is a genuinely interesting artefact — 45M parameters, 2-bit from pretraining onward, running off flash on a $5 chip — and the fine-tuning result it rests on is real and load-bearing. It deserves an evaluation that could have failed. A repository of six acceptance suites, all templates of one another, whose pass condition is a threshold nobody has published a score against, is not that yet. The pieces to make it that are already in the files.