switch: the agent loop for a model that cannot write a tool call, and what it does when the model is wrong

mdjsonmcpcopy↗ai

2026-10-31 · 13 min · agents · decision-models · evals

Why read this

Solid· top 85%

Reads all of switch's core and runs it: 71 tests pass, eleven bad deciders each end a run in a named way, and a request costs 132 to 347 tokens on the toy.

• Facts measured from source
• Runs on a laptop CPU
• Explained from first principles

·Agents & harnesses·MIT·Practitioner tool

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

1 of 3: General advice

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

1 of 3: Some original analysis

Score 49 of 100, ranked 553 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

@onusoz posted switch as "a harness framework for decision models like Jev", and the README opens with the same sentence. I have written a lot on this site about decision models themselves (Jev, Agr, d3-turbo, what they cannot do) and about five ad hoc harnesses in Five Jev harnesses, for a model that cannot write. switch is the first repository I have seen that tries to be the one harness behind all of them. So this piece does the thing the others did not: read all of the core, run it, and push bad answers into it.

Labels used throughout: measured is something I ran this session. Reported is a number from the repository's own docs. Reasoned is inference from code I read.

What switch is

switch is MIT-licensed, version 0.1.1, and needs Node 22.18 or newer. The part you import, src/, is measured at 2,134 lines of TypeScript across core, cli, models and testing. The loop is src/core/loop.ts (338 lines), the pick helper is src/core/pick.ts (127), and the model adapter for pi-ai is src/models/pi-ai.ts (101). The repository also holds three example domains (chess, wikirace, a rubric judge) and a browser viewer, which the core does not depend on.

The README states the design in one paragraph. A decision model "answers typed questions with probabilities and writes no text, so it cannot generate the tool calls that a coding agent harness such as Pi relies on. switch generates the tool calls for it in code, and the model only picks one from that finite set." That sentence is the whole idea, and the code does exactly that.

Who writes what

In a chat-model coding agent the model produces the action: a tool name and its arguments, as text, which the harness then parses. Everything that can go wrong with an action (a tool that does not exist, a malformed argument, a path the model invented) goes wrong in that text. switch moves the writing of the action out of the model. Before the model is called, your code has already decided what the legal actions are and has already filled in their arguments. The model's only contribution is a probability for each label.

Reasoned: this is why the framework fits a narrow job and fits badly a general one. A decision about "which of these 30 legal chess moves" is a menu. "Write the SQL for this report" is not, and no choice function you could write lists every query. For where decision models themselves fit, see Where to use Jev.

The loop, as written

The domain supplies a StateMachine: an initial() state, a questions(state) function, a transition(state, answers) function, and optionally a view(state, tokens, charsPerToken). A state is plain JSON. Questions are typed bool, choice or score, the same three that pi-ai's classifier interface offers, so a request written for switch also runs through the llama.cpp decision-model server or any other backend pi-ai lists.

takeSteps in loop.ts is short enough to hold in your head:

1. Ask machine.questions(state). If the result is empty, the run ends as done.
2. If a step limit is set and exceeded, end as step limit.
3. Ask the model (or take the answer from a recorded trace, which is what replay does).
4. If anything failed, end as failed with a reason string.
5. Call machine.transition(state, answers) and record the new state. Repeat.

Measured: I ran pnpm install on the root package and vitest run. All 71 tests in 8 files pass in about 4 seconds (vitest 5.0.5, Node 22.22.0). I did not install the examples' workspaces, so their tests are not in that count.

A pick is one choice question with short labels

Most domains ask one thing per step: which action next. picking() in pick.ts builds that question. Each choice has a unique key and a unique text; duplicates throw a ChoiceError before the model is called. The model sees labels o1, o2, and so on, each next to its text. The mapping from label back to key lives in the question's data field, which sendable() strips from the request, so the model never sees keys. This matters for chess, where a move key is a long algebraic string: the model reads a short label and a short text and the domain translates back.

Labels are rebuilt on every step. After o1 meant "Read log1.txt" at step one it means "Read log2.txt" at step two. The model has no stable name for an action across steps, so the only state it can use is what the view shows it.

A worked trace

I took the README's ticket-triage domain verbatim (read a log, or stop, then choose a team), gave it eight synthetic 200-character logs, and ran it against a scripted decider that reads three logs, picks stop, then answers "infra". Step through what the loop did:

Measured: five steps, ended done, 18 trace entries, 15.9 ms of loop time for the whole run. The estimated request grew from 132 to 347 tokens as the view accumulated logs, then dropped to 314 on the final question, whose menu has three options instead of six. The estimate is switch's own: estimateTokens is the JSON length of the request divided by three. It is a safety number, not a tokenizer.

What a request costs

Two things set the size of a request: the view (your state, shaped by your view function) and the menu (the options in the pick).

Measured: with the view holding no logs, a six-option menu estimated at 111 tokens and a twenty-option menu at 216. That is a slope of 7.5 tokens an option over a base of about 66. Two points on a domain with short option texts is a fit, not a law; the widget lets you slide it but it will undersell a domain whose options are paragraphs.

Measured: with 50 logs, the first request showed 20 of 51 options. The default limit is set by the domain; the README's triage uses 20 and the legal judge, per its docs, uses 30. A cut keeps the choices of lowest priority and stop is given priority -1, so the way out of the loop is never the thing that gets cut. A cut option is not lost, since reading one changes the state and the next menu is drawn from what remains. In my run, an always-first-label decider read 20 logs in 20 steps and hit the domain's step limit, never seeing logs 21 to 50 in a single menu.

Reported: the repository's first legal-judge run (docs/2026-10-25-first-legal-judge-run.md) made 319 calls with a median of 0.29 s each and 3.2 million input tokens in total, so about 10,000 input tokens a call. It reports the cost at another provider's listed Jev price of $0.042 per million input tokens as about $0.13 for 100 answers. I did not rerun this; it needs the TypeSafe API and the private answers.

Measured: loop overhead. I ran 500 steps of the triage domain with a fake decider that answers instantly, over 2,000 logs, and the whole run took 1,849 ms, 3.7 ms a step. That includes JSON-stringifying the request for the size estimate and storing every full state in the trace, and the state grows by one log each step, so this is an upper bound for a domain with a flat state. Next to a 0.29 s median model call, the framework is about 1% of a step.

When the decision model is wrong

This is the part I cared about. In a chat harness a bad action is text, and the harness spends most of its code deciding what to do with text that does not parse. In switch the model's output space is already a closed set, so there are only a few ways for it to be wrong. I wrote eleven deciders, each returning something specific, and ran each against the triage domain.

Six of the eleven runs ended failed with a named reason; the rest finished, hit the step cap, or threw. What the loop does, by row (measured, with the code that does it read):

• A label not on the menu, a label from another question, a wrong answer type, an empty answer: misfitOf checks every answer against its question. The run ends as failed after that step, with a message that names what was wrong, for example the model answered o99 to pick, which is none of its labels. There is no retry and no fallback to a default choice. This is deliberate and I think right: a decision model does not normally produce these, so a hit means the backend is broken, and carrying on would hide it.
• An API error: askModel returns the error message as the failure and the run ends failed. There is no retry in switch itself; piAiDecider passes maxRetries: 2 and a 300 s timeout down to pi-ai, so network retries happen below the loop.
• A request that does not fit the window: this is the one recoverable case. If the decider reports tooLong (the adapter matches max_tokens_exceeded, context length, too long or too many tokens in the error), ask() asks the view function for a smaller view, 0.7 times the previous token budget, and sends again, for three attempts in all. A domain with no view function gets one attempt, since there is nothing to shrink. In my runs a decider that was too long once finished the case with a note in the trace; one that was too long every time ended failed after three calls and two notes.
• A decider that throws: the exception escapes run(). The loop does not catch it and writes no failed ending; the trace has a thrown entry type, which I take to be for the runner layer above the loop, but I did not run the CLI to check. If you call run() yourself, wrap it.
• A model that never stops: with "always o1", the triage run read all eight logs, found that only stop remained so o1 was stop, then answered the team question: ten steps and done. The same decider on 50 logs ran into the 20-step cap and ended step limit. The cap is optional, off unless the domain sets it; the README's triage example sets 20.

Reasoned: the failure surface is much smaller than in a chat harness, and what remains is infrastructure failure (the API, the window) rather than model failure. The one thing the framework cannot detect is a valid wrong answer. An always-last decider stops at once and picks the last team, and the loop is content with that.

Traces, replay and forks

Every run writes a trace: each state, each question set, each answer with its probabilities. replay runs a case again from a trace without asking the model, and --fork n repeats the first n steps and lets a model carry on. The loop code shows how: when a replay is active, answersAt takes the recorded step and fitted() checks it against the questions the domain asks now, matching a pick's answer by choice key rather than by label, so a menu reordered since the recording still resolves. If the domain changed its definition, the run notes it and carries on.

Reasoned: replay plus fork is the most useful property for someone building a domain. You can change a choice function, replay yesterday's trace to see where the new menu differs, and only pay for the model from the fork point on. I read this path and the replay tests pass; I did not run a replay on a real trace.

Against a coding-agent harness

A coding agent like Codex or Pi hands the model a transcript and a set of tools, lets it write a tool call, runs it, and appends the result. The loop is open-ended and the model does the planning in text. switch has the same shape (observe, choose, act, repeat) but a different split of labour:

• Who proposes actions. The model in a coding agent; your code in switch. The model only ranks.
• State. A transcript that only grows, versus a JSON state your transition function owns. In switch the model sees whatever view shows, which can drop old evidence to fit a budget. A chat transcript can be truncated too, but it is the harness's job to decide how.
• Termination. The chat model decides to stop by writing a final message. In switch the domain stops when its question function returns nothing, and a domain that wants the model to decide offers a stop choice.
• Cost per step. Output tokens are the expensive part of a chat step and a decision model emits none. Whether that wins depends on how many input tokens you resend: the legal-judge run above averaged roughly 10,000 per call.
• What you give up. Any action your code cannot list. A coding agent can try something nobody anticipated. switch cannot.

What I did not run

• No real decision model. The accuracy claims in the README (94.6% and 95.0% agreement on 1,288 rubric items in the legal-judge run, level with DeepSeek-V4.1-Flash and above GPT-6 Luna) are reported, and the docs note the answers had been seen in earlier work and that it is not a held-out result. The README itself says that Jev is weak at chess and is not yet good enough to replace the strongest chat models as a judge.
• No example harness. Chess needs chess.js and optionally Stockfish, wikirace needs the Wikipedia network, and rubric-judge needs PDF tooling and the private cases. Disk on this machine was nearly full, so I installed only the root package and deleted it afterwards. My triage domain is the README's code, run with fakes.
• No latency for a real backend. The 0.29 s median is the repository's number for Jev through TypeSafe. Where a model runs locally on a small CPU, the cost will be that model's forward pass, which I did not measure.
• No CLI. run, view and replay as commands were not run.

What to take from it

If your task is a finite menu repeated over a state (route a ticket, move in a game, choose the next document to read), switch is a clean place to start: about 2,100 lines, a loop you can read in an afternoon, a trace and a replay that make debugging a domain tractable, and failure modes that stop loudly instead of continuing wrongly. If the task needs the model to write something, none of it applies.

The number I would carry away is the overhead one. The framework costs about 3.7 ms a step in my measurement, and the model call costs roughly eighty times that in the repository's own report. The interesting engineering is all in your choice function and your view, which is where the repository's docs spend their effort too.