OpenMuse: "compatible with any agent harness" is one ternary branch

OpenMuse is a self-hostable personal agent: a React Native app for iOS, Android and web, a Hono server, a Playwright browser worker, and an optional Docker Linux box the agent can type into. It is MIT, it is seven days old as I write this, and its README leads with a portability claim.

A personal agent with a browser, terminal, files, and work that keeps going. Compatible with any agent harness.

Three commits ago — earlier today — that sentence ran "…and work that keeps going and works with any agent harness." Someone split it and softened the verb, which is usually a sign that someone checked it.

I spent a day checking it too, because this is a claim with a shape the corpus has learned to test. An abstraction over harnesses is worth exactly its adapter list. So: how many adapters are there, what does each one do, and — the question that actually matters for a product — what is still yours once the agent behind the abstraction is somebody else's?

Counting the adapters

packages/backends/ contains one directory, src/, containing one file, openbot.ts, 343 lines. It has no package.json. It is not a workspace member — pnpm-workspace.yaml lists apps/mobile and apps/worker and nothing else. Nothing under apps/server imports it. grep -rn OpenBotAdapter over the whole tree returns the class itself, tests/openbot.test.ts — which imports it by relative path and exercises it 13 times against a fake transport — and one link in docs/OPENBOT-INTEGRATION.md. That is every reference.

The adapter itself is careful work. It takes an injected OpenBotTransport rather than calling global fetch, so it cannot carry a shared administrator token; it Zod-parses every response; it marks ambiguous mutations outcomeUnknown and refuses to retry them; and it freezes a list of the capabilities it does not cover:

// packages/backends/src/openbot.ts
readonly unsupportedCapabilities = Object.freeze([
  "gmail",
  "calendar",
  "pdf",
  "approval_persistence",
] as const)

It is also inert. probe() opens with if (this.options.enabled !== true) return { state: "disabled" }, and enabled defaults to undefined. The roadmap says the same thing in plain English: "OpenBot user/session bridge, routines, and computer backend. The disabled HTTP adapter is contract-tested; it is not a live connection."

So the adapter count for "any agent harness" is one, disabled, and connected to nothing. That is not the interesting finding, because the adapter is not where the portability comes from.

The seam is five lines

The real answer is an environment variable. AGENT_BACKEND takes three values — sample, model, agui — and apps/server/src/agent.ts, all 64 lines of it, turns that into a choice of constructor:

// apps/server/src/agent.ts
const agents: AgentsFactory = async ({ request }) => ({
  default:
    config.agentBackend === "sample"
      ? new ConversationAgent(config, service, await auth.owner(/* … */))
      : config.agentBackend === "agui"
        ? new HttpAgent({
            url: config.agentUrl ?? "http://127.0.0.1:1/unconfigured",
            headers: config.agentToken ? { Authorization: `Bearer ${config.agentToken}` } : {},
          })
        : new ConversationAgent(config, service, await auth.owner(/* … */)),
})

That is the whole of it. The agui arm is five lines. Everything else the feature needs is a two-line reachability test in agentConfigured(), a three-value union and its validation in config.ts, and two commented lines in .env.example — a dozen, generously counted, across three files. The unconfigured fallback points at port 1, which is not a routable service: fail-closed, deliberately.

The three constructors are interchangeable because they all extend AG-UI's AbstractAgent, whose contract is one method: run(input: RunAgentInput): Observable<BaseEvent>. Everything else about an agent is that agent's own business.

This is a better design than a per-harness adapter table, and it is a weaker claim at the same time — the two facts are the same fact. OpenMuse does not integrate with harnesses. It speaks a protocol, and any harness that also speaks it can be dropped into the slot. There is no LangGraph adapter, no CrewAI adapter, no Claude Code adapter, because there is nothing for such an adapter to do. .env.example is honest about the consequence in a comment the README does not repeat:

# An optional external raw AG-UI agent replaces conversational routing only.
# AGENT_BACKEND=agui
# AGENT_URL=https://your-agent.example/run

"Replaces conversational routing only." Hold onto that sentence.

What AG-UI actually specifies

Since the portability rests entirely on the protocol, it is worth pinning down what the protocol is.

AG-UI is a schema for the conversation between a front end and an agent. The client side of it, in the version OpenMuse pins, is about as small as a protocol client gets — this is the whole of HttpAgent.requestInit from the published @ag-ui/client@0.0.59 bundle, with the minifier's one-letter names expanded:

requestInit(input) {
  return {
    method: "POST",
    headers: { ...this.headers, "Content-Type": "application/json", Accept: "text/event-stream" },
    body: JSON.stringify(dropNullSubagentRunIds(input)), // one normalisation pass
    signal: this.abortController.signal,
  }
}

run() is then this.fetch(this.url, this.requestInit(input)) piped into an event parser. That is the client.

One POST. The body is a RunAgentInput: threadId, runId, optional parentRunId, state, a messages array discriminated on role, tools, context, forwardedProps. The response is a stream of typed events, and the agent's job is to emit them in a well-formed order — a run opens with RUN_STARTED and closes with RUN_FINISHED or RUN_ERROR; a message is a TEXT_MESSAGE_START, one or more TEXT_MESSAGE_CONTENT deltas, then TEXT_MESSAGE_END; a tool call is the same shape with TOOL_CALL_START / TOOL_CALL_ARGS / TOOL_CALL_END, and whoever executed the tool returns a TOOL_CALL_RESULT.

Two details are worth stating precisely, because "AG-UI compatible" gets used loosely.

First, the event vocabulary is bigger than the documentation says. The spec page describes "16 standard event types". The EventType enum shipped in @ag-ui/core@0.0.59 has 36, including THINKING_*, REASONING_*, ACTIVITY_SNAPSHOT/DELTA, SUBAGENT_STARTED/FINISHED/ERROR, and REASONING_ENCRYPTED_VALUE. The protocol has been growing toward the things frontier harnesses actually stream. A harness that emits only the documented 16 is compatible; a client that assumes only 16 will arrive is not.

Second, and more to the point here: AG-UI is deliberately transport- and behaviour-agnostic. The docs say so — "AG-UI doesn't mandate how events are delivered", and "Events don't need to match AG-UI's format exactly — they just need to be AG-UI-compatible." It standardises the shape of the conversation. It says nothing whatsoever about what tools exist, who may call them, what a tool result means, or who is allowed to approve an action.

What the swap takes with it

Here is the part that makes "replaces conversational routing only" concrete.

OpenMuse's tools are not registered on the CopilotKit runtime. They are arguments to a BuiltInAgent that ConversationAgent.run() constructs on every request, inside the branch that AGENT_BACKEND=agui replaces. Twenty-nine distinct tool names, forty registrations across two agents:

Point AGENT_URL at your own harness and every one of those names goes away. Your harness brings its own tools, which is the entire point of a harness — but it brings them to its machine, not to OpenMuse's mailbox, browser profile, Docker container or PDF store. The RunAgentInput it receives carries messages and context. It does not carry run_computer_command.

The client makes the same cut from the other side, and this is where it stops being an abstract point about layering. apps/mobile/src/chat.tsx calls CopilotKit's useRenderTool() eight times, each keyed on an exact server tool name:

// apps/mobile/src/chat.tsx
useRenderTool({
  name: "browse_web",
  description: "Follow the agent as it reads a webpage",
  parameters: displayParameters,
  render: ({ args, result, status }) => (
    <BrowserToolCard url={args.url} result={result} loading={status !== "complete"} />
  ),
})

search_mail and read_mail_thread draw the email card. browse_web draws the browser card with its live screenshot and the Take control button. delegate_task, agent_status, create_goal, watch_page and remember_fact each draw a labelled server card that deep-links into the right screen. Match by string, and no fallback that inspects an unknown tool.

So the interface — which is most of what a personal assistant is — is a lookup table on OpenMuse's own tool names. Here is the product being the interface, for forty-two seconds:

One smaller thing in the same seam, stated because I checked it rather than because it bites. ConversationAgent passes the client's declared tools through a filter before handing them to the model:

agent.run({ ...input, tools: input.tools.filter((t) => t.name === "open_workspace") })

open_workspace is not defined anywhere in the repository — that string appears exactly once, in the line above. So the filter's allow-list is empty in practice and every client-declared tool is stripped. Fail-closed, and the kind of vestige that is fine until someone reads it as an interface.

Computer use, plainly

"Browser, terminal and files" on a machine you own is the security surface, so it deserves a flat description of where the trust boundary sits. OpenMuse publishes a SECURITY.md that does most of this work itself, including the parts that are not flattering, and I will credit it as I go rather than pretend I found it.

expand

The terminal is a container, not a shell. run_computer_command is disabled by default (COMPUTER_ENABLED=false) and requires you to build a local image. When enabled, the only host process the server will ever spawn is docker, with an argv array and shell: false, and the source says why in a comment that reads like a scar: "User input is an argv element or stdin, never a host shell program. Do not add a shell fallback here." The command itself runs as:

docker exec --user 1000:1000 --workdir <cwd> <container>
  /usr/bin/timeout --signal=TERM --kill-after=2s 30s
  /bin/bash --noprofile --norc -c "<command>"

There is no allowlist and no sandbox beyond the container — arbitrary bash, up to 16,000 characters, and the isolation is entirely in how the container was created: --user 1000:1000, --read-only, --cap-drop ALL, --security-opt no-new-privileges, --network none, --ipc private, --memory 512m with swap pinned equal, --cpus 1, --pids-limit 128, one tmpfs at /tmp mounted noexec, a single named volume at /workspace, --pull never, and an entrypoint of sleep infinity. No host directory is mounted, no Docker socket, no provider key, no Google token.

The part I did not expect is that the server does not trust its own docker create. Before attaching to any existing container it runs docker container inspect and checks the result against 38 conjoined predicates — every flag above, plus the image reference, the label set, the entrypoint and command arrays, an environment allowlist, the mount list, and the network map — failing closed on the first mismatch:

// apps/server/src/computer.ts
if (!safe)
  throw new AppError(
    "Computer ownership or isolation does not match this deployment; refusing to attach",
    409,
  )

That is the difference between "we start it with the right flags" and "we will not talk to it unless it still has them," and it is the single most defensive thing in the repository.

"Files" does not mean your files. There are two file surfaces and neither is your home directory. The container tools operate inside /workspace through a path guard that rejects .., null bytes, symlinks and non-regular files, capped at 256 KB per text read or write. The app's Files are PDFs written to DATA_DIR/files/<uuid>.pdf with mode 0600 and addressed by generated id. No tool in the tree takes a host path.

The browser is a separate service with its own egress policy, and — unlike BrowserSkill, which hands the agent the Chrome you are already signed into — it is never your browser. The Playwright worker runs in its own container with its own profiles, authenticates the server with a shared WORKER_TOKEN, and exposes no JavaScript evaluation endpoint. Destinations are checked against a hand-written public-unicast test that rejects loopback, link-local, CGNAT, documentation and reserved ranges in both families, and — the good bit — Chromium is pointed at an internal loopback proxy that resolves the hostname once and then connects to that IP, so a second DNS answer cannot rebind the socket somewhere private. CONNECT tunnels are restricted to port 443; QUIC and non-proxied WebRTC are disabled. The project states the limit of this itself: it is application-enforced egress, not a kernel firewall, and Playwright's default launch disables Chromium's own sandbox.

What leaves the machine. The provider key stays server-side, but the data does not stay local, and that is inherent rather than a flaw: a model that reads your mail has to be sent your mail. Concretely, in the default model backend, a provider request can contain up to 20 message summaries with 240-character snippets, a full thread of up to 20 messages at 12,000 characters each, up to 30,000 characters of page text per browse_web, and up to 128 KB of combined command output per terminal run. For delegated tasks the system prompt is built by interpolating every stored memory, the task's prior state, its evidence list and its artifact ids as JSON — so all saved personal context goes with every run of that task. Google credentials are encrypted at rest with AES-256-GCM under a versioned envelope, the authorization flow uses PKCE with an S256 challenge and a ten-minute state row, and the scopes requested on connect are read-only (gmail.readonly, calendar.events.readonly, calendar.calendarlist.readonly) with gmail.send and calendar.events added only when you separately grant write access.

The residue

So: the model is swappable, the tools are OpenMuse's, the interface is a lookup on OpenMuse's tool names. What is actually left in the shell, once you subtract the part any harness could supply?

Three things, and they are the product.

First, the work that never reaches a model. A delegated task has a kind — agent, document, monitor, finance, plan — and three of the five are decided in deterministic TypeScript with no inference anywhere: document runs attachment → PDF → requested field values → filled copy → prepared reply; monitor runs a page check with exponential failure backoff and an automatic pause after five consecutive failures; finance parses a CSV in exact cents. Only agent and plan fall through to executeModelTask. This is the split the three-tier posts argue for, shipped without ceremony: if the job has a shape, code owns it, and the model is the fallback for jobs that do not.

expand

Second, the durable task machine. Tasks live in SQL with plans, checkpoints, attempt counts, leases and scheduled wake-ups. Two workers can race for the same task and one gets the lease; an expired lease is recovered; pause, resume, cancel and retry are real state transitions rather than UI affordances. Tool executions inside a model run are serialised through a promise queue, because a provider can request parallel tool calls and durable checkpoints must stay ordered. Each computer command carries an operationId that is hashed into an idempotency key, and re-issuing the same id with a different command is a 409 rather than a second run.

Third — and this is the one no protocol could carry — the approval gate.

The model's most powerful tools are prepare_email and prepare_event, and neither sends anything. They build a proposal, flip the task to waiting_approval, and stop. The prompt states the rule in the imperative — "External writes require prepare_email/prepare_event; there is no tool to approve them" — and the code makes it structural: the only caller of ActionService.decide with "approve" is the handler for POST /api/actions/:id/decide, an authenticated request from the app. The two calls to decide inside the task engine both pass "deny".

The gate is not a model-only safeguard, which is the part that convinced me it is structural rather than decorative. The document workflow never invokes a model at all — its reply body is a hardcoded three-line template unless you supply one — and it still ends at prepare() and waiting_approval like everything else. Nothing in this system sends mail without a stored approval, including the parts with no model in them.

What is gated is not just "did a human click yes". The proposal is hashed over its input, its connection, its target and the target's version, and approving requires presenting that hash back, so an approval is bound to the exact thing that was shown. The connection id and the Google account are re-checked at decision time, so reconnecting a different account invalidates everything pending. And the real interlock is one UPDATE … WHERE that re-tests status, expiry and the parent task's state in the same statement, so two racing approvals both pass the TypeScript and exactly one gets a row back.

What this says about harnesses

The corpus has spent a lot of words on what belongs in a harness — the harness as the thing that actually determines behaviour, deterministic code versus decision model versus reasoning model. OpenMuse is the consumer-facing end of the same argument, and reading it settles something.

The harness-agnostic framing suggests a layering where the harness is the substantial part and the app is a skin over it. The code says the opposite. HttpAgent is a POST and a stream reader. ConversationAgent is 283 lines that mostly declare tools. The parts with real content are the twenty-nine tool definitions and their caps, the isolation policy and its 38-predicate re-verification, the egress proxy that pins an IP, the task machine with its leases and idempotency keys, the proposal gate with its hash binding and its outcome_unknown state, and the eight-entry table that turns tool names into cards a person can act on.

None of that is portable, and none of it is the model's. The swappable layer is the thin one. That is not a criticism of AG-UI — it is what a well-scoped protocol looks like. It is a correction to how a sentence like "compatible with any agent harness" tends to be read. It is true, it costs five lines, and those five lines are not where the assistant lives.

The honest version of the README's promise, which the .env.example comment already gets right: you can replace who does the talking. You cannot replace who holds the keys.