agent-desktop from the source: a desktop as refs, and what a snapshot costs in tokens

mdjsonmcpcopy↗ai

2026-10-31 · 20 min · computer-use · agents · harness · rust · accessibility · desktop-automation

Why read this

Solid· top 85%

Core crate tests run (1,515 of 1,517 pass); Python port of the allocator matches the golden fixture; tokens on toy trees. CLI not run.

• Original analysis
• Runs on a laptop CPU
• Concrete numbers to act on

·Agents & harnesses·Apache-2.0·Practitioner tool

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 58 of 100, ranked 438 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

@mdlahfir introduced agent-desktop as computer use for agents: end-to-end tests of desktop apps, apps read through the accessibility tree, a way to drill into part of a window to save tokens, many agents on one app at once, macOS, Windows and Linux. The tool is Vercel Labs' and Apache-2.0. Its README puts it in one line: "A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref and snapshots again to check, with no pixel matching; the loop stays in your agent."

I cloned the repository at commit cb31212 (2026-10-30, release 1.0.2) and read it. An earlier version of this article said I had not run any of it. I have now run the platform-independent core crate's test suite (see "What running it showed"), but not the command-line tool: this machine has no desktop session and no AT-SPI accessibility bus to walk. So every claim below is one of three kinds. Measured means I computed it from the files or from my own code. Reported means the project's own documents say it and I did not re-run it. Reasoned means my arithmetic on the other two.

What it is

A Rust workspace with a command-line binary and a C library over three platform adapters. The README counts 60 command names, 56 of them operational: the four that hold a key or button down refuse to run until a daemon can release them (reported). One envelope comes back per command, {"version":"2.5","ok":…,"command":…,"data" | "error"}, with 16 error codes.

The adapters are the bulk of it. Counting lines of Rust by script (measured), the three adapters hold 134,761 production lines against 30,969 in the core, 4.4 times as much, and Linux alone is 59,331. That fits what the dependencies say. Linux speaks AT-SPI to the desktop's accessibility bus over D-Bus with zbus directly, and drives X11 with x11rb and Wayland through the desktop portal and Mutter's remote-desktop interface, so it carries a protocol client and two display servers. Windows uses the uiautomation crate and windows for UI Automation. macOS uses accessibility-sys for the AX C API and core-graphics for synthetic events (read from the Cargo.toml files).

Test code is as large as the product: 171,161 test lines against 183,065 production lines, and 6,789 #[test] attributes across the workspace (measured, counting *_tests.rs, tests.rs, tests/ directories and test-support files as tests, and crate src/ of the CLI included).

One idea: a ref instead of a pixel

snapshot walks a window's accessibility tree and gives certain nodes a ref. The format is @<snapshot_id>:e<N>, a snapshot id of a few characters and a positive integer of at most ten digits, and resolve_ref_target rejects a bare @e8 with INVALID_ARGS (read, refs/ref_token.rs). A ref belongs to its snapshot. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET and the agent takes a new snapshot.

Which nodes get a ref is a rule, and a precise one (read, ref_alloc.rs). A node is ref-able when its role is one of 18 interactive roles (button, cell, checkbox, colorwell, combobox, dockitem, incrementor, link, listbox, menubutton, menuitem, option, radiobutton, slider, switch, tab, textfield, treeitem), or when it advertises a primary action. Three actions are discounted, SetFocus, RightClick and ScrollTo, and the comment says why: web runtimes advertise context-menu and scroll-into-view on nearly every node, so counting them "would ref-allocate every inert container and text run and overflow the refmap size limit on web apps."

The refs are not only for elements the agent can click. When a depth limit cuts a branch, the cut node carries a children_count, and the allocator tries to give the cut node, or the nearest ancestor that something could later re-identify, an anchor ref so the agent has something to drill into. That is the part worth testing.

I ported the allocator and checked it against the repo's tests

The repository has a golden fixture, tests/fixtures/skeleton-tree.json, and a file of unit tests for the anchor rule. I re-implemented the allocator in Python from the Rust (is_ref_able, is_collapsible, plan_skeleton_anchors, is_resolvable_skeleton_anchor, and the ref numbering), with the simplifications listed at the top of the file: no macOS retained objects, no mutable-value subtlety in stable text identity, no refmap size limit. Then I encoded the repository's own cases as assertions.

All ten pass (measured): the port reproduces the golden skeleton tree exactly, with refs @e1 to @e4 in pre-order and no ref on the fully read Content container. A native id on an ancestor covers three unnamed boundaries, so the ancestor gets the one ref. A boundary whose only actions are RightClick and ScrollTo still gets an anchor, with its stored actions emptied. An unlabeled boundary with a rectangle gets a ref and keeps the rectangle, because geometry is its only evidence, while a named boundary has its rectangle dropped. The deepest resolvable anchor wins. A drill-down never creates anchors. The script is public/articles/agent-desktop/test_port.py; it does not prove the port matches the Rust everywhere, only on the cases the authors thought to write down.

The widget runs that same rule on a three-level tree. Try the case the rule cannot cover: an unlabeled boundary with no native id, no name, no rectangle and nothing above it. The output shows a children_count and no ref, so the agent can see that something was cut and has no handle to open it. That is the honest edge of the design.

Reading the tree on three operating systems

The walkers carry hard limits (read). On Linux a walk stops at a raw depth of 50, 100,000 edges, 4,096 children under one node and 64 KiB in one field, and it expands 8 nodes at a time over D-Bus. The node cap is 50,000 for a snapshot (an earlier version of this article said 10,000; see below). On Windows the raw depth is also 50, one parent is capped at 10,000 siblings, and a constant named BOUNDARY_COUNT_BUDGET gives the child count at a depth cut 25 ms. The depth the agent asks for is a second, logical limit: the default is 10 levels, and web wrappers do not count against it. A wrapper is a group with no name, value, description or id, which "consumes raw depth but no logical depth". The glossary's reason is that skipping them "lets a dense web app fit a default depth budget; without the skip, the same app understates its reachable content by an order of magnitude." On Windows that skip applies only to Chromium windows, found by the root window class Chrome_WidgetWin_1; on macOS and Linux it applies to every app.

The core insists that every property an adapter reads comes back as Known, Absent or Unknown, never two of them collapsed. Absent is an answer, and Unknown is a failed read that must not pass for one. The same care separates four outcomes of a failed read: a settled absence that is never retried, a retryable transport failure, an element that has gone, and a terminal failure. A tree whose enumeration fails is marked incomplete rather than reported whole. This is the thing a computer-use tool most needs to get right and the one a quick wrapper usually skips, because a hung target that returns a short tree looks exactly like a small app.

Cold trees: Chromium and Electron

Web content builds its accessibility tree lazily, and the project measured how lazily. The Windows findings ledger has a row on a cold Obsidian launch, an Electron app: first contact returns 12 descendants and the settled tree has 165, a 13.75 times understatement, and settling took 10 to 25 seconds, longer than the default 3,000 ms snapshot deadline "in every measured run" (reported, row A16-11). That is why the deadline is a constant with a comment, why --timeout-ms exists, and why a snapshot that runs out of time now returns ok: true with "complete": false, the tree it did observe and subtree_truncated on every cut node, instead of a TIMEOUT and nothing. A drill-down is stricter: it replaces refs inside an existing snapshot, so it must see its whole subtree or leave the stored map alone.

Two mechanisms close the gap (read). On Linux the walker re-reads a filling web tree every 200 ms and calls it settled after three consecutive reads with the same node count, spending at most 70% of the budget the observation started with. For renderers that have not published a tree at all, the core switches accessibility on and retries with a delay that starts at 25 ms and doubles to 400 ms, so a renderer that wakes quickly is caught quickly and one that never wakes is retried a handful of times instead of continuously. On macOS the adapter sets AXManualAccessibility on Electron apps at most once and then waits for the attribute to read back as ready, following Electron's own guidance.

Skeleton, then drill

The default snapshot returns everything to depth 10. Two flags shrink it. -i drops leaves that cannot take a ref. --compact collapses a group with one child and no payload. --skeleton caps the depth at 3, args.max_depth.min(3), and only when no --root is given. Then --root @s8f3k2p9:e3 re-reads one region (read, commands/snapshot.rs, snapshot_ref.rs).

The drill is more careful than its name suggests. It loads the stored snapshot, strictly re-resolves the root ref against the live UI, observes only that subtree, removes every ref that was previously allocated under that root, allocates fresh ones with the stored path as a prefix, and writes the result back under a file lock with a 2,000 ms timeout. Refs from the rest of the snapshot stay valid. An agent can map a Slack-sized window cheaply, open one region, act, and re-read the same region to verify.

I measured what each flag does on two synthetic trees built to match what the documents describe. One is a native-style file manager: 521 nodes, a 120-row table, shallow, no wrapper chains. The other is an Electron-style chat app: 2,262 nodes, 150 messages, chains of anonymous wrappers, DOM ids on the messages. Tokens are counted with the Qwen3 tokenizer on the compact JSON the CLI writes, plus the envelope. These are toys, not recordings; read the ratios.

• The flags do nothing for a flat, wide app. The file manager costs 15,854 tokens plain and 15,833 with every flag on. The skeleton cuts depth, not breadth, and 120 rows of three cells each are 391 refs however the depth is set. The project's own skill says as much: "A small app needs no skeleton."
• They do a lot for a deep one. The chat app goes from 38,070 tokens plain, to 31,480 with --compact, to 27,752 with -i, to 23,790 with both, to 8,708 with --skeleton -i --compact.
• A drill is nearly free. Re-reading the composer region costs 200 tokens and the channel header 139, and that number does not grow with the app.

The scaling is the point. Over 25 to 600 messages, a full -i --compact snapshot grows by about 149 tokens per message (5,848 to 91,467), the skeleton by about 43 (3,346 to 28,058, one anchor per message container), and the composer drill stays at 200 (all measured on the toy).

What that costs against a screenshot

Computer-use agents that read the screen send an image. I did not measure image tokens here; I used the published rule of (width x height) / 750, with a 1,568 px long-edge cap, which gives about 1,365 tokens for a 1280 x 800 capture and about 1,844 for a 1920 x 1080 frame resized (reasoned). On those numbers a full snapshot of a heavy page costs far more than the picture it replaces: 23,790 tokens against 1,365 to 1,844.

The accessibility tree wins only if you read it in regions. A skeleton once, then a drill per observation, costs 8,708 + 200 k tokens for k observations, against 1,365 k for screenshots. The break-even is k of about 7.5, or 5.3 against the larger image. Below that a screenshot is cheaper.

Three things that this model does not capture, and which matter more than the totals. A screenshot gives the model coordinates and a guess; a ref gives an action that goes through the accessibility API and fails closed. The tree carries state a picture does not, such as checked, expanded, enabled and the value of a field. And the tree is text from inside the app, which is a different trust question, below. The repository itself ships a helper aimed at the cost. Its jev-desktop skill describes keeping "a 150-element JSON tree out of your context" by posting a description of the screen on every turn to a decision endpoint, TypeSafe's by default, so that a decision model of the kind covered in Where to use Jev picks the element. The helper lives in scripts/jev/, ships in no installed package, needs an API key, and sends element roles, names, up to sixty characters of each value and the window title off the machine (read, from its skill file). The native CLI calls no model.

Acting without double-clicking by accident

Reading is half the job. The vocabulary the project defines for acting is the most carefully argued part of its CONCEPTS.md, and it reads like a list of past incidents.

• Strict ref resolution. A stored ref re-identifies its element in graded tiers: the stored positional path first, then a bounded broad search, confirmed by native id first and then by stable text. Only a ref with no meaningful identity may resolve by unique geometry, and on Windows and Linux only when the stored rectangle has positive area. A missing, stale or ambiguous match is rejected instead of guessed. Text that merely mirrors a field's current value does not count as identity.
• Window identity. A window handle can be recycled after a window closes, so identity is the handle plus a process-generation token, and the check is repeated just before each write.
• Auto-wait. A ref action polls for the target to be visible, stable, enabled and unoccluded, for 5,000 ms by default (type adds 30 ms a character), and fails fast on terminal conditions such as an unsupported action or a policy refusal.
• Delivery, not success. Every error and action result carries a disposition: delivery is one of unknown, not_delivered, delivery_uncertain, delivered_unverified and delivered_verified, and retry is safe only for not_delivered. On Windows, SendInput reports success whether or not the input was blocked by integrity levels, so physical steps there report delivered_unverified and the tool detects the integrity boundary by comparing token integrity levels instead. The skill tells the agent to retry only when retry is safe.
• Headless by default. Ref actions use accessibility calls and do not take focus, move the pointer or synthesize keys. If no semantic path exists the action fails closed. --headed is an explicit opt-in, and the skill says never to switch to it to recover from a headless failure.
• Secure fields. A password field's value is withheld on reads, and writes into it never echo the attempted or observed value in steps, messages or post-state.

None of this is exotic on its own. What is unusual is that the failure semantics are specified per failure, which is what an agent needs in order not to submit a form twice.

Many agents on one desktop

The post claims many agents on one app at once. The mechanism has two layers, and the second is the limit.

Reading is concurrent. A snapshot lives in a namespace, the global one or a session (--session, or AGENT_DESKTOP_SESSION), under a state root that defaults to ~/.agent-desktop, and lookups are confined to the namespace they were made in. Each process writes its own trace segment. A session can run in multi-agent mode, where each --agent-id gets its own presentation cursor on macOS, Windows and Linux X11, while snapshots stay shared by session (reported, CONCEPTS.md).

Acting is serialized. All input through the operating system is one pointer and one focus, so mutating commands take an interaction lease first. On Unix it is a file lock at /tmp/agent-desktop-<uid>/interaction.lock plus a process guard, with an inheritable file descriptor so a child process can adopt a held lease (read, session/interaction_lease.rs; I did not read the Windows implementation). The ordering in ref_action_wait.rs is the interesting part: the auto-wait poll runs without the lease, then the lease is acquired, then the ref is resolved again, because anything resolved while waiting was observed without exclusivity. The comment calls that second resolution "the correctness boundary, not redundant work." The result carries a contention count and the lease hold time, so an agent can see that it queued behind another.

So many agents can look at one app at once; they take turns to touch it.

What running it showed

I ran the repository's own tests for the core crate, the one that holds the ref allocator, the snapshot logic and the limits and has no platform dependencies: cargo test -p agent-desktop-core at cb31212, on the toolchain the repo pins (Rust 1.89.0), Linux, four-core box, two build jobs. Logs and commands are in public/articles/agent-desktop/verify/.

• The core tests pass, apart from two that cannot here (measured). 1,515 of 1,517 passed in 20 seconds, including the repository's own test_skeleton_fixture_matches_golden and the drill-down refmap golden. The two failures are private_write_rejects_an_intermediate_directory_symlink and unreadable_segment_is_skipped_with_warning. Both rely on the operating system refusing access, and I ran as root, which it never refuses. They test file permissions, not the snapshot; I did not re-run them as an ordinary user.
• The port holds against the real fixtures (measured). Against the same skeleton-tree.json that the Rust test loads, all ten checks of my Python port pass. run_tokens.py re-run with Qwen3's tokenizer.json wrote a results_all.json byte-identical to the committed one, so every token figure in this article regenerates. I also checked the serialisation by reading AccessibilityNode: field names, the omission of empty fields and the snapshot-qualified @<id>:e<N> refs match the toy's output. The toy trees are still mine, so the ratios are not a recording of a real app.
• Defaults confirmed (measured, by reading the constants and the tests that use them). Depth 10 (default_value = "10" in cli_args/snapshot_tree.rs), a 3,000 ms snapshot deadline (DEFAULT_SNAPSHOT_TIMEOUT_MS), skeleton depth min(3) applied only without --root. Counts: 16 error codes, 60 command names with 56 operational as the README says. The Linux settle poll is 200 ms, three equal reads, 70% of the budget; the activation retry runs 25 ms to 400 ms; the Windows boundary count budget is 25 ms; nodes in flight on Linux are 8.
• One claim was wrong. I wrote that a Linux walk stops at 10,000 nodes. That constant (DEFAULT_MAX_NODES in walk_limits.rs) belongs to WalkLimits::default(), which the gate, resolution and search walks use. A snapshot builds its limits from the request's budget (WalkLimits::for_request), and ObservationBudget::default() allows 50,000 nodes, 100,000 edges, 4,096 children per node and 64 KiB per field. So a snapshot of a very large Linux window gets five times the node allowance I stated; raw depth stays 50. This changes no token figure here, since the toy apps are smaller than either cap.

What still rests on reading: the walkers on real trees, the Chromium settle behaviour, and every latency number. Those need a live accessibility bus.

Evidence, and how often it contradicted the plan

Two directories of probe scripts and a findings ledger per platform explain why the design looks the way it does. Each row records an experiment, what the project assumed before it, what happened and a verdict: CONFIRMS, CONTRADICTS, NEW-EDGE or DEFERRED. Counting the rows my pattern matched (measured), the Linux ledger has 774, 93 of them CONTRADICTS (12%) and 385 NEW-EDGE, and the Windows ledger has 231, 18 of them CONTRADICTS (8%).

One row is worth quoting for its shape. The project had taken it as given that batching property reads with a UI Automation cache request is "3-5x faster". On a 220-node Explorer window with 8 properties it measured 2.69 times overall, below the claim, and the row says the single number hides a split: building the cache makes the find pass slower, 180.25 ms against 117.38 ms, while the read pass falls from 371.62 ms to 1.25 ms, about 298 times (reported, row A6-1). The verdict is CONTRADICTS, and the claim was replaced with the decomposition. That is a project that writes down when its assumption was wrong.

The same culture shows in the checks: a gate must assert something to pass, and a fixture proves the gate still detects what it exists to detect, so a check that quietly stopped finding anything fails instead of reporting a clean tree.

What it can and cannot do to you

The native CLI does not need a model service, and the FAQ says plainly that snapshots, window titles, clipboard reads, notifications, screenshots and session traces contain private desktop content. Debug HTML embeds a screenshot, and replay artifacts are unredacted. Permissions differ by system: macOS needs the Accessibility grant, Windows needs none for apps at your terminal's integrity level, and Linux needs the graphical session with at-spi2-core (reported, README). The security policy asks for private vulnerability reports through GitHub, covering the CLI, installer, FFI, release artifacts and adapters.

One risk is not in the repository's documents, so it is reasoned. The accessibility tree is text that the application, and therefore anyone who controls its content, wrote. A web page's labels, a chat message and a window title all enter the model's context through snapshot the same way. A tool that makes this easy to read does not make it safe to trust, and the headless default, the delivery dispositions and the refusal to climb to physical input are useful brakes, but none of them tells the agent that a button's label was written by a stranger. The sandbox articles on this site make the same point from the other side: Claude Code's sandbox from the source confines what a command can reach, and nothing here confines what a screen can say.

What I did not do

• I did not run the agent-desktop binary: it needs an AT-SPI bus and a graphical app to read, and neither exists here. The measured numbers about snapshot size are still my own, from the port and the toy trees; the core test run checks the allocator and the skeleton, not a real window.
• The toy trees are shaped by me. A real Slack or Finder window would change every absolute token count.
• Tokens are counted with one tokenizer; another model's would shift each number by a margin I did not measure. The screenshot cost is a published rule, not an API measurement.
• The macOS and Windows adapters were skimmed for structure and dependencies, not read function by function, and I did not read the Windows interaction lease or the FFI layer.
• The latency numbers, the Chromium settle times and the cache-request timing are the project's, from its Windows ledger.

The port and the measurements are in public/articles/agent-desktop/: axport.py, gen_apps.py, run_tokens.py, test_port.py and results_all.json. run_tokens.py needs Qwen3's tokenizer.json and regenerates every table here (I re-ran it from the shipped files and got identical numbers); test_port.py takes the path of the repository's tests/fixtures/skeleton-tree.json.

Sources

• vercel-labs/agent-desktop at cb31212: README.md, CONCEPTS.md, docs/json-output.md, docs/faq.md, SECURITY.md, skills/agent-desktop/SKILL.md, probes/windows/FINDINGS.md, probes/linux/FINDINGS.md, and under crates/core/src: snapshot/, refs/, session/interaction_lease.rs, ref_action/ref_action_wait.rs, tree/renderer_accessibility.rs.
• @mdlahfir's launch post.
• On this site: Cua Spaces and arc-driver for other computer-use stacks, Agent harnesses for the loop around the model, and Where to use Jev for the decision models the optional helper calls.