# agent-desktop from the source: a desktop as refs, and what a snapshot costs in tokens

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/agent-desktop
> date: 2026-10-31
> tags: computer-use, agents, harness, rust, accessibility, desktop-automation

[@mdlahfir](https://x.com/mdlahfir/status/2116316228046500348) introduced `agent-desktop` as computer use for agents:
end-to-end tests of desktop apps, apps read through the accessibility tree, a way to drill into part of a window to save
tokens, many agents on one app at once, macOS, Windows and Linux. The tool is Vercel Labs' and Apache-2.0. Its README
puts it in one line: "A snapshot returns JSON with a ref for every element an agent can act on. The agent acts on a ref
and snapshots again to check, with no pixel matching; the loop stays in your agent."

<RepoCard repo="vercel-labs/agent-desktop" />

I cloned the repository at commit `cb31212` (2026-10-30, release 1.0.2) and read it. An earlier version of this article
said I had not run any of it. I have now run the platform-independent core crate's test suite (see "What running it
showed"), but not the command-line tool: this machine has no desktop session and no AT-SPI accessibility bus to walk. So
every claim below is one of three kinds. **Measured** means I computed it from the files or from my own code. **Reported** means the project's own
documents say it and I did not re-run it. **Reasoned** means my arithmetic on the other two.

## What it is

A Rust workspace with a command-line binary and a C library over three platform adapters. The README counts 60 command
names, 56 of them operational: the four that hold a key or button down refuse to run until a daemon can release them
(**reported**). One envelope comes back per command, `{"version":"2.5","ok":…,"command":…,"data" | "error"}`, with 16
error codes.

<AdapterStack />

The adapters are the bulk of it. Counting lines of Rust by script (**measured**), the three adapters hold 134,761
production lines against 30,969 in the core, 4.4 times as much, and Linux alone is 59,331. That fits what the
dependencies say. Linux speaks AT-SPI to the desktop's accessibility bus over D-Bus with `zbus` directly, and drives X11 with
`x11rb` and Wayland through the desktop portal and Mutter's remote-desktop interface, so it carries a protocol client and two
display servers. Windows uses the
`uiautomation` crate and `windows` for UI Automation. macOS uses `accessibility-sys` for the AX C API and
`core-graphics` for synthetic events (**read** from the `Cargo.toml` files).

Test code is as large as the product: 171,161 test lines against 183,065 production lines, and 6,789 `#[test]`
attributes across the workspace (**measured**, counting `*_tests.rs`, `tests.rs`, `tests/` directories and test-support
files as tests, and crate `src/` of the CLI included).

## One idea: a ref instead of a pixel

`snapshot` walks a window's accessibility tree and gives certain nodes a ref. The format is
`@<snapshot_id>:e<N>`, a snapshot id of a few characters and a positive integer of at most ten digits, and
`resolve_ref_target` rejects a bare `@e8` with `INVALID_ARGS` (**read**, `refs/ref_token.rs`). A ref belongs to its
snapshot. When the UI changes, an action returns `STALE_REF` or `AMBIGUOUS_TARGET` and the agent takes a new snapshot.

Which nodes get a ref is a rule, and a precise one (**read**, `ref_alloc.rs`). A node is ref-able when its role is one of
18 interactive roles (`button`, `cell`, `checkbox`, `colorwell`, `combobox`, `dockitem`, `incrementor`, `link`,
`listbox`, `menubutton`, `menuitem`, `option`, `radiobutton`, `slider`, `switch`, `tab`, `textfield`, `treeitem`), or when
it advertises a primary action. Three actions are discounted, `SetFocus`, `RightClick` and `ScrollTo`, and the
comment says why: web runtimes advertise context-menu and scroll-into-view on nearly every node, so counting them
"would ref-allocate every inert container and text run and overflow the refmap size limit on web apps."

The refs are not only for elements the agent can click. When a depth limit cuts a branch, the cut node carries a
`children_count`, and the allocator tries to give the cut node, or the nearest ancestor that something could later
re-identify, an anchor ref so the agent has something to drill into. That is the part worth testing.

## I ported the allocator and checked it against the repo's tests

The repository has a golden fixture, `tests/fixtures/skeleton-tree.json`, and a file of unit tests for the anchor rule.
I re-implemented the allocator in Python from the Rust (`is_ref_able`, `is_collapsible`,
`plan_skeleton_anchors`, `is_resolvable_skeleton_anchor`, and the ref numbering), with the simplifications listed at the top
of the file: no macOS retained objects, no mutable-value subtlety in stable text identity, no refmap size limit. Then I
encoded the repository's own cases as assertions.

All ten pass (**measured**): the port reproduces the golden skeleton tree exactly, with refs `@e1` to `@e4` in
pre-order and no ref on the fully read `Content` container. A native id on an ancestor covers three unnamed boundaries, so
the ancestor gets the one ref. A boundary whose only actions are `RightClick` and `ScrollTo` still gets an anchor, with
its stored actions emptied. An unlabeled boundary with a rectangle gets a ref and keeps the rectangle, because geometry is
its only evidence, while a named boundary has its rectangle dropped. The deepest resolvable anchor wins. A drill-down never
creates anchors. The script is `public/articles/agent-desktop/test_port.py`; it does not prove the port matches the
Rust everywhere, only on the cases the authors thought to write down.

<RefLab />

The widget runs that same rule on a three-level tree. Try the case the rule cannot cover: an unlabeled boundary with no
native id, no name, no rectangle and nothing above it. The output shows a `children_count` and no ref, so the agent can
see that something was cut and has no handle to open it. That is the honest edge of the design.

## Reading the tree on three operating systems

The walkers carry hard limits (**read**). On Linux a walk stops at a raw depth of 50, 100,000 edges,
4,096 children under one node and 64 KiB in one field, and it expands 8 nodes at a time over D-Bus. The node cap is 50,000 for a snapshot (an earlier version of this article said 10,000; see below). On Windows the raw depth
is also 50, one parent is capped at 10,000 siblings, and a constant named `BOUNDARY_COUNT_BUDGET` gives the child count at a depth cut 25 ms. The depth the agent asks for is a second, logical limit: the default is 10 levels, and web wrappers do not count
against it. A wrapper is a `group` with no name, value, description or id, which "consumes raw depth but no logical
depth". The glossary's reason is that skipping them "lets a dense web app fit a default depth budget; without the skip, the
same app understates its reachable content by an order of magnitude." On Windows that skip applies only to Chromium windows, found by the root window class
`Chrome_WidgetWin_1`; on macOS and Linux it applies to every app.

The core insists that every property an adapter reads comes back as `Known`, `Absent` or `Unknown`, never two of them
collapsed. `Absent` is an answer, and `Unknown` is a failed read that must not pass for one. The same care separates four
outcomes of a failed read: a settled absence that is never retried, a retryable transport failure, an element that has
gone, and a terminal failure. A tree whose enumeration fails is marked incomplete rather than reported whole. This is the
thing a computer-use tool most needs to get right and the one a quick wrapper usually skips, because a hung target that
returns a short tree looks exactly like a small app.

## Cold trees: Chromium and Electron

Web content builds its accessibility tree lazily, and the project measured how lazily. The Windows findings ledger has a row
on a cold Obsidian launch, an Electron app: first contact returns 12 descendants and the settled tree has 165, a 13.75 times
understatement, and settling took 10 to 25 seconds, longer than the default 3,000 ms snapshot deadline "in every measured
run" (**reported**, row A16-11). That is why the deadline is a constant with a comment, why `--timeout-ms` exists, and why
a snapshot that runs out of time now returns `ok: true` with `"complete": false`, the tree it did observe and
`subtree_truncated` on every cut node, instead of a `TIMEOUT` and nothing. A drill-down is stricter: it replaces refs
inside an existing snapshot, so it must see its whole subtree or leave the stored map alone.

Two mechanisms close the gap (**read**). On Linux the walker re-reads a filling web tree every 200 ms and calls it settled
after three consecutive reads with the same node count, spending at most 70% of the budget the observation started with.
For renderers that have not published a tree at all, the core switches accessibility on and retries with a delay that starts
at 25 ms and doubles to 400 ms, so a renderer that wakes quickly is caught quickly and one that never wakes is retried a
handful of times instead of continuously. On macOS the adapter sets `AXManualAccessibility` on Electron apps at most once and
then waits for the attribute to read back as ready, following Electron's own guidance.

## Skeleton, then drill

The default `snapshot` returns everything to depth 10. Two flags shrink it. `-i` drops leaves that cannot take a ref.
`--compact` collapses a `group` with one child and no payload. `--skeleton` caps the depth at 3, `args.max_depth.min(3)`,
and only when no `--root` is given. Then `--root @s8f3k2p9:e3` re-reads one region (**read**, `commands/snapshot.rs`,
`snapshot_ref.rs`).

The drill is more careful than its name suggests. It loads the stored snapshot, strictly re-resolves the root ref against the
live UI, observes only that subtree, removes every ref that was previously allocated under that root, allocates fresh ones
with the stored path as a prefix, and writes the result back under a file lock with a 2,000 ms timeout. Refs from the rest of
the snapshot stay valid. An agent can map a Slack-sized window cheaply, open one region, act, and re-read the same region to
verify.

<SnapshotModes />

I measured what each flag does on two synthetic trees built to match what the documents describe. One is a native-style file
manager: 521 nodes, a 120-row table, shallow, no wrapper chains. The other is an Electron-style chat app: 2,262 nodes, 150
messages, chains of anonymous wrappers, DOM ids on the messages. Tokens are counted with the Qwen3 tokenizer on the
compact JSON the CLI writes, plus the envelope. These are toys, not recordings; read the ratios.

- **The flags do nothing for a flat, wide app.** The file manager costs 15,854 tokens plain and 15,833 with every flag on.
  The skeleton cuts depth, not breadth, and 120 rows of three cells each are 391 refs however the depth is set. The
  project's own skill says as much: "A small app needs no skeleton."
- **They do a lot for a deep one.** The chat app goes from 38,070 tokens plain, to 31,480 with `--compact`, to 27,752 with
  `-i`, to 23,790 with both, to 8,708 with `--skeleton -i --compact`.
- **A drill is nearly free.** Re-reading the composer region costs 200 tokens and the channel header 139, and that number
  does not grow with the app.

The scaling is the point. Over 25 to 600 messages, a full `-i --compact` snapshot grows by about 149 tokens per
message (5,848 to 91,467), the skeleton by about 43 (3,346 to 28,058, one anchor per message container), and the
composer drill stays at 200 (all **measured** on the toy).

## What that costs against a screenshot

Computer-use agents that read the screen send an image. I did not measure image tokens here; I used the published rule of
(width x height) / 750, with a 1,568 px long-edge cap, which gives about 1,365 tokens for a 1280 x 800 capture and about 1,844
for a 1920 x 1080 frame resized (**reasoned**). On those numbers a full snapshot of a heavy page costs far more than the
picture it replaces: 23,790 tokens against 1,365 to 1,844.

The accessibility tree wins only if you read it in regions. A skeleton once, then a drill per observation, costs
8,708 + 200 k tokens for k observations, against 1,365 k for screenshots. The break-even is k of about 7.5, or 5.3 against the
larger image. Below that a screenshot is cheaper.

<LoopCost />

Three things that this model does not capture, and which matter more than the totals. A screenshot gives the model
coordinates and a guess; a ref gives an action that goes through the accessibility API and fails closed. The tree carries
state a picture does not, such as `checked`, `expanded`, `enabled` and the value of a field. And the tree is text from
inside the app, which is a different trust question, below. The repository itself ships a helper aimed at the cost. Its
`jev-desktop` skill describes keeping "a 150-element JSON tree out of your context" by posting a description of the screen on
every turn to a decision endpoint, TypeSafe's by default, so that a decision model of the kind covered in
[Where to use Jev](/articles/where-to-use-jev) picks the element. The helper lives in `scripts/jev/`, ships in no installed
package, needs an API key, and sends element roles, names, up to sixty characters of each value and the window title off the
machine (**read**, from its skill file). The native CLI calls no model.

## Acting without double-clicking by accident

Reading is half the job. The vocabulary the project defines for acting is the most carefully argued part of its
`CONCEPTS.md`, and it reads like a list of past incidents.

- **Strict ref resolution.** A stored ref re-identifies its element in graded tiers: the stored positional path first, then a bounded
  broad search, confirmed by native id first and then by stable text. Only a ref with no meaningful identity may resolve
  by unique geometry, and on Windows and Linux only when the stored rectangle has positive area. A missing, stale or
  ambiguous match is rejected instead of guessed. Text that merely mirrors a field's current value does not count as identity.
- **Window identity.** A window handle can be recycled after a window closes, so identity is the handle plus a
  process-generation token, and the check is repeated just before each write.
- **Auto-wait.** A ref action polls for the target to be visible, stable, enabled and unoccluded, for 5,000 ms by default
  (`type` adds 30 ms a character), and fails fast on terminal conditions such as an unsupported action or a policy
  refusal.
- **Delivery, not success.** Every error and action result carries a disposition: `delivery` is one of `unknown`,
  `not_delivered`, `delivery_uncertain`, `delivered_unverified` and `delivered_verified`, and `retry` is `safe` only for
  `not_delivered`. On Windows, `SendInput` reports success whether or not the input was blocked by integrity levels, so
  physical steps there report `delivered_unverified` and the tool detects the integrity boundary by comparing token
  integrity levels instead. The skill tells the agent to retry only when `retry` is `safe`.
- **Headless by default.** Ref actions use accessibility calls and do not take focus, move the pointer or synthesize keys.
  If no semantic path exists the action fails closed. `--headed` is an explicit opt-in, and the skill says never to switch
  to it to recover from a headless failure.
- **Secure fields.** A password field's value is withheld on reads, and writes into it never echo the attempted or observed
  value in steps, messages or post-state.

None of this is exotic on its own. What is unusual is that the failure semantics are specified per failure, which is what
an agent needs in order not to submit a form twice.

## Many agents on one desktop

The post claims many agents on one app at once. The mechanism has two layers, and the second is the limit.

Reading is concurrent. A snapshot lives in a namespace, the global one or a session (`--session`, or
`AGENT_DESKTOP_SESSION`), under a state root that defaults to `~/.agent-desktop`, and lookups are confined to the namespace
they were made in. Each process writes its own trace segment. A session can run in multi-agent mode, where each
`--agent-id` gets its own presentation cursor on macOS, Windows and Linux X11, while snapshots stay shared by session
(**reported**, `CONCEPTS.md`).

Acting is serialized. All input through the operating system is one pointer and one focus, so mutating commands take an
interaction lease first. On Unix it is a file lock at `/tmp/agent-desktop-<uid>/interaction.lock` plus a process guard,
with an inheritable file descriptor so a child process can adopt a held lease (**read**, `session/interaction_lease.rs`; I did
not read the Windows implementation). The ordering in `ref_action_wait.rs` is the interesting part: the auto-wait poll runs
without the lease, then the lease is acquired, then the ref is resolved again, because anything resolved while waiting was
observed without exclusivity. The comment calls that second resolution "the correctness boundary, not redundant work." The
result carries a contention count and the lease hold time, so an agent can see that it queued behind another.

So many agents can look at one app at once; they take turns to touch it.

## What running it showed

I ran the repository's own tests for the core crate, the one that holds the ref allocator, the snapshot logic and the
limits and has no platform dependencies: `cargo test -p agent-desktop-core` at `cb31212`, on the toolchain the repo pins
(Rust 1.89.0), Linux, four-core box, two build jobs. Logs and commands are in `public/articles/agent-desktop/verify/`.

- **The core tests pass, apart from two that cannot here (measured).** 1,515 of 1,517 passed in 20 seconds, including the
  repository's own `test_skeleton_fixture_matches_golden` and the drill-down refmap golden. The two failures are
  `private_write_rejects_an_intermediate_directory_symlink` and `unreadable_segment_is_skipped_with_warning`. Both rely
  on the operating system refusing access, and I ran as root, which it never refuses. They test file permissions, not
  the snapshot; I did not re-run them as an ordinary user.
- **The port holds against the real fixtures (measured).** Against the same `skeleton-tree.json` that the Rust test
  loads, all ten checks of my Python port pass. `run_tokens.py` re-run with Qwen3's `tokenizer.json` wrote a
  `results_all.json` byte-identical to the committed one, so every token figure in this article regenerates. I also checked the
  serialisation by reading `AccessibilityNode`: field names, the omission of empty fields and the snapshot-qualified
  `@<id>:e<N>` refs match the toy's output. The toy trees are still mine, so the ratios are not a recording of a
  real app.
- **Defaults confirmed (measured, by reading the constants and the tests that use them).** Depth 10 (`default_value =
  "10"` in `cli_args/snapshot_tree.rs`), a 3,000 ms snapshot deadline (`DEFAULT_SNAPSHOT_TIMEOUT_MS`), skeleton depth
  `min(3)` applied only without `--root`. Counts: 16 error codes, 60 command names with 56 operational as the README says.
  The Linux settle poll is 200 ms, three equal reads, 70% of the budget; the activation retry runs 25 ms to 400 ms; the
  Windows boundary count budget is 25 ms; nodes in flight on Linux are 8.
- **One claim was wrong.** I wrote that a Linux walk stops at 10,000 nodes. That constant (`DEFAULT_MAX_NODES` in
  `walk_limits.rs`) belongs to `WalkLimits::default()`, which the gate, resolution and search walks use. A snapshot builds
  its limits from the request's budget (`WalkLimits::for_request`), and `ObservationBudget::default()` allows 50,000
  nodes, 100,000 edges, 4,096 children per node and 64 KiB per field. So a snapshot of a very large Linux window gets five
  times the node allowance I stated; raw depth stays 50. This changes no token figure here, since the toy apps are
  smaller than either cap.

What still rests on reading: the walkers on real trees, the Chromium settle behaviour, and every latency number. Those
need a live accessibility bus.

## Evidence, and how often it contradicted the plan

Two directories of probe scripts and a findings ledger per platform explain why the design looks the way it does. Each row
records an experiment, what the project assumed before it, what happened and a verdict: `CONFIRMS`, `CONTRADICTS`,
`NEW-EDGE` or `DEFERRED`. Counting the rows my pattern matched (**measured**), the Linux ledger has 774, 93 of them
`CONTRADICTS` (12%) and 385 `NEW-EDGE`, and the Windows ledger has 231, 18 of them `CONTRADICTS` (8%).

One row is worth quoting for its shape. The project had taken it as given that batching property reads with a UI Automation
cache request is "3-5x faster". On a 220-node Explorer window with 8 properties it measured 2.69 times overall, below the claim,
and the row says the single number hides a split: building the cache makes the find pass slower, 180.25 ms against 117.38 ms,
while the read pass falls from 371.62 ms to 1.25 ms, about 298 times (**reported**, row A6-1). The verdict is `CONTRADICTS`,
and the claim was replaced with the decomposition. That is a project that writes down when its assumption was wrong.

The same culture shows in the checks: a gate must assert something to pass, and a fixture proves the gate still detects
what it exists to detect, so a check that quietly stopped finding anything fails instead of reporting a clean tree.

## What it can and cannot do to you

The native CLI does not need a model service, and the FAQ says plainly that snapshots, window titles, clipboard reads,
notifications, screenshots and session traces contain private desktop content. Debug HTML embeds a screenshot, and replay
artifacts are unredacted. Permissions differ by system: macOS needs the Accessibility grant, Windows needs none for apps at
your terminal's integrity level, and Linux needs the graphical session with `at-spi2-core` (**reported**, README). The
security policy asks for private vulnerability reports through GitHub, covering the CLI, installer, FFI, release artifacts and
adapters.

One risk is not in the repository's documents, so it is **reasoned**. The accessibility tree is text that the application, and
therefore anyone who controls its content, wrote. A web page's labels, a chat message and a window title all enter the model's
context through `snapshot` the same way. A tool that makes this easy to read does not make it safe to trust, and the
headless default, the delivery dispositions and the refusal to climb to physical input are useful brakes, but none of them
tells the agent that a button's label was written by a stranger. The sandbox articles on this site make the same point from
the other side: [Claude Code's sandbox from the source](/articles/claude-code-sandbox-internals) confines what a command can
reach, and nothing here confines what a screen can say.

## What I did not do

- I did not run the `agent-desktop` binary: it needs an AT-SPI bus and a graphical app to read, and neither exists here. The measured numbers about snapshot size are still my own, from the port and the toy trees; the core test run checks the allocator and the skeleton, not a real window.
- The toy trees are shaped by me. A real Slack or Finder window would change every absolute token count.
- Tokens are counted with one tokenizer; another model's would shift each number by a margin I did not measure. The screenshot
  cost is a published rule, not an API measurement.
- The macOS and Windows adapters were skimmed for structure and dependencies, not read function by function, and I did not read
  the Windows interaction lease or the FFI layer.
- The latency numbers, the Chromium settle times and the cache-request timing are the project's, from its Windows ledger.

The port and the measurements are in `public/articles/agent-desktop/`: `axport.py`, `gen_apps.py`, `run_tokens.py`,
`test_port.py` and `results_all.json`. `run_tokens.py` needs Qwen3's `tokenizer.json` and regenerates every table here
(I re-ran it from the shipped files and got identical numbers); `test_port.py` takes the path of the repository's
`tests/fixtures/skeleton-tree.json`.

## Sources

- [vercel-labs/agent-desktop](https://github.com/vercel-labs/agent-desktop) at `cb31212`: `README.md`, `CONCEPTS.md`,
  `docs/json-output.md`, `docs/faq.md`, `SECURITY.md`, `skills/agent-desktop/SKILL.md`, `probes/windows/FINDINGS.md`,
  `probes/linux/FINDINGS.md`, and under `crates/core/src`: `snapshot/`, `refs/`, `session/interaction_lease.rs`,
  `ref_action/ref_action_wait.rs`, `tree/renderer_accessibility.rs`.
- [@mdlahfir's launch post](https://x.com/mdlahfir/status/2116316228046500348).
- On this site: [Cua Spaces](/articles/cua-spaces) and [arc-driver](/articles/arc-cua) for other computer-use stacks,
  [Agent harnesses](/articles/agent-harness) for the loop around the model, and
  [Where to use Jev](/articles/where-to-use-jev) for the decision models the optional helper calls.