vercel-labs/agent-desktop v1.2.0 发布，新增 MCP 支持

Official release notes (https://github.com/vercel-labs/agent-desktop/releases/tag/v1.2.0) ## What's Changed * feat(macos): select list rows that only their container can select by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/503 * fix(macos): draw the cursor overlay above menus and system panels by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/511 * docs: simplify the README and repo docs by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/481 * docs: document the homebrew-core install by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/524 * feat: agent-desktop mcp by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/519 * fix(macos): ignore exited apps in the workspace inventory by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/371 * fix: preserve partial snapshots after observation expires by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/372 * feat: accept stdin for text commands by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/373 * fix: start the macos cursor overlay outside the caller's directory by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/381 * chore: release 1.2.0 with mcp support by @lahfir in https://github.com/vercel-labs/agent-desktop/pull/568 **Full Changelog**: https://github.com/vercel-labs/agent-desktop/compare/v1.1.0...v1.2.0 Repository README (current repository documentation, not release changes) # agent-desktop Computer use for AI agents: see and operate desktop apps on macOS, Windows, and Linux through the OS accessibility tree. agent-desktop gives an AI agent computer use without screenshots or pixel matching. It reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element the agent can act on. The agent clicks or types on a ref, reads the UI again to check the result, and repeats. The loop lives in your agent; agent-desktop has no model inside. It ships as a CLI and a C library. Snapshots, clipboard reads, screenshots, command output, and debug artifacts can contain private desktop content. Keep them out of public logs and uploads. ## Install ```bash npm install -g agent-desktop agent-desktop status # platform, permissions, supported surfaces agent-desktop permissions --guide # macOS: walks you through the Accessibility grant ``` Bun, Yarn and pnpm can install the same package. It includes native binaries and platform helpers for x64 and ARM64. For direct execution without a JavaScript launcher, download the matching native archive from [GitHub Releases](https://github.com/vercel-labs/agent-desktop/releases) and keep its helper beside the executable. On macOS and Linux, Homebrew also works: `brew install agent-desktop`. There is no official Scoop package yet. The repository no longer maintains a custom Homebrew tap or Scoop bucket. Release archives and building from source are covered in [Installation](https://agent-browser.dev/desktop/installation). ## Quick start ```bash agent-desktop snapshot --app Finder -i # interactive elements, each with a ref agent-desktop click @s8f3k2p9:e8 # act on a ref from that snapshot agent-desktop snapshot --app Finder -i # observe again to verify ``` Each command prints one JSON envelope. A trimmed snapshot: ```json { "version": "2.6", "ok": true, "command": "snapshot", "data": { "app": "Finder", "snapshot_id": "s8f3k2p9", "ref_count": 59, "complete": true, "tree": { "role": "window", "name": "Downloads", "children": [ { "role": "treeitem", "name": "Documents", "ref_id": "@s8f3k2p9:e8", "available_actions": ["Click"] } ] } } } ``` - Refs carry their snapshot id. A bare `@e8` fails with `INVALID_ARGS`. - If the UI changed, an action returns `STALE_REF` or `AMBIGUOUS_TARGET`. Take a new snapshot and use its refs. - Actions are headless by default: no pointer movement, no focus change. Pass `--headed` for physical input. - `type`, `set-value`, and `clipboard-set` accept `--stdin` for piped text, keeping it out of process arguments. Run secret steps as separate commands; batch JSON text appears in process arguments. - In PowerShell, quote every ref (`'@s8f3k2p9:e8'`), because a bare `@token` is the splatting operator. The full walkthrough is in the [quick start](https://agent-browser.dev/desktop/quick-start). ## How it works Your agent calls the CLI, or the C library in process. Both run the same core, which reaches the OS only through `PlatformAdapter`. Each binary is built for one OS and talks to that OS's accessibility API. ## For AI agents The binary carries guides that match its version: ```bash agent-desktop skills get desktop # the loop, refs, verification and retry rules agent-desktop skills get platform # what differs on the OS you run on agent-desktop --help # flags for one command ``` Install the skill into an agent harness from [skills.sh](https://skills.sh/vercel-labs/agent-desktop/agent-desktop): ```bash npx skills add https://github.com/vercel-labs/agent-desktop --skill agent-desktop ``` Or add this to your `AGENTS.md`: ```markdown Use `agent-desktop` to see and operate desktop apps. Before the first use, run `agent-desktop skills get desktop` and `agent-desktop skills get platform`, and follow them. ``` Retry a failed command only when `error.disposition.retry` is `safe`. See [JSON output](https://agent-browser.dev/desktop/json-output) for the envelope, error codes, and exit codes. ### MCP `agent-desktop mcp` serves every command as an MCP tool over stdio. A tool is named after its command (`desktop_snapshot`, `desktop_click`, `desktop_session_start`), takes that command's arguments, and returns its JSON envelope: ```json { "mcpServers": { "agent-desktop": { "command": "agent-desktop", "args": ["mcp"] } } } ``` Global flags given to `mcp`, such as `--session`, `--headed` or `--trace`, apply to every call. ## Commands agent-desktop has 60 command names, 56 operational commands. The other four (`key-down`, `key-up`, `mouse-down`, `mouse-up`) refuse to run until a daemon can release a held key or button. [Commands](https://agent-browser.dev/desktop/commands) documents each one. | Group | Commands | |---|---| | Observe | `snapshot`, `find`, `get`, `is`, `screenshot`, `list-surfaces` | | Act on a ref | `click`, `double-click`, `triple-click`, `right-click`, `type`, `set-value`, `clear`, `focus`, `select`, `toggle`, `check`, `uncheck`, `expand`, `collapse`, `scroll`, `scroll-to` | | Keyboard and mouse | `press`, `hover`, `drag`, `mouse-move`, `mouse-click`, `mouse-wheel`, `key-down`, `key-up`, `mouse-down`, `mouse-up` | | Apps and windows | `launch`, `close-app`, `list-apps`, `list-windows`, `list-displays`, `focus-window`, `resize-window`, `move-window`, `minimize`, `maximize`, `restore`, `open-system-surface` | | Notifications | `list-notifications`, `dismiss-notification`, `dismiss-all-notifications`, `notification-action` | | Clipboard | `clipboard-get`, `clipboard-set`, `clipboard-clear` | | Wait | `wait` | | Sessions | `session`, `trace`, `cursor-overlay` | | System | `status`, `permissions`, `version`, `batch`, `skills` | ## Platform support | Platform | Requirement | Accessibility API | |---|---|---| | [macOS](https://agent-browser.dev/desktop/platforms/macos) | macOS 13 or later, Intel or Apple Silicon | AX API | | [Windows](https://agent-browser.dev/desktop/platforms/windows) | Windows 10 1809 or later, x64 or ARM64 | UI Automation | | [Linux](https://agent-browser.dev/desktop/platforms/linux) | Ubuntu 24.04 or later, x86_64 or aarch64, X11 or Wayland | AT-SPI | Each platform page lists what works headless, what needs `--headed`, and what refuses. ## C library Each release ships `libagent_desktop_ffi`, so a program can call agent-desktop in process. See the [C library docs](https://agent-browser.dev/desktop/ffi). ```python import ctypes lib = ctypes.CDLL("./lib/libagent_desktop_ffi.dylib") lib.ad_adapter_create.restype = ctypes.c_void_p lib.ad_adapter_destroy.argtypes = [ctypes.c_void_p] assert lib.ad_init(7) == 0 # AD_RESULT_OK when the header's AD_ABI_VERSION_MAJOR matches the library adapter = lib.ad_adapter_create() lib.ad_adapter_destroy(adapter) ``` ## Development Each platform adapter builds only on its own OS, so scope cargo to the host's packages. On macOS: ```bash HOST_PKGS=(-p agent-desktop-core -p agent-desktop-macos -p agent-desktop -p agent-desktop-ffi) cargo test "${HOST_PKGS[@]}" cargo clippy "${HOST_PKGS[@]}" --all-targets -- -D warnings ``` On Windows, use `agent-desktop-windows` instead of `agent-desktop-macos`; on Linux, `agent-desktop-linux`. [CONTRIBUTING.md](CONTRIBUTING.md) lists every gate. ## Documentation - [Introduction](https://agent-browser.dev/desktop): what it is and how it works - [Snapshots and refs](https://agent-browser.dev/desktop/snapshots-and-refs) - [Errors and recovery](https://agent-browser.dev/desktop/errors-and-recovery) - [Skills](https://agent-browser.dev/desktop/skills) - [CONCEPTS.md](CONCEPTS.md): vocabulary - [CONTRIBUTING.md](CONTRIBUTING.md), [SECURITY.md](SECURITY.md), [SUPPORT.md](SUPPORT.md) ## License Apache-2.0