智能体
dev.to ·
@Terminal Chai
Cua：面向 AI Agent 的开源 Computer-Use 基础设施与驱动
文章介绍开源平台 Cua（trycua/cua），目标是解决 computer-use agent 的工程障碍：OS 级 GUI 驱动脆弱、无头环境缺少显示服务、agent 常抢占开发者物理鼠标。作者将其定位为“Computer-Use 2.0”，即 agent 在同一任务中在 shell 脚本、REST API 与图形应用之间切换。
AI推荐理由 介绍开源项目 Cua（trycua/cua），针对 computer-use agent 的 OS 级 GUI 驱动、无头环境显示服务、避免抢占物理鼠标等具体工程痛点，对 Agent 开发者有实际参考价值；但正文为项目推广式叙述，缺少性能数据、版本与实测证据，需自行验证。
72
正文
The landscape of AI-assisted software development is undergoing a fundamental evolution. For the past two years, AI coding assistants have operated almost entirely inside terminal buffers and text editors.
However, practical software engineering tasks frequently require interacting with graphical user interfaces: verifying layout behavior in browsers, inspecting spreadsheet data, configuring desktop virtualization tools, or triaging issues across native desktop applications.
This evolution is known as Computer-Use 2.0—a paradigm where an AI agent dynamically navigates between executing shell scripts, calling REST APIs, and operating graphical applications within a single task lifecycle.
Until now, developers building computer-use agents faced severe hurdles: OS-level GUI drivers were brittle, headless environments lacked display servers, and agents frequently hijacked the developer's physical mouse cursor.
Cua (trycua/cua) is an open-source infrastructure platform designed from the ground up to solve these challenges.
Here is a comprehensive breakdown of Cua's architecture, its cross-OS desktop drivers, sandboxed cloud fleets, and how it connects to modern AI agents.
The Architecture of Cua
Cua does not attempt to be another monolithic AI agent. Instead, it acts as the operating system and automation layer that any AI agent or model can plug into.
┌─────────────────────────────────────────────────────────────┐
│ AI Agents & Models (Claude Code, Cursor, Codex, CUA-S1) │
└──────────────────────────────┬──────────────────────────────┘
│ (CLI / MCP / Typed SDKs)
▼
┌─────────────────────────────────────────────────────────────┐
│ Cua Driver │
│ (Background event delivery for macOS, Windows, Linux) │
└──────────────────────────────┬──────────────────────────────┘
│
┌────────────────────┼────────────────────┐
▼ ▼ ▼
[ Cua Fleets ] [ Lume VMs ] [ Local Desktop ]
(Isolated Cloud) (Apple Silicon VM) (Uninterrupted BG)
Enter fullscreen mode
Exit fullscreen mode
Cua is organized into five tightly integrated modules:
1. Cua Driver: Background Desktop Automation
Traditional OS-level automation tools (like PyAutoGUI) simulate physical hardware events. The moment an agent begins interacting with an app, your mouse jumps across the display, windows pop into the foreground, and you cannot type without interrupting the agent.
Cua Driver introduces native OS-level integration across macOS, Windows, and Linux that supports background delivery. Where supported by the platform, the agent can click buttons, type values, and inspect native windows without moving your mouse pointer or stealing focus from your active code editor.
2. Cua Fleets: Ephemeral Cloud Desktops
For untrusted agent execution or scalable evaluation runs, running actions on your local physical workstation is undesirable.
Cua Fleets allows developers to provision isolated, sandboxed Linux cloud desktops on-demand. Using the Sandbox SDK, an agent can:
Claim a desktop instance from a pre-warmed pool.
Execute bash commands and inspect files.
Launch graphical software (LibreOffice, Chrome, Inkscape).
Capture high-resolution viewport screenshots and clean up resources automatically.
3. CUA-S1: Fast "System 1" Decision Models
General-purpose LLMs (like Claude 3.5 Sonnet or GPT-4o) are incredible at high-level reasoning, but using them token-by-token to decide which form input to click is slow, expensive, and wasteful.
Cua introduces CUA-S1, a family of compact, specialized models designed for fast, bounded interface decisions. The initial research model (CUA-S1-FORMS) scores UI element decisions directly from structured accessibility trees and document elements, providing sub-second decision speed at a fraction of the compute cost.
4. Lume: Native Apple Silicon VM Management
Running local macOS VMs has historically required cumbersome third-party software. Lume is a lightweight CLI tool that manages local macOS and Linux virtual machines on Apple Silicon using Apple's native Virtualization.Framework. Developers can spin up clean macOS Tahoe VMs directly from Apple restore images and connect via SSH in seconds.
5. Cua Bench: Standardized Evaluation & Trajectories
To build reliable computer-use agents, engineers need reproducible benchmarks. Cua Bench provides a framework to define desktop tasks, run reference solutions, verify task completion with evaluators, and export step-by-step trajectories to train future models.
Installing and Using Cua Driver
Cua Driver can be installed via a one-line installer across all major operating systems:
macOS & Linux:
/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
Enter fullscreen mode
Exit fullscreen mode
Windows (PowerShell):
irm https://cua.ai/driver/install.ps1 | iex
Enter fullscreen mode
Exit fullscreen mode
Connecting to AI Coding Agents
If you are already using Claude Code, Codex, Cursor, or Antigravity, you can instruct your agent to install and integrate Cua automatically:
Set up browser-skill and cua-driver on this machine by following https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/README.md
Enter fullscreen mode
Exit fullscreen mode
Once installed, agents can interact with native desktop apps using clean CLI commands or through Model Context Protocol (MCP) server endpoints.
Example: Running a Benchmark Task with Cua Bench
You can run automated computer-use tasks without spinning up heavy cloud infrastructure using Python 3.12+ and uv:
# Install Cua Bench with browser support
uv tool install 'cua-bench[browser]'
uv tool run --from 'cua-bench[browser]' playwright install chromium
# Run and verify a reference task
cua-bench run --task example-task
Enter fullscreen mode
Exit fullscreen mode
The evaluator verifies agent interaction, checks UI state transitions, and outputs a normalized reward score along with action logs.
Why Cua Matters for the Agent Ecosystem
As agents progress from simple code-generation scripts to comprehensive engineering partners, they require the full breadth of computing capabilities available to humans.
By providing cross-platform background drivers, sandboxed cloud fleets, and specialized decision models under an open-source MIT license, Cua provides the foundational infrastructure layer needed to make Computer-Use 2.0 a reliable reality.
GitHub Repository: https://github.com/trycua/cua
License: MIT
Documentation & Cloud Fleets: https://run.cua.ai