Computer Use
AI that can see a screen and control a computer — mouse, keyboard, and all.
Computer Use (introduced by Anthropic in October 2024 with Claude 3.5 Sonnet) is the capability for an AI agent to interact with a computer interface — viewing screenshots, moving the cursor, clicking buttons, typing text, and navigating GUIs — just as a human would.
How it works: The agent receives a screenshot of the screen as a vision input, decides what action to take (move mouse to coordinates, click, type), executes that action via tool calls, observes the resulting screenshot, and repeats.
Key use cases:
- Automating legacy software with no API (filling forms, extracting data from closed systems)
- Browser automation for complex multi-step web tasks
- Testing UI workflows autonomously
- Completing tasks across multiple applications in sequence
Current limitations (mid-2026):
- Slower than API-based automation (~1 action/second vs. milliseconds for APIs)
- Visual recognition can fail on small UI elements or unusual layouts
- Non-deterministic — the same task may take different paths across runs
- Requires careful sandboxing to avoid unintended system actions
Implementations: Claude's Computer Use API, OpenAI's Operator, Microsoft's UFO (Windows automation), and various open-source wrappers.
In plain terms
A remote desktop session, except the person operating the keyboard is an AI agent rather than a human on the other end.
Related concepts
AI Agents
AI that plans and takes actions without you guiding every step.
Function Calling
Letting an AI invoke real code and APIs mid-reasoning.
Multimodal AI
AI that understands and generates more than one type of media — text, images, audio, or video.
Prompt Injection
A security attack where malicious instructions hidden in content hijack an AI agent.