Claude Code Computer Use: Anthropic JUST DROPPED THE BIGGEST Upgrade to Claude Code YET!
Anthropic has added computer use functionality to Claude Code, allowing it to control Mac apps through GUI interaction, clicking buttons, and taking screenshots. This enables Claude to write code, compile it, test the app visually, find bugs, and fix them all in one workflow without human intervention for GUI-based tasks.
Summary
Anthropic has released a major update to Claude Code by adding computer use capabilities, which allows Claude to interact with native Mac applications through GUI control rather than being limited to terminal commands and file operations. This feature enables Claude to open apps, click through interfaces, type, scroll, inspect visual elements, and take screenshots to understand what's happening on screen. The system is designed as a fallback tool that only activates when more precise methods like MCP servers, bash commands, or browser automation aren't applicable, making it ideal for testing native apps, simulators, and GUI-only tools that lack proper APIs or CLI interfaces. The feature includes several safety measures: it's disabled by default and must be manually enabled per project, requires app-by-app approval for each session, implements different control tiers based on app sensitivity (view-only for browsers, click-only for terminals, full control for others), and includes warnings for sensitive applications. While Claude controls the computer, other apps are hidden, only one Claude session can have control at a time, and users can stop the process anytime by pressing escape or Ctrl+C. The feature is currently limited to macOS as a research preview, requires Claude Pro or Max plans, needs authentication through claude.ai, and only works in interactive mode. Despite these limitations, it addresses a significant gap in the development workflow by handling the 'last mile' verification tasks like visual bug reproduction, UI testing, and simulator interaction that previously required manual intervention.
Key Insights
- Computer use serves as the broadest but slowest tool that Claude only uses as a fallback when more precise tools like MCP servers, bash commands, or browser automation don't apply
- The system implements app-by-app, session-by-session approval with different control tiers - browsers get view-only access, terminals get click-only access, and other apps get full control
- Most annoying bugs in development are not in the code generation step but in the verification step, where code compiles but has visual issues like wrong spacing, unresponsive buttons, or slow loading screens
Topics
Full transcript available for MurmurCast members
Sign Up to Access