---
name: computer-use-guidance
version: 1.2.0
description: "OS-level operation guidance for Computer Use automation. Provides platform-specific instructions for common desktop tasks on macOS and Windows, including app management, window control, file operations, system settings, terminal, office apps, and dev tools."
description_zh: "Computer Use 桌面自动化的操作系统级操作指南。提供 macOS 和 Windows 平台的常见桌面操作指引，涵盖应用管理、窗口控制、文件操作、系统设置、终端、办公软件和开发工具。"
---

# Computer Use Guidance

This skill provides OS-level operation guidance for desktop automation via Computer Use tools (`screenshot`, `click`, `type`, `key`, `scroll`, etc.). It helps you perform common desktop tasks correctly and efficiently on macOS and Windows.

## Tool Selection Strategy

Before resorting to Computer Use (GUI automation), always consider whether a more reliable and efficient tool is available. Follow this priority order:

| Priority | Tool Type | When to Use | Examples |
|----------|-----------|-------------|---------|
| 1st | **Built-in Tools** | File I/O, code editing, shell commands, git operations | `Read`, `Write`, `Edit`, `Bash`, `Glob`, `Grep` |
| 2nd | **MCP Tools** | Structured API interactions with specific services | Database queries, API calls, service-specific operations |
| 3rd | **Browser Tools** | Web page interaction with DOM-level precision | `browser-use` for navigating websites, filling forms, extracting web content |
| 4th | **Computer Use** | GUI-only operations with no programmatic alternative | System settings, native app UIs, drag-and-drop, visual verification |

**Key principles:**

- **Prefer programmatic over visual.** If a task can be done via CLI, API, or built-in tool, do NOT use Computer Use. For example, use `Bash` to run `open -a Safari https://example.com` instead of clicking through the Dock.
- **Prefer Browser Tools over Computer Use for web tasks.** Browser tools operate at the DOM level with reliable element selection; Computer Use relies on screenshot-based coordinate guessing which is fragile for web pages.
- **Use Computer Use for what only it can do.** Native OS dialogs, system preferences, desktop app UIs without automation APIs, multi-app workflows that span the entire desktop — these are where Computer Use shines.
- **Combine tools when appropriate.** A task might start with `Bash` to open an app, then switch to Computer Use for GUI interactions within that app.
- **When launching desktop GUI apps from shell, prefer non-blocking launch patterns.** On Windows especially, avoid foreground launches that can keep the tool call open until the app exits. Prefer detached launch forms such as `Start-Process` or `cmd /c start "" <app>`, then continue with Computer Use after a short `wait` and screenshot verification.

## General Best Practices

### 1. Screenshot First

Always take a screenshot (`screenshot`) before performing any operation to confirm the current screen state. Never guess coordinates or UI element positions.

### 2. Auto-Screenshot After Action

Action tools (`click`, `mouse_move`, `drag`, `type`, `key`, `scroll`, `wait`) **automatically return a screenshot** in their result, so you don't need to call `screenshot` separately after each action.

**Default behavior:**
1. **Operate**: Execute the action
2. **Wait**: The tool automatically waits `screenshotDelay` ms (default 300ms) for the UI to respond
3. **Verify**: A screenshot is captured and included in the result — check it immediately

**Controlling the delay:**
- Default `screenshotDelay` is 300ms — good for most button clicks, menu opens, and dialog appearances
- Increase it for slow transitions: `click({..., screenshotDelay: 1000})` for page loads
- For very slow operations (app launches, large file downloads), use `wait({ms: 3000})` instead — `wait` automatically returns a screenshot when done

**Rapid action sequences** — use `returnScreenshot: false` to skip intermediate screenshots:
```
click({..., returnScreenshot: false})         // Click input field (no need to see)
type({text: "hello", returnScreenshot: false}) // Type text (no need to see)
key({key: "enter"})                            // Press Enter (keep screenshot to verify)
```

**When to use the standalone `screenshot` tool instead:**
- **Initial exploration**: First look at the screen before any action
- **Zoom inspection**: `screenshot({zoomCenter: [x, y], displayId: 0})` for detail
- **High quality**: `screenshot({quality: "high"})` when you need sharper images
- **All displays**: `screenshot({captureAllDisplays: true})` for multi-monitor overview

### 3. Coordinate Accuracy (Normalized Coordinate System)

Screenshots are pre-normalized to a standard width (~1280px, maintaining aspect ratio). This means coordinate resolution is consistent regardless of the physical display resolution.

**How it works:**

1. Take a `screenshot` — the returned image is already normalized
2. Identify the target pixel position `[x, y]` directly in the screenshot image
3. Pass `coordinate: [x, y]` and `displayId` (from screenshot metadata) to `click`, `mouse_move`, or `drag`
4. The system automatically converts screenshot coordinates to screen coordinates

**Key rules:**

- Always use coordinates from the **most recent** screenshot — never reuse old coordinates after the screen has changed
- Aim for the **center** of the target element, not its edge
- The `displayId` is **required** for click/mouse_move/drag — get it from the screenshot's `【Current Display (Screenshot)】` metadata
- Do NOT attempt manual coordinate conversion; the system handles Retina scaling, display offsets, and normalization automatically
- When a UI element is too small to identify precisely, use `zoomCenter` on `screenshot` to zoom in for visual inspection (see "Zoom for Detail" below)
- Screenshots include a red crosshair marker at the current cursor position by default — use this to verify whether a click landed on the intended target. Pass `hideCursor: true` only if the marker interferes with visual inspection

### 4. Multi-Display Awareness

When multiple displays are connected, always confirm which display you are operating on before clicking:

- Use `list_displays` to get each display's ID, resolution, bounds, normalization size, and primary flag
- In `screenshot` text output, check:
  - `【Current Display (Screenshot)】` for the captured display's ID and name
  - `【Coordinate System】` for the normalized image dimensions and displayId to use
  - `【All Available Displays】` for the full monitor layout
  - `【Current Cursor Position】` for cursor global coordinates and current display ID
- Use `cursor_position` when you need a precise current `{x, y, displayId}` check before operation
- If you need a full desktop overview in one step, call `screenshot` with `captureAllDisplays=true` (returns one image per display, each with its own displayId)
- If a target UI element seems "missing", first check whether it is actually on another display by using `captureAllDisplays=true`
- **Drag operations** use a single `displayId` — both start and end points must be on the same display

### 5. Zoom for Detail

When a UI element is too small to identify precisely in the normalized screenshot, use `zoomCenter` to zoom in:

1. Take a normal `screenshot` first
2. Identify the pixel coordinates of the area you want to inspect
3. Call `screenshot` with `zoomCenter: [x, y]` and `displayId` — coordinates are in the screenshot's pixel space
4. Optionally set `zoomLevel` (2-6, default 3) to control magnification
5. The system computes a crop region centered on that point and upscales it for visual inspection

**Important:** Zoomed images are for **visual inspection only**. Click coordinates must still use the **full screenshot** coordinate space, not the zoomed image. After zooming to confirm your target, use the coordinates from the original full screenshot for click/mouse_move/drag.

### 6. Input Method Awareness

Before typing text with `type`:
- Check if the correct input method is active (especially for CJK text)
- On macOS, use `Ctrl+Space` or `Cmd+Space` (depending on configuration) to switch input methods
- On Windows, use `Win+Space` or `Alt+Shift` to switch input methods
- When typing URLs, commands, or code, ensure the input method is set to English/ASCII

### 7. Error Recovery

When an operation doesn't produce the expected result (check the auto-returned screenshot):
1. Press `Escape` to dismiss any unexpected dialogs or menus
2. If the app is frozen, use `wait({ms: 3000})` — its auto-screenshot will show the current state
3. If stuck, try navigating back to a known state (e.g., click on desktop, open Spotlight/Start Menu)

### 8. Scrolling Strategy

- Use `scroll` with moderate amounts (2-3 units) to avoid overshooting
- The auto-returned screenshot shows the new viewport — check whether the target element is now visible
- For long pages, scroll in increments rather than large jumps

## Platform-Specific Guides

Based on the current operating system, refer to the corresponding guide for detailed operation instructions:

- **macOS**: Read [macos/common-operations.md](macos/common-operations.md) for macOS-specific keyboard shortcuts, app interactions, and system navigation
- **Windows**: Read [windows/common-operations.md](windows/common-operations.md) for Windows-specific keyboard shortcuts, app interactions, and system navigation

Each guide covers 9 categories of common operations:
1. App Management — launching, switching, and closing applications
2. Window Management — resizing, splitting, and organizing windows
3. File Operations — navigating, copying, moving, and managing files
4. System Settings — accessing and navigating OS settings
5. Text Editing — universal editing shortcuts and text selection
6. Browser Operations — tab management, navigation, and dev tools
7. Terminal / Command Line — opening terminal, running commands, shell operations
8. Office Software — Word, Excel, PowerPoint common operations
9. Dev Tools — VSCode, Xcode/Visual Studio IDE operations
