OpenParable
Skip to content
← Back to skills

open-computer-use

Apps & Automation

Platform-neutral guidance for using Open Computer Use, the open-source Computer Use MCP server and CLI for macOS, Linux, and Windows. Use when an agent needs to install, verify, troubleshoot, configure, or operate Open Computer Use through its native CLI, stdio MCP server, or direct Computer Use tool calls.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/QwenLM/open-computer-use/blob/HEAD/skills/open-computer-use/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/open-computer-use, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Open Computer Use

Overview

Open Computer Use exposes Computer Use as a local CLI and stdio MCP server, usable by any agent runtime.

It supports the same core tool surface across macOS, Linux, and Windows: list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.

Core Workflow

  1. Check the CLI is installed with open-computer-use -h. If installation or setup is missing, read references/installation.md.
  2. On macOS, run open-computer-use doctor before the first real GUI task. If permissions are missing, ask the user to approve Accessibility and Screen Recording in the onboarding UI.
  3. Inspect available apps before acting: open-computer-use call list_apps.
  4. Capture current UI state with open-computer-use call get_app_state --args '{"app":"TextEdit"}'.
  5. Prefer element-targeted actions using element_index from the latest get_app_state result.
  6. For multi-step CLI work, use open-computer-use call --calls '<json-array>' so one process can reuse the latest element index mapping.
  7. For agent runtimes that support local MCP servers, configure open-computer-use mcp and call the exposed Computer Use tools directly. Read references/usage.md.
  8. If communication, permission, or desktop-session access fails, read references/troubleshooting.md.

Operating Rules

  • Treat the target desktop as the user's real session. Do not inspect password managers, unrelated private content, or sensitive apps unless the user explicitly asked for that task.
  • Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes.
  • Always use the installed open-computer-use CLI or an explicit MCP config; do not assume host-specific plugin helpers exist.
  • Always run get_app_state before using element_index; do not guess indexes across sessions or after large UI changes.
  • Prefer semantic actions and set_value for editable controls. Use coordinate click, scroll, and drag only when the element tree does not expose a safer target.
  • On macOS, do not enable OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 unless the user explicitly wants diagnostic behavior that may move the real pointer.
  • On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.

Common CLI Actions

open-computer-use -h
open-computer-use doctor
open-computer-use call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call click --args '{"app":"TextEdit","element_index":0}'
open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'

For a short sequence that reuses state in one process:

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

MCP Usage

For runtimes that can launch local MCP servers over stdio, use:

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

Read references/usage.md for JSON config examples, direct tool-call patterns, and platform notes.

References