Phone Use
Apps & Automationphone-usePhone Use is an Agent Skill for apps & automation published in the pockebot/openpocket repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Use OpenPocket MCP tools to inspect or control an Android emulator, physical Android phone, Android TV, or ADB-backed app. Trigger for phone use, Android automation, mobile UI testing, emulator control, physical-device control, app navigation, screenshots, tapping, typing, and requests that should not use desktop computer-use automation.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/pockebot/openpocket/blob/HEAD/plugins/openpocket-phone/claude/openpocket-phone/skills/phone-use/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/phone-use, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Phone Use With OpenPocket
Use the plugin-provided openpocket-phone MCP server for Android work. Do not use desktop computer-use automation to operate an emulator window unless the MCP server is unavailable and the user explicitly accepts that fallback.
OpenPocket is Android-first. This plugin does not control iOS Simulator or iPhone targets.
Start Every Task
- Confirm the target belongs to the user or is authorized for testing.
- Call
target_statusbefore taking any action. - If the configured target is an emulator and no booted emulator is online, call
start_emulatorwhen starting it will not disrupt other local work. - If multiple devices are online, ask which serial to use and pass that
deviceIdto every subsequent tool. - Never bypass Android trust prompts, lock screens, account prompts, or OS security controls.
Interaction Loop
- Read state with
ui_snapshot,visible_text, orcurrent_app. - Open apps with
open_appwhen only the label is known, orlaunch_appwhen the package name is exact. - Locate controls with
find_textand prefertap_textortap_elementover raw coordinates. - After navigation, launch, search, or scrolling, use
wait_for_textinstead of repeatedly polling screenshots. - Use
screenshotwhen visual layout, imagery, canvas content, or uncertain text extraction matters. - Read the metadata returned by
screenshotorui_snapshot, includingcurrentApp,deviceId,uiElements,visibleTextLines,secureSurfaceDetected, capture metrics, and screen dimensions. - Use
type_textonly after the intended input field is focused. - Use
key_eventfor BACK, HOME, ENTER, SEARCH, and similar Android keys. - Use
swipe,drag, andlong_press_dragfor gestures. - Use
adb_shellonly for narrow Android inspection or deterministic setup. Avoid broad or destructive commands. - Re-read the screen after every state-changing action and stop when the user goal is complete or the state becomes ambiguous.
Tool Guide
target_status: configured target type, online devices, booted devices, and resolved device.ui_snapshot: default text-only UI state with element metadata.visible_text: lightweight visible and accessibility text extraction.find_text: locate elements by text, content description, resource ID, or class.tap_text: tap a labeled control without coordinates.wait_for_text: wait for a known screen state.screenshot: visual evidence plus current app, text, elements, and capture metadata.open_app: discover and open an app by label or package.list_apps: list launchable app labels and package names.adb_shell: narrow Android inspection only.
Sensitive Boundaries
Pause for explicit user confirmation before:
- submitting a purchase or payment
- sending a message, post, like, follow, bid, or other irreversible social action
- changing account, security, privacy, or payment settings
- entering a password, OTP, 2FA code, recovery code, card detail, government ID, or private health or finance data
- using camera, microphone, photos, contacts, files, location, biometric, NFC, or SMS capabilities
If OpenPocket Human Auth is configured and the user asks for an approval-driven flow, prefer the OpenPocket human-auth path. Otherwise ask the user to provide the required data or approve the action in chat.
If a secure surface produces a black or incomplete screenshot, use returned UI metadata only. Do not infer hidden sensitive content.
Recovery
If tools are missing immediately after installation or update, restart the client and open a new task. Plugin-provided MCP tools are loaded per client task and an existing task may retain the previous tool surface.
Run node plugins/openpocket-phone/scripts/doctor.mjs from an OpenPocket checkout to validate both host bundles. If a UI element disappears, capture a fresh snapshot before trying again. If a physical device is unavailable, ask the user to authorize USB debugging or Wi-Fi ADB first.