OpenParable
Skip to content
← Back to skills

Browser Harness Skill

Apps & Automation
browser-harness-skill

Browser Harness Skill is an Agent Skill for apps & automation published in the zeenie-ai/OpenCompany repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Drive the user's real Chrome over raw CDP by writing Python against browser-harness helpers - screenshot, coordinate clicks, JS evaluation, form fill, tabs. For tasks needing full freedom or the user's own logins.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/zeenie-ai/OpenCompany/blob/HEAD/server/skills/web_agent/browser-harness-skill/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/browser-harness-skill, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Browser Harness Skill

Controls the user's real Chrome through raw CDP (browser-use/browser-harness). Unlike the browser tool (accessibility tree + @eN refs), this tool is vision-first: you look at screenshots and click coordinates, with js() for reading the DOM. Use it when you need the user's logged-in browser, sites that fight automation frameworks, or interactions the structured tool can't express.

Core Workflow

The see -> act -> verify loop, written as Python in operation: run_python:

  1. capture_screenshot() to SEE the page (the returned path is an image you can view)
  2. click_at_xy(x, y) at coordinates you identified from the screenshot
  3. wait_for_load() after anything that navigates
  4. capture_screenshot() again to verify the result
  5. js(...) when you need exact text/attributes instead of pixels
{"operation": "run_python", "code": "ensure_real_tab()\ngoto_url('https://example.com')\nwait_for_load()\nprint(page_info())"}

browserHarness Tool

FieldTypeRequiredDescription
operationstringYesrun_python (primary), goto, screenshot, js, tabs, doctor
codestringrun_pythonPython executed with helpers pre-imported. Print a JSON object as the LAST line for structured output
urlstringgotoURL to open
expressionstringjsJavaScript to evaluate in the page
full_pageboolscreenshotCapture the full scrollable page (default: false)
timeoutintnoScript timeout in seconds (default: 60)

Helper Reference (inside run_python)

All pre-imported — never import them.

Navigation & tabs

  • goto_url(url) — navigate current tab
  • new_tab(url="about:blank") / close_tab(target=None) / switch_tab(target)
  • list_tabs(include_chrome=True) — enumerate open tabs
  • current_tab() / ensure_real_tab() — make sure a real page tab is focused (call first!)
  • page_info() — {url, title, w, h, sx, sy, pw, ph} (viewport + scroll + page size)
  • iframe_target(url_substr) — target for js/interaction inside an iframe

Seeing

  • capture_screenshot(path=None, full=False, max_dim=None) — returns the saved PNG path. Use max_dim=1200 to keep images small

Acting

  • click_at_xy(x, y, button="left", clicks=1) — coordinates from the screenshot
  • type_text(text) — types into the focused element (click the field first)
  • fill_input(selector, text, clear_first=True) — CSS-selector form fill
  • press_key(key, modifiers=0) / dispatch_key(selector, key="Enter")
  • scroll(x, y, dy=-300, dx=0) — scroll at a point (negative dy = down varies; verify with page_info)
  • upload_file(selector, path)

Reading & escape hatches

  • js(expression, target_id=None) — evaluate JS, returns the value. Your precision tool for text, attributes, element rects
  • cdp(method, session_id=None, **params) — raw CDP call when nothing else fits
  • http_get(...) — plain HTTP fetch without the browser

Waiting

  • wait_for_load(timeout=15.0) — after navigation/clicks that navigate
  • wait_for_element(selector, ...) / wait_for_network_idle(...) / wait()

Patterns

Find something on a page (screenshot first, then read precisely):

ensure_real_tab()
goto_url("https://news.ycombinator.com")
wait_for_load()
titles = js("[...document.querySelectorAll('.titleline > a')].slice(0,5).map(a => a.textContent)")
import json  # stdlib imports are fine; helpers are pre-imported
print(json.dumps({"top5": titles}))

Click something you can see (get coordinates from the element itself, not by eye):

r = js("(() => { const el = [...document.querySelectorAll('a')].find(a => a.textContent.includes('More')); const b = el.getBoundingClientRect(); return {x: b.x + b.width/2, y: b.y + b.height/2}; })()")
click_at_xy(r["x"], r["y"])
wait_for_load()
print(page_info())

Fill a login/search form:

fill_input("input[name=q]", "opencompany")
dispatch_key("input[name=q]", "Enter")
wait_for_load()
print(page_info())

Rules

  • Always start with ensure_real_tab() in the first call of a session — the daemon may be attached to a chrome:// or extension target.
  • Print a JSON object as the final line — the tool parses it into result for you; everything else lands in output.
  • Prefer js()-computed coordinates (getBoundingClientRect) over guessing pixels from a screenshot; screenshots confirm state, JS locates precisely.
  • One logical step per call. Short scripts fail cleanly; long ones lose all progress on any exception.
  • Screenshots return a file path under the harness tmp dir — mention the path in your reply if the user should look at it.
  • If a call fails with a Chrome-connection error, run {"operation": "doctor"} and relay its report: the user may need to start Chrome with --remote-debugging-port=9222 or allow remote debugging via chrome://inspect.
  • The harness drives one shared browser — do not assume parallel calls get isolated windows. State (cookies, logins, open tabs) persists across calls by design.
  • This is the user's REAL browser: never touch logged-in accounts, submit forms, or make purchases beyond what the task explicitly asks.

When to use browser instead

Prefer the structured browser tool when the site is simple and the accessibility tree works — its @eN snapshot loop is cheaper (no screenshots) and more deterministic. Come to browserHarness for: the user's own Chrome profile/logins, canvas/shadow-DOM-heavy or bot-hostile sites, iframes needing iframe_target, or anything requiring raw CDP.