Screenshot Analysis
Documentsscreenshot-analysisScreenshot Analysis is an Agent Skill for documents published in the A-EVO-Lab/a-evolve repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
How to interpret accessibility tree elements and correlate them with screenshot regions for accurate GUI interaction.
QUICK START
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/A-EVO-Lab/a-evolve/blob/HEAD/seed_workspaces/osworld/skills/screenshot-analysis/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/screenshot-analysis, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Screenshot & Accessibility Tree Analysis
Reading the Accessibility Tree
The a11y_tree is your primary guide for finding interactive elements:
[role=window] "Document - LibreOffice Calc"
[role=menubar]
[role=menuitem] "File"
[role=menuitem] "Edit"
[role=toolbar]
[role=button] "Save" (x=120, y=45)
[role=table] "Sheet"
[role=cell] "A1" (x=80, y=200)
Coordinate mapping
- a11y_tree provides (x, y) center coordinates for each element
- Use these for pyautogui.click(x, y)
- If coordinates seem wrong, the element may be scrolled off-screen
Element roles cheat sheet
button: clickable action (Save, OK, Cancel)menuitem: menu entry (click to open submenu or execute)textbox/text: editable text field (click then type)combobox: dropdown (click to open, then select)checkbox: toggle (click to check/uncheck)cell: spreadsheet cell (click to select, then type)tab: tab selector (click to switch)dialog: modal window (must handle before other actions)
When a11y_tree is insufficient
- Some custom widgets don't expose a11y info — use screenshot + coordinates
- For canvas-based apps (GIMP): rely on menu navigation, not direct canvas clicks
- For web content in Chrome: a11y_tree reflects DOM, usually reliable
Tips
- Always check a11y_tree AFTER an action to verify the state changed
- If an element disappears from a11y_tree, the UI updated (menu closed, dialog dismissed)
- Count elements to verify you're clicking the right one (e.g., 3rd button in toolbar)