Desktop UI Verify
Testing & Qualitydesktop-ui-verifyDesktop UI Verify is an Agent Skill for testing & quality published in the open-ani/animeko repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Validate the Animeko desktop (Compose Desktop) app by building and launching the real executable, then screenshot/click/type against the macOS window. Can also record the window and diff frames to catch sub-second transient glitches (flicker, jump-and-revert bugs). Use for desktop-only code paths, JCEF/browser behavior, VLC/video playback integration, native libraries, desktop packaging, window chrome, or when asked for PC screenshots/evidence. For Android emulator verification (interactive taps/swipes, Android screenshots, wide-screen simulation) use .agents/skills/android-ui-verify instead.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/open-ani/animeko/blob/HEAD/.agents/skills/desktop-ui-verify/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/desktop-ui-verify, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Animeko Desktop UI Verification
Use this skill from the ani repo root when a change needs desktop runtime evidence. Anything Android — including PC-style wide-screen layout checks, which run fine on an Android emulator — belongs to .agents/skills/android-ui-verify; only desktop-only code paths (Compose Desktop behavior, packaging, JCEF, VLC/native libraries, window chrome) need this skill.
The toolbox script is:
DESK=.agents/skills/desktop-ui-verify/scripts/desk.sh
The core loop mirrors the Android skill: act → screenshot → Read the PNG → decide next action. Never claim desktop behavior works without a captured window screenshot.
1. Build
Two launch modes:
- Dev launch:
./gradlew :app:desktop:runstarts the app from Gradle withapp/desktop/test-sandboxas working directory. Quick, but misses packaging problems. - Executable validation (preferred for evidence): build and launch the packaged
.app— catches runtime-image/JCEF/native-library packaging problems thatrunmisses.
./gradlew :app:shared:compileKotlinDesktop # cheap compile-only gate first
$DESK build # createDistributable with a JCEF-capable JBR
buildrequires a JBR containing JCEF (jmods/jcef.jmod+lib/libjcef.dylib);find_jcef_jbr.shauto-selects one and exports it asANI_COMPOSE_JAVA_HOME. A full first build takes many minutes — run it in the background.- It builds with
-Pani.api.server=http://localhost:4394; for real API data start the repo-local server first (see../ani-api-serverskillanimeko-server-local-test-server) — desktop reaches it as plainlocalhost. - When backgrounding Gradle, do NOT pipe to
tail— the pipe replaces the build's exit code with tail's, so failures read as success. Use-q(quiet) and read the full output file, or check the artifact ($DESK app, mtime) before trusting a "successful" build. - On macOS the packed app needs the JBR's
Contents/Frameworks(CEF + jcef helpers) copied next to the packed runtime'sHome;app/desktop/build.gradle.ktsdoes this for bothcreateDistributableandcreateReleaseDistributable(fixed 2026-07-05 — before that, non-release distributables crashed ~4 s after launch with a SIGSEGV inlibjcef.dylib FindClass).
2. Launch and interact
$DESK launch # opens the newest built .app, waits, prints the process
$DESK screenshot # activates + resizes window to 1440x900, prints PNG path — ALWAYS Read it
$DESK click 300 200 # window-relative POINTS — on retina, screenshot PNG pixel / 2
$DESK type hello # ASCII keystrokes into the frontmost window
$DESK key return # also: tab, esc, space, delete, arrows
$DESK quit
Injected input (preferred): no cursor move, no focus steal
agent-attach puts a tiny -javaagent (source: scripts/input-agent/InputAgent.java) into the .app's jpackage cfg; it injects synthetic AWT events in-process via a loopback socket — the system cursor never moves, the app need not be frontmost, and no Accessibility permission is involved. Validated end-to-end on the real Ani Compose window 2026-07-10 (clicks closed a dialog, switched nav, opened search; type inserted text into the search TextField — all while another app was frontmost). Injected events land on the Skia canvas (inject replies -> org.jetbrains.skiko.SkiaLayer$1) and Compose routes them internally. Ani draws its own window chrome, so info reports content == frame:
inject coordinates = window points = screenshot PNG px / 2, no title-bar offset.
$DESK agent-attach # once per built .app, BEFORE launch
$DESK launch
$DESK inject info # window bounds + content-area origin/size
$DESK inject click 300 200 # window-CONTENT points = window points minus title bar (~28pt);
# screenshot PNG pixel / 2, then subtract (content.y - frame.y)
$DESK inject type hello # goes to focus owner, else last injected-click target
$DESK inject key 10 # AWT keycodes: 10=Enter, 27=Esc, 9=Tab, 37-40=arrows
-
inject typegoes to Compose's internal focus, which follows injected clicks — click INTO the text field first (opening a screen is not enough: after opening search, typing did nothing until the field itself was clicked). -
For window-id screenshots WITHOUT activating/normalizing the window (keeps the app in the background), capture directly:
swift scripts/find_window_id.swift --pid $(pgrep -xn Ani)thenscreencapture -x -l <id> out.png. These PNGs include a shadow border (e.g. 3016x1936 for a 1440x900 window — subtract (PNG_w−2880)/2 = 68px per side before converting px/2 → points). -
Fall back to
click/type/cliclick below only if injection misbehaves. Injected hover (inject move) drives Compose hover states from event coordinates, but anything reading the REAL cursor position will disagree. -
Screenshots capture the window by CGWindowID (via
find_window_id.swift), so they show the app's own content even when other windows overlap it — the user may be actively using the desktop. The AppleScript step first normalizes the window to position (40,80) size 1440x900 points. -
Coordinate mapping: retina PNGs are 2x, so
window point = PNG pixel / 2(1440x900 window → 2880x1800 PNG; sanity-check your math against that). -
Click immediately after a fresh screenshot: toasts and update dialogs auto-dismiss, and a click computed from a stale screenshot lands on whatever is underneath.
clickprints the AX element it hit (e.g.scroll area 1 of group 1 …) — read it to confirm the intended target. -
click/type/keyand the window-resize step drive macOS System Events and require Accessibility permission;screencaptureneeds Screen Recording permission (both verified granted for this terminal). On failure report the permission gap instead of concluding app breakage. -
AppleScript
click atcan silently no-op on screens whose Compose accessibility tree is shallow (observed 2026-07-06 on the subject detail page: explore-page card clicks worked, then every detail-page click — buttons, back arrow — did nothing, with the AX hit always reporting the same genericUI element 1 of group 1 …). Fall back to real CGEvents:cliclick c:<screenX>,<screenY>(Homebrew-installed; screen = window point + window origin (40,80)) afterset frontmost to true— clicked the same button first try. -
cliclick is a GLOBAL click — it hits whatever is topmost at that point. If the user switches to another app mid-verification, every cliclick lands in their app (observed 2026-07-06: several clicks went into the user's Chrome session before a full-desktop capture revealed Ani wasn't frontmost). Immediately before each cliclick:
set frontmost to true, then VERIFY withosascript -e 'tell application "System Events" to get name of first process whose frontmost is true'==Ani; if not, stop clicking and report. Also keep the window fully on-screen first (this display is 1728pt wide; a 1440-wide window moved to x=600 puts its right side off-screen where clicks silently miss). -
The screenshot can capture the WRONG window. The process may own several CGWindows (e.g. a stale 1004x688 popup-host window stacked over the real 1440x900 main window, both at (40,80)); the frontmost-match then screenshots the static one, so the UI looks frozen and clicks look like no-ops even though they worked (observed 2026-07-06: playback had actually started while screenshots kept showing the explore page). Sanity-check every capture: PNG pixel size must equal AX window size × 2 (+136px shadow for
screencapture -l); on mismatch, enumerate CGWindows by ownerAnimeko(swift CGWindowListCopyWindowInfo) and capture the right id viascreencapture -l<id>. -
There is no semantics/hierarchy dump on desktop (unlike Android's
droid.sh tree); element targeting is visual. Prefer stable landmarks (window corners, sidebar order) and re-screenshot after every action. -
Wait 1–3 s after actions; JCEF/browser content and network data need longer.
3. Logs
$DESK logs 150 # newest app log: ~/Library/Application Support/*Ani*/logs/, or app/desktop/test-sandbox/logs/ for dev runs
The app logs dataDir/logsDir on startup and installs an uncaught-exception handler that logs !!!ANI FATAL EXCEPTION!!! — grep for that when the app dies.
4. Screen recording — catching transient glitches
For bugs visible under a second (a flash of wrong state, a one-frame layout jump, loading flicker), screenshots are not enough — record the window and diff the frames:
REC=$($DESK record-start) # starts recording in background, prints the output .mov path
$DESK click ... / key ... # do the interaction that should (or should not) glitch
$DESK record-stop # stops (SIGINT-finalizes), prints the video path
$DESK frames "$REC" # scans all frames, prints change events + exported images
frames (= scripts/frame_diff.py, a symlink to the android skill's copy; needs ffmpeg on PATH) scores every consecutive-frame difference, groups spikes into events, and per event exports a contact sheet (frames tiled left→right, top→bottom — Read it to see the whole sequence at a glance) plus full-res frames named by video timestamp. How to work with it:
- Recording captures the window by CGWindowID like screenshots, so it stays correct when other windows overlap. It is VFR at up to ~60 fps: frames are written only when the window content changes, so a timestamp gap proves the UI was static there.
- Judge events against your own actions: an event when nothing should have changed, or a change-then-revert pair, is the glitch. Timestamps are video-relative (capture starts before
record-startreturns — the window-id lookup takes a few seconds), so correlate by order/spacing, not wall clock. Mouse hovers create real events too (hover highlights, carousel arrows) — keep the pointer still when it shouldn't participate. - Zoom into a moment with
$DESK frames <video> --around <t> --window 0.5; print every frame's score with--list. - The video includes the window shadow: a 1440x900 window records as 3016x1936, content at offset (68,68) — i.e. video px = screenshot px + 68.
--crop 2880:1800:68:68trims to exactly the screenshot framing; for a sub-region use--crop W:H:(x+68):(y+68)with screenshot-pixel values. - Small elements barely move the frame-wide score — crop to the region of interest and/or lower
--threshold(default 0.01, try 0.003). record <seconds>is the blocking variant for no-interaction captures (launch animations, video playback checks).
5. Verify & report
- For each checked behavior state: the action, the expected result, and the screenshot path proving it.
- If the build fails, report the exact failing Gradle task and the first actionable compiler/jlink error.
- Do not claim desktop UI validation succeeded without a launched executable and a captured screenshot.
- Clean up:
$DESK quitunless the user wants the app kept open.
Desktop test tasks
./gradlew :app:desktop:test
./gradlew :app:shared:compileKotlinDesktop
Other desktop Gradle tasks observed here: :app:desktop:runDistributable, :app:desktop:packageDistributionForCurrentOS, and release variants. For Compose screenshot assertions in tests, see utils/ui-testing: SemanticsNodeInteraction.assertScreenshot(expectedResource) is implemented for Skiko-backed (desktop) targets.
Environment facts
- Desktop main class:
me.him188.ani.app.desktop.AniDesktop; System Events process nameAni(override withANI_DESKTOP_PROCESS), but the CGWindow owner name isAnimeko— window lookups therefore match by PID, not name. - App data dir:
AppFolderResolverwith app nameAni(Ani-debugin debug builds), observed at~/Library/Application Support/me.Him188.Ani/. - Healthy launch spawns JCEF helper processes (
jcef Helper (GPU)etc.) next to the main process —pgrep -fl "Ani.app/Contents/MacOS"is a quick liveness check. app/desktop/build.gradle.ktssupportsANI_COMPOSE_JAVA_HOME; the JBR must containjmods/jcef.jmodandlib/libjcef.dylib.- On this machine,
/Users/him188/Library/Java/JavaVirtualMachines/jbrsdk_jcef-21.0.4/Contents/Homewas verified end-to-end on 2026-07-05 (packages AND runs). - macOS ships bash 3.2 — scripts here must stay bash-3.2 compatible (no
mapfile, nodeclare -A). - The May-2026 desktop compile blocker (
CommonKoinModule.ktApiInvoker type mismatch) was verified FIXED on 2026-07-05::app:shared:application:compileKotlinDesktoppasses.