flutter-mcp-toolkit-control

When to use

Use this skill when you need to drive a running Flutter app as a user would:

• Tap buttons, icons, list items, or any interactive widget.
• Type text into fields, submit forms, clear inputs.
• Scroll or swipe to reveal off-screen content.
• Navigate between routes programmatically (push, pop, popUntil).
• Dismiss dialogs and bottom sheets.
• Press keyboard keys (Enter, Escape, Tab, arrows, ASCII chars).
• Hot-reload or hot-restart after editing Dart source files.
• Combine reload + screenshot + semantics in one round-trip for fast iteration.

Selectors

Most interaction tools target a widget by ref — a short string like "s_0" returned by semantic_snapshot. For visible widgets, call semantic_snapshot, scan the returned nodes, find the right ref, then pass it. For off-screen targets with stable semantics text or identifier, use reveal_search; it performs a bounded snapshot → match → scroll loop and returns a fresh ref/snapshotId.

Snapshot node fields to filter on:

Example — find the "Login" button ref:

semantic_snapshot()
→ nodes: [{ref:"s_0", label:"Login", actions:["tap"]}, ...]
tap_widget(ref: "s_0")

Pass snapshotId (from the snapshot response) to any interaction call. If the tree has changed, the call returns stale_snapshot with both IDs so you know to re-snapshot. Refs are only valid against the most recent snapshot.

Recipes

Tap a widget by text

semantic_snapshot()
→ find node where label == "Submit" → ref "s_3"
tap_widget(ref: "s_3", snapshotId: <id>)

Fill a login form

semantic_snapshot() → email ref "s_1", password ref "s_2"
fill_form(fields: [{ref:"s_1", text:"user@example.com"}, {ref:"s_2", text:"secret"}], snapshotId: <id>)
→ one round-trip; stops on first failure

Scroll to find an item

scroll(direction: "down", distance: 300)
semantic_snapshot() → item now visible → ref "s_5"
tap_widget(ref: "s_5")

Reveal an off-screen field by identifier

reveal_search(query: "greeting_input_field", matchBy: "identifier", direction: "down", maxAttempts: 4)
→ returns ref "s_14", snapshotId 2, match, attempts trace
enter_text(ref: "s_14", snapshotId: 2, text: "hello")

Wait for a widget to appear

wait_for(predicate: {kind: "text", text: "Welcome"}, timeoutMs: 8000)
→ returns fresh snapshot when text appears
tap_widget(ref: <ref from wait_for snapshot>)

Navigate to a route

navigate(action: "push", route: "/settings", arguments: {tab: "account"})
semantic_snapshot() → fresh refs in the new screen

Hot reload after a code change

hot_reload_and_capture()
→ screenshot + semantic snapshot + errors in one call

Press the back hardware button

press_key has no Back key. Use navigate(action: "pop") for Navigator pop; handle_dialog(action: "dismiss") for dialogs; press_key(key: "Escape") on desktop.

navigate(action: "pop")

Tool reference

tap_widget

Tap a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "tap_widget", "arguments": {"ref": "s_3", "snapshotId": 7}}

Returns: {"via": "semantic_action", "ref": "s_3"} — Failures: stale_snapshot, ref_not_found

long_press

Long-press a widget by ref. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "long_press", "arguments": {"ref": "s_2"}}

Returns: {"via": "semantic_action"} — Failures: stale_snapshot, ref_not_found

enter_text

Enter text into a text field; taps to focus before typing. ref • string • required. text • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "enter_text", "arguments": {"ref": "s_1", "text": "hello@example.com"}}

Returns: {"via": "editable_state"} — Failures: stale_snapshot, ref_not_found

reveal_search

Find a semantic target that may be off-screen. query • string • required. matchBy • string • optional (text|identifier|label|value|hint, default text). direction • string • optional (up|down|left|right, default down). maxAttempts • integer • optional • max 10. distance • number • optional. connection • object • optional.

{"name": "reveal_search", "arguments": {"query": "greeting_input_field", "matchBy": "identifier", "direction": "down", "maxAttempts": 4}}

Returns: {"ref": "s_14", "snapshotId": 2, "match": {...}, "attempts": [...]} — Failures: missing_query, target_not_found, scroll_blocked

fill_form

Batch text entry: fills multiple fields in one call. Stops on first failure. snapshotId validated on first field only. fields • array of {ref, text} • required. snapshotId • integer • optional. connection • object • optional.

{"name": "fill_form", "arguments": {"fields": [{"ref":"s_1","text":"user"},{"ref":"s_2","text":"pass"}], "snapshotId": 5}}

Returns: {"filled": 2} — Failures: stale_snapshot, ref_not_found

scroll

Scroll to reveal content. "down" reveals content below (finger swipes up). direction • string • required (up|down|left|right). ref • string • optional (falls back to screen center). distance • number • optional • default 300. snapshotId • integer • optional. connection • object • optional.

{"name": "scroll", "arguments": {"direction": "down", "ref": "s_0", "distance": 500}}

Returns: {"via": "semantic_action"} — Failures: ref_not_found, stale_snapshot

swipe

High-velocity fling. Same direction model as scroll. Always Tier 2 pointer events. direction • string • required. ref • string • optional. distance • number • optional • default 300. snapshotId • integer • optional. connection • object • optional.

{"name": "swipe", "arguments": {"direction": "left", "ref": "s_4"}}

Returns: {"via": "pointer_events"} — Failures: ref_not_found, web_gesture_not_supported

drag

Drag from one widget to another. Always Tier 2. fromRef • string • required. toRef • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "drag", "arguments": {"fromRef": "s_2", "toRef": "s_7"}}

Returns: {"via": "pointer_events"} — Failures: ref_not_found, web_gesture_not_supported

hover

Synthesize a mouse hover. Desktop/web only — no hover concept on mobile. ref • string • required. snapshotId • integer • optional. connection • object • optional.

{"name": "hover", "arguments": {"ref": "s_5"}}

Returns: {"via": "pointer_events"} — Failures: ref_not_found, platform error on mobile

press_key

Synthesize key press (down+up). Accepted: Enter Escape Tab Backspace Delete Space ArrowUp ArrowDown ArrowLeft ArrowRight plus single ASCII (a-z 0-9). key • string • required. ctrl/shift/alt/meta • boolean • optional • default false. connection • object • optional.

{"name": "press_key", "arguments": {"key": "Enter"}}

Returns: {"key": "Enter"} — Failures: unsupported_key, no_focus

wait_for

Wait for a UI predicate; returns fresh semantic snapshot. Predicates: {kind:"text",text} | {kind:"noText",text} | {kind:"time",ms} | {kind:"stable",stableWindowMs}. predicate • object • required. timeoutMs • integer • optional • default 5000 • max 30000. connection • object • optional.

{"name": "wait_for", "arguments": {"predicate": {"kind": "text", "text": "Dashboard"}, "timeoutMs": 8000}}

Returns: fresh semantic snapshot — Failures: timeout, invalid_predicate

navigate

Drive the registered Navigator. Requires MCPToolkitBinding.instance.navigatorKey = key in the app. action • string • required (push|pop|popUntil). route • string • required for push/popUntil. arguments • object • optional (for push). connection • object • optional.

{"name": "navigate", "arguments": {"action": "push", "route": "/profile", "arguments": {"userId": "42"}}}

Returns: {"action": "push", "route": "/profile"} — Failures: navigator_not_configured, route_not_found

handle_dialog

Dismiss the topmost popup/dialog route. Only action: "dismiss" supported. Requires navigatorKey = key on MCPToolkitBinding.instance in the app. action • string • required (must be "dismiss"). connection • object • optional.

{"name": "handle_dialog", "arguments": {"action": "dismiss"}}

Returns: {"dismissed": true} — Failures: navigator_not_configured, no_dialog

hot_reload_flutter

Hot reload the app. Preserves state. force • boolean • optional • default false (reload even without source changes). connection • object • optional.

{"name": "hot_reload_flutter", "arguments": {}}

Returns: "Hot reload completed" + report JSON — Failures: vm_not_connected, compilation_error

hot_restart_flutter

Full restart. App state not preserved. No required params. connection • object • optional.

{"name": "hot_restart_flutter", "arguments": {}}

Returns: {"report": {"type": "Success", "success": true}} — Failures: vm_not_connected

hot_reload_and_capture

Hot reload then capture screenshot + semantics + errors in one call. compress • boolean • default true. includeSemantics • boolean • default true. includeErrors • boolean • default true. errorsCount • integer • default 4. connection • object • optional.

{"name": "hot_reload_and_capture", "arguments": {"includeErrors": true}}

Returns: screenshot (base64) + semantic snapshot + errors — Failures: vm_not_connected, compilation_error

Patterns

Always wait_for before tap_widget after navigation

After navigate(action: "push") the new route's widgets are not in the tree yet. Use wait_for with a text predicate to confirm the destination has rendered, then snapshot and act.

navigate(action: "push", route: "/checkout")
wait_for(predicate: {kind: "text", text: "Order Summary"}, timeoutMs: 5000)
semantic_snapshot() → tap target widgets

Prefer fill_form over multiple enter_text calls

Each enter_text is a separate VM round-trip. fill_form sends all field/text pairs in one call; snapshotId is checked once (on the first field). For any form with 2+ fields, always prefer fill_form.

# Avoid: 2 round-trips
enter_text(ref: "s_1", text: "Alice")
enter_text(ref: "s_2", text: "secret")

# Prefer: 1 round-trip
fill_form(fields: [{ref: "s_1", text: "Alice"}, {ref: "s_2", text: "secret"}])

After hot_reload_*, wait for the new tree before continuing

Hot reload completes asynchronously. Use wait_for(predicate: {kind:"stable", stableWindowMs:300}) to confirm the tree has settled before calling semantic_snapshot. Or use hot_reload_and_capture which returns a post-reload snapshot directly.

hot_reload_flutter()
wait_for(predicate: {kind: "stable", stableWindowMs: 300})
semantic_snapshot() → interact with reloaded widgets

# Or in one call (preferred):
hot_reload_and_capture() → screenshot + semantics + errors already post-reload