clawgui-eval

ClawGUI-Eval -- GUI Grounding Model Evaluation

Evaluate GUI grounding models on standard benchmarks (ScreenSpot-Pro, ScreenSpot-v2, UIVision, MMBench-GUI, OSWorld-G, AndroidControl). The eval framework lives at clawgui-eval/ inside the workspace.

All commands below MUST be run with cd clawgui-eval first (relative paths for data/, image/, output/ depend on this).

---

Phase 1: Environment Check

Before running any evaluation, execute these checks and report results to the user.

GPU

nvidia-smi --query-gpu=index,name,memory.total,memory.free --format=csv,noheader

Count the lines to determine available GPU count. Use this as NUM_GPUS unless the user specifies otherwise.

System resources

free -h | head -3
df -h .

Python & CUDA

cd clawgui-eval && python3 -c "import torch; print(f'torch={torch.__version__}, cuda={torch.cuda.is_available()}, gpus={torch.cuda.device_count()}')"

FlashAttention-2

python3 -c "import flash_attn; print(f'flash_attn={flash_attn.__version__}')" 2>&1

If flash_attn is NOT installed, you MUST clearly warn the user:

"FlashAttention-2 未安装。框架会自动降级为 SDPA（PyTorch 内置），可以正常运行但精度可能略有下降。强烈建议安装 FlashAttention-2 以获得最佳精度："

pip install flash-attn --no-build-isolation

如果从源码编译太慢，可以从 https://github.com/Dao-AILab/flash-attention/releases 下载预编译 wheel 安装。

Always include this warning prominently in the environment check report if flash_attn is missing. Do not silently skip it.

Data integrity

ls clawgui-eval/data/ | head -5
ls clawgui-eval/image/ | head -3

Both data/ and image/ must exist. If image/ is missing, tell the user to download and unzip the dataset first.

Model path

If the user provides a local MODEL_PATH, verify it exists:

ls <MODEL_PATH>/config.json 2>/dev/null && echo "OK" || echo "NOT FOUND"

Report all results to the user. If any critical check fails (no GPU, no CUDA, missing data/image), stop and explain what is needed.

---

Phase 2: Run Evaluation

The full pipeline is: Infer → Judge → Metric. You MUST execute all three steps in sequence -- do NOT stop after Judge. The Metric step produces the final accuracy numbers that the user needs.

The user must provide:

• MODEL_TYPE -- which model architecture (see script table below)
• MODEL_PATH -- HuggingFace ID or local directory
• EXPERIMENT_NAME -- a label for this run (output goes to output/<EXPERIMENT_NAME>/...)
• BENCHMARK -- which benchmark to evaluate on (see naming rules below)

Supported models (transformers backend)

Only the following MODEL_TYPE values are supported. If the user requests a model not in this list, do NOT proceed -- tell them the model is not yet supported by ClawGUI-Eval and show this table for reference.

- means the combination is not supported. If the user requests a MODEL_TYPE + BENCHMARK combination marked -, refuse and explain it is not available.

Users can substitute different sizes within the same model family by changing the size in the HuggingFace ID (e.g. Qwen3-VL-2B → Qwen3-VL-4B). The MODEL_TYPE stays the same regardless of model size. Users can also provide a local path instead of a HuggingFace ID.

API-only models (gemini, seed) are NOT supported by this skill -- they require the API/vLLM backend which is outside scope.

Step 1: Inference (transformers)

Script templates are at clawgui-eval/scripts/infer/transformers/. Pick the one matching the MODEL_TYPE:

How to create and run the inference script

Do NOT modify the template scripts in-place. Follow this procedure:

1. Read the matching template script with read_file to get its full content.
2. Create directory clawgui-eval/scripts/generate_scripts/ if it does not exist.
3. Copy the content into a new script at clawgui-eval/scripts/generate_scripts/<EXPERIMENT_NAME>.sh.
4. In the new script, only modify these variables at the top:

Leave all other variables (MAX_TOKENS, TEMPERATURE, TOP_P, TOP_K, MIN_PIXELS, MAX_PIXELS, TV_OR_VT, SYSTEM_PROMPT, USE_CACHE, etc.) at the template defaults unless the user explicitly asks to change them.

5. Run the generated script (see "Running inference with monitoring" below).

Benchmark naming

Format: <dataset>-<MODEL_TYPE>

Five dataset prefixes:

• screenspot-pro-<MODEL_TYPE>
• screenspot-v2-<MODEL_TYPE>
• uivision-<MODEL_TYPE>
• mmbench-gui-<MODEL_TYPE>
• osworld-g-<MODEL_TYPE>

Example: MODEL_TYPE=qwen3vl → BENCHMARK=screenspot-pro-qwen3vl

Same model family with different sizes (e.g. Qwen3 2B/4B/8B) share the same BENCHMARK; differentiate via EXPERIMENT_NAME and MODEL_PATH only.

AndroidControl uses: androidcontrol-high-<MODEL_TYPE> or androidcontrol-low-<MODEL_TYPE>.

Running inference with monitoring

Inference is long-running (minutes to hours). The exec tool has a 600s timeout, so use background execution with log redirection.

Step 1a: Get total sample count

cd clawgui-eval && python3 -c "import json; print(len(json.load(open('data/<BENCHMARK>.json'))))"

Save this number as TOTAL_SAMPLES.

Step 1b: Launch the generated script in background

cd clawgui-eval && mkdir -p output/<EXPERIMENT_NAME>/<BENCHMARK> && \
nohup bash scripts/generate_scripts/<EXPERIMENT_NAME>.sh \
  > output/<EXPERIMENT_NAME>/<BENCHMARK>/run.log 2>&1 &
echo "PID=$!"

Save the PID.

Step 1c: Poll progress

Periodically check (start at ~30s intervals, increase to ~60s as it stabilizes):

# Is the process still running?
ps -p <PID> -o pid=,stat=,etime= 2>/dev/null || echo "DONE"

# How many samples completed?
wc -l < clawgui-eval/output/<EXPERIMENT_NAME>/<BENCHMARK>/predictions.jsonl 2>/dev/null || echo 0

# GPU utilization
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total --format=csv,noheader

# Last few log lines
tail -5 clawgui-eval/output/<EXPERIMENT_NAME>/<BENCHMARK>/run.log

Report progress to the user: "Completed X / TOTAL_SAMPLES samples (Y%), GPU utilization Z%".

Step 1d: Detect completion

Inference is done when:

• ps -p <PID> returns nothing (process exited), OR
• predictions.jsonl line count equals TOTAL_SAMPLES

When done, check the log tail for errors:

tail -30 clawgui-eval/output/<EXPERIMENT_NAME>/<BENCHMARK>/run.log

If the exit was abnormal (errors in log), report to the user. The --resume flag means re-running will continue from where it stopped.

Step 2: Judge

Judge script templates are at clawgui-eval/scripts/judge/. Pick the one matching the benchmark:

Same procedure as inference:

1. Read the matching template with read_file.
2. Copy to clawgui-eval/scripts/generate_scripts/<EXPERIMENT_NAME>_judge.sh.
3. Modify only the variables at the top: EXP_NAME and MODEL_TYPE (the BENCHMARKS array is derived from MODEL_TYPE automatically in most templates).
4. Run it:

cd clawgui-eval && bash scripts/generate_scripts/<EXPERIMENT_NAME>_judge.sh

Judging is fast (seconds), so run directly with exec -- no need for background execution.

After Judge completes, immediately proceed to Step 3 (Metric). Do NOT stop here.

Step 3: Metric (REQUIRED)

Metric script templates are at clawgui-eval/scripts/metric/. Pick the one matching the benchmark:

Same procedure:

1. Read the matching template with read_file.
2. Copy to clawgui-eval/scripts/generate_scripts/<EXPERIMENT_NAME>_metric.sh.
3. Modify the variables at the top: EXP_NAME, and BENCHMARK (or MODEL_TYPE if the template derives BENCHMARK from it).
4. Run it:

cd clawgui-eval && bash scripts/generate_scripts/<EXPERIMENT_NAME>_metric.sh

---

Phase 3: Report Results

After all three steps (Infer → Judge → Metric) complete:

1. Read output/<EXPERIMENT_NAME>/<BENCHMARK>/metrics.json with read_file and extract the accuracy numbers.
2. Read the tail of run.log to extract inference statistics (throughput, elapsed time).
3. Look up the official baseline from the reference table below for comparison.
4. Present results to the user as a comparison table, showing the user's result alongside the official baseline.

Example format:

=== Evaluation Results ===

Experiment:  qwen3vl-2b-screenspot-pro
Model:       Qwen/Qwen3-VL-2B-Instruct (qwen3vl)
Benchmark:   screenspot-pro-qwen3vl
GPUs:        8

| Metric    | Your Result | Official Baseline |
|-----------|-------------|-------------------|
| Accuracy  | 44.12%      | 48.50%            |

Sub-category breakdown:
  - Mobile:  XX.X%
  - Desktop: XX.X%
  - Web:     XX.X%

Inference: 12.3 samples/s, 103.4s total

If the user ran multiple benchmarks, present a summary comparison table at the end:

| Benchmark       | Your Result | Official Baseline |
|-----------------|-------------|-------------------|
| screenspot-pro  | XX.X%       | XX.X%             |
| screenspot-v2   | XX.X%       | XX.X%             |
| ...             | ...         | ...               |

Official baseline reference

Use this table to look up official baselines for comparison. - means no official number is available.

When looking up the baseline, match by model name and size. If the exact size is not in the table (e.g. user tested a new size), show the closest available sizes for reference. If no baseline exists (-), note that no official number is available for this combination.

---

Pitfalls

• Always cd clawgui-eval first. All relative paths (data/, image/, output/) depend on this.
• Never omit --benchmark. Without it, main.py runs ALL benchmarks in the registry, which takes extremely long.
• maiui uses TV_OR_VT=tv, not vt. Do not change this to match other models.
• --resume is on by default. Re-running the same experiment+benchmark continues from where it stopped rather than starting over.