# Model Checker

> Model Checker is an Agent Skill for testing & quality published in the openvinotoolkit/openvino.genai repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Validate a newly supported optimum-intel model with OpenVINO GenAI. Use when: checking new model support, verifying model export to OpenVINO IR, running GenAI inference test with llm_bench, benchmarking model accuracy with who-what-benchmark.

- Skill name: `model-checker`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/openvinotoolkit/openvino.genai/blob/HEAD/.github/skills/model-checker/SKILL.md
- Repository: openvinotoolkit/openvino.genai (553 GitHub stars when collected)

## SKILL.md

# Model Checker

Validates that a HuggingFace model exported via optimum-intel works correctly with OpenVINO GenAI pipelines and passes accuracy benchmarks.

## When to Use

- A new model was added to optimum-intel and needs GenAI validation
- Verify a HuggingFace model exports to OpenVINO IR and runs inference
- Check model accuracy after conversion using who-what-benchmark

## Inputs

The user must provide:

- **model_id**: HuggingFace model identifier (e.g. `tencent/HY-MT1.5-1.8B`) **or** path to an existing OpenVINO IR directory (e.g. `/tmp/my_model_ir`). When a local path is provided, export is skipped automatically.
- **task**: optimum-cli export task. Supported values:
  - `text-generation-with-past`
  - `image-text-to-text`
  - `text-to-image`
  - `image-to-image`
  - `feature-extraction`
  - `text-classification`
  - `text-to-video`
  - `automatic-speech-recognition`

## Prerequisites

Ensure the Python virtual environment is activated before running any commands.

1. **Locate the virtual environment** — check for common directories at the repository root: `.venv/`, `venv/`, `env/`. Use `list_dir` to find it. If none is found, ask the user for its location.
2. **Check if already activated**: if `which python` or `where python` points inside the virtual environment, it's already activated. If not, proceed to activate it.
3. **Activate** based on the current platform:
   - **Linux/macOS**: `source <venv_path>/bin/activate`
   - **Windows (cmd)**: `<venv_path>\Scripts\activate.bat`
   - **Windows (PowerShell)**: `<venv_path>\Scripts\Activate.ps1`
4. The background terminal doesn't inherit the venv activation. Run it with the venv activated in the same command.

## Procedure

### Step 1: Run check_model.py

Run the checker script from the repository root:

```
python3 .github/skills/model-checker/scripts/check_model.py \
    --model-id <model_id_or_path> \
    --task <export_task> \
    --work-dir .model_enabler/model_checker
```

When `--model-id` is a path to an existing directory, the script treats it as a pre-converted OpenVINO IR model and automatically skips the export step.

Run `python3 .github/skills/model-checker/scripts/check_model.py --help` for the full argument reference including defaults. The `--work-dir` is where all intermediate files, logs, and outputs will be stored. Do not pipe with any additional logging or redirection — the script handles its own logging.

#### Skip flags (for re-runs after a fix)

When a previous run already passed some steps (e.g. export succeeded but inference test failed), use skip flags to avoid repeating expensive passed steps:

- `--skip-export` — reuse existing IR in `<work-dir>/model_ir` instead of re-exporting (avoids re-downloading weights). When `--model-id` is a local path, export is bypassed automatically and that directory is used directly instead of `<work-dir>/model_ir`.
- `--skip-llm-bench` — skip the llm_bench inference test
- `--skip-wwb` — skip the who-what-benchmark accuracy check
- `--skip-wwb-ground-truth` — skip WWB ground-truth collection and reuse `<work-dir>/wwb/gt.csv`; useful when iterating on GenAI target evaluation after ground truth was already collected

Do **not** use skip flags on the first run. Only use them when retrying after a targeted fix.

### Step 2: Interpret Results

The script logs progress for each step and exits with code 0 (pass) or non-zero (fail).

**Pass criteria:**

- Export: exit code 0
- Inference test (llm_bench): exit code 0, metrics line logged
- WWB accuracy (depends on `--wwb-base` mode):
  - `--wwb-base optimum` (default):
    1. Optimum ground truth generation: exit code 0
    2. GenAI target evaluation: similarity ≥ `SIMILARITY_THRESHOLD`
  - `--wwb-base hf`:
    1. HF ground truth generation: exit code 0
    2. Optimum target evaluation: similarity ≥ `SIMILARITY_THRESHOLD`
    3. GenAI target evaluation: similarity ≥ `SIMILARITY_THRESHOLD`

  Note: the WWB step is skipped automatically for `automatic-speech-recognition` (no WWB support).

**Log files:** each tool writes its own dedicated log; paths are printed during execution. When a step fails, read the corresponding log for the full traceback and context before drawing any conclusions.

**work-dir:** work-dir is in current workspace, prefer to use tool calls to access logs and outputs instead of custom bash commands.

### Step 3: Report Results

Results format:

- **Model**: `<model_id>` (`<task>`)
- **Validation**: PASSED / FAILED
- **Performance** (if passed):
  - 1st token latency, 2nd token latency, throughput
  - Optimum similarity / GenAI similarity (if applicable)
- **Logs**: paths to export log, llm_bench log, WWB logs
- **Failed step analysis** (if failed): summary of the failure and relevant log path for details

### Security

- **NEVER** invoke `optimum-cli`, `wwb`, or `llm_bench` directly. Always go through `check_model.py`.
- **NEVER** modify `model_id` — pass it exactly as provided by the user.

---

Source: https://www.openparable.com/skill/testing-quality/model-checker (OpenParable). Cite the canonical URL above.
