# Ort Test

> Ort Test is an Agent Skill for testing & quality published in the microsoft/onnxruntime repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Run ONNX Runtime tests. Use this skill when asked to run tests, debug test failures, or find and execute specific test cases in ONNX Runtime.

- Skill name: `ort-test`
- Category: [Testing & Quality](https://www.openparable.com/skill/testing-quality/1)
- Source: https://github.com/microsoft/onnxruntime/blob/HEAD/.agents/skills/ort-test/SKILL.md
- Repository: microsoft/onnxruntime (21,094 GitHub stars when collected)

## SKILL.md

# Running ONNX Runtime Tests

ONNX Runtime uses **Google Test** for C++ and **unittest** (preferred) / **pytest** for Python.

## C++ tests

### Test executables

| Executable | What it tests |
|---|---|
| `onnxruntime_test_all` | Core framework, graph, optimizer, session tests |
| `onnxruntime_provider_test` | Operator/kernel tests (Conv, MatMul, etc.) across execution providers |

### Two `attention_op_test.cc` files — don't confuse them

There are two same-named files testing **different operators**. Both build into
`onnxruntime_provider_test`:

| Path | Operator | gtest suite |
|---|---|---|
| `test/providers/cpu/llm/attention_op_test.cc` | **ONNX-domain** `Attention` (opset 23/24) | `AttentionTest.*` |
| `test/contrib_ops/attention_op_test.cc` | **contrib** MultiHeadAttention / GroupQueryAttention | `ContribOpAttentionTest.*` |

The MEA negative-offset regression tests (`Attention_Causal_NonPadKVSeqLen_MEA_*`,
e.g. `..._MEA_NegOffset_ForceFlashDisabled_FP16_CUDA`) live in the **providers/cpu/llm** file —
the ONNX-domain op.

Use `--gtest_filter` to select specific tests:

```bash
./onnxruntime_provider_test --gtest_filter="*Conv3D*"
```

### Running tests

**Always run from the build output directory** — tests may fail to find dependencies otherwise.

```bash
# Linux
cd build/Linux/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"

# macOS
cd build/MacOS/Release
./onnxruntime_provider_test --gtest_filter="*TestName*"

# Windows
cd build\Windows\Release
.\onnxruntime_provider_test.exe --gtest_filter="*TestName*"
```

You can also run all tests via the build script (assumes a prior successful build):

```bash
./build.sh --config Release --test
.\build.bat --config Release --test    # Windows
```

### Locating the build output directory

The default path follows the pattern `build/<Platform>/<Config>/` where Platform is `Linux`, `MacOS`, or `Windows`. With Visual Studio multi-config generators on Windows, the config may appear twice (e.g., `build/Windows/Release/Release/`). The path can also be customized via `--build_dir`.

If you can't find a test binary, search for it:

```powershell
# Windows
Get-ChildItem -Path build -Recurse -Filter "onnxruntime_provider_test.exe" | Select-Object -ExpandProperty FullName

# Linux/macOS
find build -name "onnxruntime_provider_test" -type f
```

## Python tests

Use `pytest` as the test runner:

```bash
pytest onnxruntime/test/python/test_specific.py                          # entire file
pytest onnxruntime/test/python/test_specific.py::TestClass::test_method  # specific test
pytest -k "test_keyword" onnxruntime/test/python/                        # by keyword
```

Python test naming convention: `test_<method>_<expected_behavior>_[when_<condition>]`

## Agent tips

- **Activate a Python virtual environment** before running tests. See "Python > Virtual environment" in `AGENTS.md`.
- **Beware false-green results** — a green run does not always prove anything. See the
  "False-green taxonomy" section below for the four ways a test can pass without testing
  your change.
- **Redirect test output to a file** (e.g., `> test_output.txt 2>&1`) — output can be large.
- For C++ tests, verify the build directory exists and a prior build completed before running.
- Use `--gtest_filter` to run a targeted subset when the full suite takes too long.
- **Running WebGPU tests locally on Linux without a GPU** — WebGPU op tests build into `onnxruntime_provider_test` and can run against a software Vulkan adapter (Mesa lavapipe). See the `webgpu-local-testing` skill.

## False-green taxonomy — ways a test can "pass" without proving anything

A green result is not always a real pass. Watch for all five modes:

1. **Zero-match filter.** A `--gtest_filter` that matches no tests still exits 0 (green).
   Confirm the `[==========] N tests ran` line is non-zero — a zero-match run prints
   `0 tests from 0 test suites`. Many operator/kernel gtests run only in
   **`onnxruntime_provider_test`** (CI runs this), NOT `onnxruntime_test_all`; the wrong
   binary matches nothing and looks green.
2. **Stale binary from an incremental build.** If the build did not actually recompile your
   change (e.g. a header not tracked by the compiler's depfile), the "passing" run executes
   the OLD code. A test that was failing cannot truly flip to passing without a real
   rebuild — treat an unexpected FAIL→PASS with suspicion and confirm the linked artifact's
   mtime advanced. CUDA/CUTLASS instance (nvcc depfiles don't track `cutlass_fmha/*.h`): see
   the `cuda-cutlass-fmha-incremental-rebuild` skill.
3. **Checking the wrong artifact's freshness.** With a dlopen'd shared provider (e.g.
   `libonnxruntime_providers_cuda.so`), the test executable is NOT relinked when the provider
   recompiles — its mtime stays old while the `.so` advances. Verify the artifact that
   actually links your change, not the test exe. Detail: `cuda-cutlass-fmha-incremental-rebuild`
   skill.
4. **A correct fallback path masks the intended path.** A value-only assertion can pass via a
   *different, correct* code path without ever exercising the one you meant to test (e.g. a
   test meant for MEA silently handled by the unfused fallback). Assert/verify **which path
   ran**, not just the output value — see "Verify which path/kernel actually executed" below.
5. **Arch-portability false-green (verified on only one GPU arch).** A CUDA kernel that
   launches on a large-dynamic-smem arch (e.g. sm90/H100, ~227KB) can **fail to launch** on a
   smaller opt-in cap (sm86/89 ~99KB, sm80 ~163KB) with `CUDA failure 1: invalid argument` —
   and a path with no fallback (e.g. ORT's MEA) turns that into a hard error, not a silent
   degrade. So a green run on your local GPU can mask a launch failure on CI's arch. Verify
   arch-portability, or pick a config whose shared-memory footprint fits **every** target arch
   (e.g. a small `head_size`). Concrete instance: CUTLASS MEA `head_size=512` FP16 exceeds
   sm86's smem opt-in cap and dies at launch — live bug #28388 (the
   `cuda-attention-kernel-patterns` skill §1 has the dispatch detail).

## Verify which path/kernel actually executed

Value equality alone does not prove the intended code path ran — a correct fallback can
produce the right answer (false-green mode 4 above). When a test targets a specific
kernel/path, confirm it actually dispatched there instead of trusting the output:

- Enable verbose logging and check the dispatch log line. ORT attention logs one of these
  exact strings (`core/providers/cuda/llm/attention.cc`):
  - `ONNX Attention: using Flash Attention` (:1400)
  - `ONNX Attention: using Memory Efficient Attention` (:1451)
  - `Attention: using unified unfused path` (:1482) — note: **no `ONNX ` prefix** and it
    reads "unified unfused path", not "Unfused".
- Or force the path via the relevant env var / build config AND add a compile-time guard so
  the test **SKIPs** (not silently passes) when the target path is unavailable — e.g.
  `SKIP_IF_MEA_NOT_COMPILED`.

Operator-specific routing/forcing details: `cuda-attention-kernel-patterns` skill §1/§7.

---

Source: https://www.openparable.com/skill/testing-quality/ort-test (OpenParable). Cite the canonical URL above.
