Experimental CI Only Validator

DevOps & Security

experimental-ci-only-validator

Experimental CI Only Validator is an Agent Skill for devops & security published in the chromium/chromium repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Checks if a Chromium test suite is barred by `ci_only = True` configuration on coverage trybots, which prevents the test suite from running automatically on Gerrit CQ runs and can cause underreported code coverage.

View on GitHub ↗

QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

Open your project in Codex.
Copy and paste the prompt below.
Review files and risks before approving any action.

Prompt to paste

Inspect the Agent Skill at https://github.com/chromium/chromium/blob/HEAD/agents/skills/experimental-ci-only-validator/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/experimental-ci-only-validator, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

CI-Only Validator Skill

This skill provides a quick way to check if a test suite has been configured as ci_only = True on the code coverage trybots.

When to Use This Skill

Use this skill when:

You are triaging an underreported code coverage bug on Gerrit.
You have already mapped the target source or test file to its parent executable test suite (e.g., using TestSuiteMapper).
You suspect that the test suite is configured on the coverage builders but is not executing in tryjobs.

How to Use This Skill

If you have identified the test suite name (e.g., chrome_junit_tests), run the quick check tool:

python3 tools/code_coverage/ci_only_validator.py <test_suite_name>

Checking at a Past Revision (Historical Triage)

To verify the ci_only status of a test suite at a specific historical point (e.g., the revision of a Gerrit CL), pass the --revision (or -r) flag:

python3 tools/code_coverage/ci_only_validator.py <test_suite_name> --revision <commit_hash_or_branch>

This is highly recommended for historical triaging because configurations on main might have changed since the CL was reviewed.

Interpreting the Results

The tool will scan CQ coverage trybots and coverage-specific trybots generated under infra/config/generated/builders/try/ and report their configuration for the target test suite:

ci_only = True: The test suite is barred from executing on this trybot by default during CQ runs. Gerrit coverage details for tests in this suite will not be generated automatically unless the developer explicitly requests them via a Gerrit footer.
ci_only = False: The test suite is allowed to run on the trybot in CQ.
Warning: Test suite ... was not found...: The test suite is not configured to run on any coverage trybots, meaning no coverage details are collected on CQ at all.

Example Runs

1. Checking Current Configuration (HEAD)

python3 tools/code_coverage/ci_only_validator.py chrome_junit_tests

Output:

Checking test suite 'chrome_junit_tests' across 30 coverage trybots...
  Trybot: android-code-coverage          | Builder: chromium.coverage:android-code-coverage          | ci_only = True
  Trybot: android-desktop-x64-rel        | Builder: chromium.android.desktop:android-desktop-x64-rel-15-tests | ci_only = False
  Trybot: android-x86-code-coverage      | Builder: chromium.coverage:android-x86-code-coverage      | ci_only = True
  Trybot: android-x86-rel                | Builder: chromium.android:android-10-x86-nofieldtrial-rel | ci_only = True

This indicates that chrome_junit_tests is barred by ci_only = True in standard CQ at HEAD.

2. Checking Historical Configuration (at a revision)

python3 tools/code_coverage/ci_only_validator.py chrome_junit_tests --revision verify-junit-coverage

Output:

Checking test suite 'chrome_junit_tests' across 30 coverage trybots at revision verify-junit-coverage...
  Trybot: android-code-coverage          | Builder: chromium.coverage:android-code-coverage          | ci_only = False
  Trybot: android-desktop-x64-rel        | Builder: chromium.android.desktop:android-desktop-x64-rel-15-tests | ci_only = False
  Trybot: android-x86-code-coverage      | Builder: chromium.coverage:android-x86-code-coverage      | ci_only = False
  Trybot: android-x86-rel                | Builder: chromium.android:android-10-x86-nofieldtrial-rel | ci_only = False

This indicates that on the verify-junit-coverage branch, the ci_only restriction has been successfully disabled.