Experimental Code Coverage Cq Debugger
Testing & Qualityexperimental-code-coverage-cq-debuggerExperimental Code Coverage Cq Debugger is an Agent Skill for testing & quality published in the chromium/chromium repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Executes the phased, multi-skill code coverage debugging playbook for triaging underreported code coverage in the Gerrit Commit Queue (CQ).
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/chromium/chromium/blob/HEAD/agents/skills/experimental-code-coverage-cq-debugger/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/experimental-code-coverage-cq-debugger, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
State Management
Throughout the triage process, maintain a state file at
scratch/triage_state.json. In Step 1, you MUST load the existing state
file (initialized by the orchestrator) and initialize ALL variables defined in
templates/triage_state_template.json in
scratch/triage_state.json initially, even if many of them will initially be
null. Update these variables in scratch/triage_state.json as they are
resolved in subsequent steps.
Variables to Initialize:
When initializing variables in Step 1, reference
templates/triage_state_template.json for
the required variables and data contract. All variables from this schema MUST be
loaded into scratch/triage_state.json at the start of Phase 1.
Commit Queue (CQ) Coverage Debugging Playbook
This skill defines the end-to-end triaging and resolution process for Gerrit Commit Queue (CQ) underreported coverage bugs. It guides the agent through Phase 1 (Information Gathering), Phase 2 (Audit Configuration), Phase 3 (Prepare Experiment), and Phase 4 (Validation).
[!IMPORTANT] Sequential Execution & Status Updates:
- Sequential Execution: You must execute each step in the workflow sequentially. Do NOT skip any steps or proceed to a subsequent step until the current step has fully completed, returned its outputs, and validated its constraints.
- Status Updates: You MUST update the
statusfield inscratch/triage_state.jsonwhenever transferring to another step (i.e., at the end of each step) to track progress. The status value should reflect the step name or number just completed (e.g.,"CQ Triage: Step 5 Completed"). The final terminal status values must be set to either"FIXED"or"ESCALATED".
Phase 1: Information Gathering
Step 1: Initialize State File
- Action: Load
scratch/triage_state.jsonand initialize/append ALL variables defined in templates/triage_state_template.json inscratch/triage_state.json(setting unresolved variables tonullor empty dictionaries). Ensure all existing variables already defined inscratch/triage_state.json(such asbug_id,status,bug_details, etc., initialized by the orchestrator) are preserved. - Required Outputs: All schema variables appended and initialized in
scratch/triage_state.json.
Step 2: Scrape Gerrit CL Details & Verify Repository
- Action: Extract the CL URL from
bug_detailsinscratch/triage_state.jsonand store it in theoriginal_clvariable inscratch/triage_state.json. Then, verify the repository and retrieve modified files fororiginal_cl. (Note: You can usetools/code_coverage/parse_gerrit_url.pyto extracthost,project,change, andpatchsetdetails from any CL URL when needed). - Skill: Use the gerrit-cli skill.
- Required Outputs:
original_cl(the CL URL) and the list ofmodified_files. Update these variables inscratch/triage_state.json. - Verify Repository Contract: Confirm that the target CL belongs strictly to
the
chromium/srcrepository. If the CL targets recipe or infrastructure repositories (e.g.infra/build), stop immediately and inform the user that recipe triage is out of scope for this playbook. - State: Update
original_clandmodified_filesinscratch/triage_state.json. - Note: If the CL URL cannot be extracted from
bug_detailsor is not provided, prompt the user to get it.
Step 3: Validate Against Bug Description & Filter Target Files
- Action: Cross-reference
modified_files(retrieved in Step 2) with the developer's bug report (found inbug_detailsinscratch/triage_state.json). - Filter Files & Metrics:
- Files: Identify which specific application source files (
.cc,.cpp,.java, etc.) the developer is complaining about inbug_details. Settarget_filesto only contain those specific files matching the bug report. If the bug report does not specify files, keep all modified application source files frommodified_files. - Metrics: Identify which specific coverage metrics the developer is
complaining about in
bug_details. Setmetrics_of_concernto only contain matching metrics from the standard options ("absolute_coverage","incremental_coverage","absolute_unit_tests_coverage","incremental_unit_tests_coverage"). If the bug report does not specify, keep all four standard coverage metrics.
- Files: Identify which specific application source files (
- Required Outputs: The resolved
target_fileslist andmetrics_of_concernlist under investigation. Update bothtarget_filesandmetrics_of_concerninscratch/triage_state.json.
Step 4: Read Coverage Metrics
- Action: Fetch code coverage metrics and unexecuted line details strictly
for
target_filesusing a sub-agent. - Invocation: Launch a
metrics_readersub-agent equipped with the experimental-read-code-coverage-metrics skill. You MUST explicitly passcl_url(readoriginal_clfromscratch/triage_state.json),artifact_path: "scratch/original_metrics.json", andtarget_files(read fromscratch/triage_state.json) in the subagent prompt. - Gate: Wait for the
metrics_readersub-agent to complete before proceeding. - Required Outputs: The subagent records coverage metrics and unexecuted
code lines strictly for
target_filesinscratch/original_metrics.json.
Step 5: Map Source Files to Test Suites
- Action: Identify which test suites (e.g.
unit_tests,browser_tests) compile the modified files. - Skill: Use the experimental-test-suite-mapper skill.
- Required Outputs: The list of mapped
test_suitesfor the target files. Updatetest_suitesinscratch/triage_state.json.
Step 6: Identify Builders of Concern
- Action: Identify the initial list of
builders_of_concernbased on the file language:- Java Files: Use only the Android try builders:
android-arm64-rel,android-x86-rel,android-12-x64-rel,android-desktop-x64-rel(android-internal-desktop-x64-relfor internal developers),android-cronet-x64-dbg-14-tests
- JavaScript / TypeScript (WebUI) Files: Use only the JS coverage try
builders:
linux-js-coverage-rel,chromeos-js-coverage-rel
- C++ Files: Use all standard platform builders (none are ruled out):
- Linux:
linux-rel - Mac/iOS:
mac-rel,ios-simulator,ios-simulator-full-configs - Windows:
win-rel - ChromeOS:
linux-chromeos-rel - Android:
android-arm64-rel,android-x86-rel,android-12-x64-rel,android-desktop-x64-rel(android-internal-desktop-x64-relfor internal developers),android-cronet-x64-dbg-14-tests
- Linux:
- Java Files: Use only the Android try builders:
- Note on Android Desktop Configurations:
- The public
android-desktop-x64-relCQ bot and its internal equivalentandroid-internal-desktop-x64-relboth inherit their base build configuration frominfra/config/subprojects/chromium/try/tryserver.chromium.android.desktop.star(via theequivalent_builderproperty). - For specific test suite configurations on internal Clank bots, check the
*.pylfiles insidehttps://chrome-internal.googlesource.com/clank/internal/apps/+/HEAD/build/bot.
- The public
- Required Outputs: The initial list of
builders_of_concern. Updatebuilders_of_concerninscratch/triage_state.json.
Phase 2: Audit Configuration Patching
Step 7: Check & Patch Test Suite Execution on Builders
- Action: For each builder in
builders_of_concern:- Verify if the mapped test suites are configured to run on it.
- If a mapped test suite is NOT configured to run on the builder, evaluate
whether the test suite would be compatible to run with it. At your
discretion, if compatible:
- Create a new branch from main (e.g.,
verify-coverage-fix) iffix_branch_nameisNone, and updatefix_branch_nameinscratch/triage_state.json. Check outfix_branch_name. - Add the test suite configuration to the builder in Starlark.
- Regenerate the configuration files:
lucicfg generate infra/config/main.star
- Create a new branch from main (e.g.,
- Required Outputs: Confirmation that test suites are configured (or added)
on all
builders_of_concern, and updatefix_branch_nameinscratch/triage_state.json(if created).
Step 8: Audit Builder Configurations (Parallel Audit)
- Action: Audit the builder configurations for all
builders_of_concernin parallel by invoking a specialized sub-agent for each builder. - Invocations: Launch a
builder_auditorsub-agent for each builder inbuilders_of_concern. Equip each sub-agent with theexperimental-code-coverage-config-validatorskill and theexperimental-ci-only-validatorskill. When launching each sub-agent, instruct it to:- Pass
builders_of_concernandlanguage_context("cpp","objc","rust","java","js", or"ts", inferred fromtarget_files) toexperimental-code-coverage-config-validatorto validate GN arguments. - Run
experimental-ci-only-validatorto verify if any of the mappedtest_suites(fromscratch/triage_state.json) are barred byci_only = Trueon that builder (checking all mapped test suites, not just one).
- Pass
- Gate: Wait for all
builder_auditorsub-agents to return their results before proceeding to Phase 3. - Resolution: If any sub-agent reports GN argument validation failures or
that any test suite is barred by
ci_only = True:- Apply all suggested GN arg fixes to the Starlark files.
- Remove the
ci_only = Trueflag for any barred test suite on that builder in Starlark. - Commit all configuration fixes together in a single commit to the
fix_branch_namebranch (creating the branch from main iffix_branch_nameisNone). - Regenerate the configuration files:
lucicfg generate infra/config/main.star
- Required Outputs: Confirmation that GN arguments and
ci_onlysettings are valid across allbuilders_of_concernand all mappedtest_suites. If configuration fixes were applied, updatefix_branch_nameinscratch/triage_state.json(if created).
Step 9: Compile Configuration Fixes Summary
- Action: Compile a summary of all configuration audits and fixes performed
in Step 7 and Step 8 across all
builders_of_concern. - Required Outputs: Create a markdown artifact at
scratch/config_summary.mddocumenting whether any test suites were added or patched, any GN arguments were corrected, or anyci_onlyflags were removed (or confirming that existing builder configurations were valid without changes).
Phase 3: Prepare Experiment
Step 10: Analyze Required CLs & Branches
- Action: Determine CL strategy based on
fix_branch_name:fix_branch_nameis NOT None: We need two branches:- An Experimental Branch (already exists as
fix_branch_name). - A Baseline Branch (create a new branch from main, e.g.
verify-coverage-baseline, and setcontrol_branch_nameto this branch).
- An Experimental Branch (already exists as
fix_branch_nameis None: We only need one branch:- A Baseline Branch (create a new branch from main, e.g.
verify-coverage-baseline, and setcontrol_branch_nameto this branch).
- A Baseline Branch (create a new branch from main, e.g.
- Required Outputs: The branch name
control_branch_nameand, if it wasn't already created,fix_branch_name. Updatecontrol_branch_nameandfix_branch_nameinscratch/triage_state.json.
Step 11: Generate CLs (Parallel CL Generation)
- Action: Generate the baseline and, if necessary, experimental CLs in parallel by invoking a specialized sub-agent for each CL.
- Invocations: Launch up to two
cl_generatorsub-agents in parallel (one for the baseline branch and one for the experimental fix branch if created). Equip both sub-agents with theexperimental-code-coverage-verification-prepskill to inject dummy comments, optimize Starlark configs, and upload CLs to Gerrit. When launching eachcl_generatorsub-agent, instruct it to read its primary inputs (target_files,builders_of_concern,test_suites,bug_id) directly fromscratch/triage_state.json. You MUST passrole("Control"or"Fix"). For"Fix"runs, also pass in a summary of the configuration and recipe fixes applied (e.g. fromscratch/config_summary.md,scratch/swarming_summary.md, orscratch/local_debugging.md) so the sub-agent can generate a descriptive CL description. - Gate: Wait for all
cl_generatorsub-agents to return their results before proceeding to Step 12. - Required Outputs: The Gerrit CL URLs for
control_clandfix_cl(if applicable). Updatecontrol_clandfix_clinscratch/triage_state.json. - Note: If
control_clis already instantiated with a non-null value inscratch/triage_state.json(such as during subsequent iterations or if pre-populated), skip generatingcontrol_cland do NOT launch acl_generatorsub-agent for the control run. Only launch the sub-agent for the experimental fix branch (role: "Fix").
Step 12: Trigger Try Jobs (Parallel Execution)
- Action: Launch try jobs on the target builders for the prepared CLs in parallel using sub-agents.
- Invocations: Launch up to two
build_invokersub-agents in parallel (one for the control CL and one for the fix CL if created). Equip both sub-agents with theexperimental-code-coverage-build-invokerskill. When launching each sub-agent, instruct it to:- For
control_cl: passbuilders_of_concernand the designatedcontrol_clURL (read fromscratch/triage_state.json). - For
fix_cl(if created): passbuilders_of_concernand the designatedfix_clURL (read fromscratch/triage_state.json).
- For
- Gate: Wait for all
build_invokersub-agents to return their results before proceeding to Phase 4. - Required Outputs: The Buildbucket build links/URLs for all triggered try
jobs, mapped to their builder name inside
control_builds(for control CL) orfix_builds(for fix CL). Updatecontrol_buildsandfix_buildsinscratch/triage_state.json. - Note: If
control_buildsis already instantiated with non-empty values inscratch/triage_state.json(such as during subsequent iterations or if pre-populated), skip triggering try jobs forcontrol_cland do NOT launch abuild_invokersub-agent for the control run. Only trigger try jobs forfix_cl.
Phase 4: Explore and Validate Experiment Results
Step 13: Inspect Builds (Parallel Inspection)
- Action: Monitor and verify the status of all triggered builds in parallel by invoking a specialized sub-agent for each builder.
- Invocations: Launch a
build_inspectorsub-agent for each target builder inbuilders_of_concern. Equip each sub-agent with theexperimental-code-coverage-build-inspectorskill. When launching each sub-agent, the parent agent MUST passbuilder_name(e.g.,"linux-rel"). Instruct each sub-agent to:- Read
control_builds[builder_name]andfix_builds[builder_name](if present) fromscratch/triage_state.json. - Call
experimental-code-coverage-build-inspectorfor the build link(s), passinglanguage_context("cpp","objc","rust","java","js", or"ts"),test_suites,metrics_of_concern, andtarget_files(read fromscratch/triage_state.json), to verify GN arguments, inspect merge script logs for the specific target test suites, and check whether coverage data is generated fortarget_filesin the HTML report and downloaded JSON coverage artifacts. (Note: It is also possible that we only have a control build with no fix build. Iffix_builds[builder_name]is omitted, inspect only the control build). - Generate a comparative inspection artifact report at
scratch/inspection_<builder_name>.mddocumenting whether coverage processing succeeded on the fix build compared to the control (or documenting baseline results if no fix build exists).
- Read
- Gate: Wait for all
build_inspectorsub-agents to return their results. - Polling Mandate: If any tryjob is still actively running on LUCI, instruct
the
build_inspectorsub-agent to invoke thescheduletool (CronExpression="*/10 * * * *") to check back every 10 minutes until terminal build completion before continuing to Step 14. - Required Outputs: The final status (SUCCESS/FAILURE) of each monitored
build stored in
control_buildsandfix_builds, and the generated inspection artifacts (scratch/inspection_<builder_name>.mdfor each builder of concern). Updatecontrol_buildsandfix_buildsinscratch/triage_state.json.
Step 14: Compare Builder Inspection Artifacts
- Action: Read and compare all builder inspection artifacts
(
scratch/inspection_<builder_name>.mdgenerated in Step 13) across allbuilders_of_concern. - Evaluation: Determine whether code coverage instrumentation, profile generation, and merging succeeded on the experimental fix builds compared to the baseline control builds across all target builders. (Note: It is also possible that we only had a control build with no fix build. If no fix build exists, evaluate whether the baseline control build alone successfully generated coverage).
- Required Outputs: Create an artifact called
scratch/inspection_summary.mdwhich summarizes all the information from all the inspectedscratch/inspection_<builder_name>.mdfiles across allbuilders_of_concern, confirming whether coverage generation changed/resolved due to our configuration fixes (or confirming baseline coverage generation if no fix build was triggered).
Step 15: Read Verification Coverage (Parallel Execution)
- Action: Fetch code coverage metrics for the Baseline/Control CL and, if applicable, the Experimental/Fix CL in parallel using sub-agents.
- Invocations: Launch up to two
metrics_readersub-agents in parallel (one for the control CL and one for the fix CL if created). Equip both sub-agents with the experimental-read-code-coverage-metrics skill. When launching each sub-agent, you MUST explicitly pass:- For control CL: pass
cl_url(control_cl),artifact_path: "scratch/control_metrics.json", andtarget_files(read fromscratch/triage_state.json). - For fix CL (if created): pass
cl_url(fix_cl),artifact_path: "scratch/fix_metrics.json", andtarget_files(read fromscratch/triage_state.json).
- For control CL: pass
- Gate: Wait for all
metrics_readersub-agents to complete before proceeding. - Required Outputs: The subagents fetch coverage details and store them in
scratch/control_metrics.json(for baseline control CL) andscratch/fix_metrics.json(for experimental fix CL). - Note: If
scratch/control_metrics.jsonalready exists (such as during iteration 2 or 3), skip launching themetrics_readersubagent for the control CL and only launch one forscratch/fix_metrics.json.
Step 16: Artifact Analysis & Comparison
- Action: Read and compare the coverage JSON metric artifacts across all available CL states (Original, Baseline/Control, and Experimental/Fix).
- Required Outputs: Create a summary comparing all the metric JSON files at
scratch/metrics_summary.json. - Evaluation & Resolution: Compare the uncovered line snippets and
low-coverage code areas identified in
scratch/original_metrics.jsonagainst the coverage data inscratch/control_metrics.jsonandscratch/fix_metrics.json.- If coverage increased and the specific code areas initially reported by
Gerrit as underreported are now covered (in either the baseline control or
experimental fix run), mark
statusas"FIXED"inscratch/triage_state.jsonand jump directly to Step 20 to formulate Root Cause Analysis. - Otherwise, proceed to Step 17 to investigate Swarming profile outputs.
- If coverage increased and the specific code areas initially reported by
Gerrit as underreported are now covered (in either the baseline control or
experimental fix run), mark
Step 17: Swarming Output Validation (Fast Remote Audit)
- Action: For completed tryjobs, audit remote profile generation integrity.
- Invocation: Launch a
swarming_validatorsub-agent equipped with theexperimental-code-coverage-swarming-output-validatorskill. You MUST passtest_suitesandbuild_url(read fromscratch/triage_state.json). Instruct the subagent to create a summary document of all steps taken and insights gained atscratch/swarming_summary.md.
Step 18: Local Debugging (Workstation & Universal Test Runner)
- Action: If remote profiles are corrupt or crash, reproduce locally.
- Invocation: Launch a
local_debuggersub-agent equipped with theexperimental-code-coverage-local-debuggerskill. Instruct the subagent to create a summary document of all steps taken and insights gained atscratch/local_debugging.md.
Step 19: Evaluate Additional Repairs & Iterative Experiment Loop
- Action: Read
scratch/swarming_summary.mdandscratch/local_debugging.mdto evaluate whether findings indicate that an additional fix (such as a Starlark recipe fix, GN argument change, or test harness patch) is required. - Iteration Gate & Archival:
- If further fixes are needed and the current iteration count (tracked in
scratch/triage_state.jsonunder"iteration", defaulting to1) is less than3:- Increment
"iteration"by1inscratch/triage_state.json. - Make the required Starlark/GN/code repairs directly on the same branch
(
fix_branch_name) and commit them so subsequent uploads append new patchsets to the same experimental CL (fix_cl). - Archive the current iteration's scratch verification artifacts
(
scratch/fix_metrics.json,scratch/metrics_summary.json,scratch/inspection_*.md,scratch/swarming_summary.md,scratch/local_debugging.md) by moving them intoscratch/iteration_<N-1>/. Do NOT archivescratch/control_metrics.jsonas it is preserved across iterations. Also copyscratch/triage_state.jsontoscratch/iteration_<N-1>/triage_state.jsonto snapshot prior build URLs. Then, in the activescratch/triage_state.jsonfile, reset fix-related variables ("fix_builds": {}) while preserving Phase 1 metadata and control baseline variables ("control_builds","control_cl"). - Jump back to Step 10 to generate/update CL patchsets and run the new verification experiment.
- Increment
- Otherwise (if no further repairs are indicated or maximum iterations reached), proceed to Step 20.
- If further fixes are needed and the current iteration count (tracked in
Step 20: Generate Final Hypothesis & Resolution Summary
- Action: Invoke a
hypothesis_summarizersub-agent to synthesize all debugging iterations, tryjob links, metrics, and findings into a concise final report. - Invocation: Launch a
hypothesis_summarizersub-agent. Instruct it to:- Read
scratch/triage_state.json, current iteration scratch files, and any archivedscratch/iteration_*/verification files. - Create a concise final summary report at
scratch/final_hypothesis_summary.mddocumenting:- Summary of the issue the bug reporter is listing
- All debugging iterations performed (including full Buildbucket tryjob URLs/links for baseline control and experimental fix builds across iterations).
- The exact debugging steps taken across Phase 1 through Phase 4.
- The synthesized Root Cause Analysis hypothesis (addressing Subsystem Context, Failure Mechanism, and Fix Rationale). (Note: Keep the document concise and well-structured with clear tables and concise bullet points to avoid overflowing).
- Read
- State Save: Write the structured RCA dictionary into
scratch/triage_state.jsonunder the"rca"variable (referencing templates/triage_state_template.json for the variable contract). - Resolution: Set terminal
statusto"FIXED"or"ESCALATED"inscratch/triage_state.json.