OpenParable
Skip to content
← Back to skills

Flaky Test Detector

Testing & Quality
flaky-test-detector

Flaky Test Detector is an Agent Skill for testing & quality published in the dotnet/dotnet repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Detect flaky tests by scanning recent AzDo CI builds for test failures recurring across multiple unrelated PRs. Use when investigating intermittent failures, CI instability, deciding which tests to quarantine, or checking if RunTestCasesInSequence no-ops are causing parallel-safety issues.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/dotnet/dotnet/blob/HEAD/src/fsharp/.github/skills/flaky-test-detector/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/flaky-test-detector, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Flaky Test Detector

Identifies tests that fail intermittently across unrelated PRs — a strong signal of flakiness rather than a genuine regression. Also cross-references with existing fix PRs.

When to Use

  • Investigating CI instability ("is this test failure my fault or flaky?")
  • Periodic hygiene: finding tests to quarantine or fix
  • Before marking a test as Skip = "Flaky" — confirm it actually is flaky
  • Checking if RunTestCasesInSequence (a no-op in xUnit 2) is masking parallelism bugs

How It Works

  1. Queries Azure DevOps builds API directly for recent failed fsharp-ci PR builds
  2. Extracts test failures from each build via Get-BuildErrors.ps1
  3. Aggregates by test name across distinct PRs
  4. Cross-references with GitHub PRs that may address the flaky tests
  5. Tests failing in 3+ distinct PRs are flagged as flaky

Usage

Quick scan (last 14 days, 50 builds, threshold = 3)

pwsh .github/skills/flaky-test-detector/scripts/Get-FlakyTests.ps1

Custom parameters

# More aggressive: 2+ PRs over 7 days
pwsh .github/skills/flaky-test-detector/scripts/Get-FlakyTests.ps1 -MinPRFailures 2 -DaysBack 7

# Wider net: 100 builds over 30 days  
pwsh .github/skills/flaky-test-detector/scripts/Get-FlakyTests.ps1 -MaxBuilds 100 -DaysBack 30

Parameters

ParameterDefaultDescription
-MaxBuilds50Maximum number of failed builds to scan from AzDo
-MinPRFailures3Min distinct PRs a test must fail in to be flagged
-DaysBack14Only consider builds within this time window
-DefinitionId90AzDo pipeline definition ID (90 = fsharp-ci)
-Orgdnceng-publicAzure DevOps organization
-ProjectpublicAzure DevOps project

Output

The script produces:

  1. Console report with ranked flaky tests, PR numbers, job names, and sample errors
  2. Structured objects (PowerShell) for programmatic consumption

Interpreting Results

  • DistinctPRs ≥ 5: Almost certainly flaky — consider quarantining immediately
  • DistinctPRs = 3–4: Likely flaky — investigate root cause
  • DistinctPRs = 2: Possibly flaky or a shared dependency issue — monitor

Follow-up Actions

After identifying a flaky test:

  1. Check if there's already a GitHub issue for it
  2. If not, file one with the Area-flaky-test label
  3. Consider marking with [<Fact(Skip = "Flaky: #ISSUE")>] if it blocks CI
  4. Fix the root cause (timing, file locking, thread safety, etc.)