fleet-flaky-test-doctor

Fleet Flaky Test Doctor

Overview

Analyze failing or skipped Fleet tests to determine root cause, then act:

• Investigation (all modes): produce a structured report with verdict, evidence, and root cause.
• Fix / delete verdicts (Modes 1 and 3): after the report, apply the fix, run the affected tests locally, then ask for approval before opening a PR.
• Other verdicts: present the report and let the human act (close issue, trigger re-run, escalate).

Human-gated on merging and issue closure: never merge PRs, never close GitHub issues directly.

Fleet test domains (non-exhaustive): Agents, Agent Policies, Package Policies, EPM (Elastic Package Manager), Policy Secrets, Knowledge Base, Fleet Tasks, Settings, Space Awareness, Outputs, Enrollment — and any other area owned by the Fleet team. When a test path or issue title refers to a Fleet-owned plugin or config, treat it in scope even if the domain isn't listed here.

---

Pick your mode

Relationship — fan-out, not chain:

Mode 2 (one run, ~30 min)
  ├─ close-stale rows    → batch close comments          (no further mode needed)
  ├─ flaky-rerun rows    → re-run by test type (see §Flaky Runner Support)
  ├─ fix-cluster rows    → Mode 3 per cluster
  ├─ escalate/cross-team → human assignment / label change
  └─ residual singletons → Mode 1 (a handful only)

---

Required sub-skills

• ON FTR API INTEGRATION TESTS: fleet-ftr-testing — which config.*.ts to use, Docker setup, FLEET_PACKAGE_REGISTRY_PORT, --grep. Delegate to it; do not duplicate.
• ON LOCAL KIBANA SETUP: kibana-local-dev — yarn es snapshot, yarn start, SSL flags, port conflicts.

---

Prerequisites

Before running any mode, verify these are in place:

If gh is not authenticated, check_fleet_test_status.sh will still run but the Sibling Open Issues and Recent Merged PRs blocks will be empty — cluster detection and stale-fix signals will be missing. Authenticate first with gh auth login.

---

Step 0a — Test type classification (run first, before everything else)

Identify the test type from the file path. Each type has different reproduction commands, environment signals, and fix conventions. Getting this wrong means applying the wrong advice.

Once type is known, Steps 1, 3, and 5 branch accordingly (see each step).

---

Quick diagnostic

Run this first for any Mode 1 or Mode 3 investigation to gather context:

bash scripts/check_fleet_test_status.sh <test-file-path>

Path relative to the kibana repo root. Output: skip status, linked issues, FTR config file, git history, skip/unskip history, sibling open issues, staleness signal, test count.

---

Mode 1: Single-issue triage

Complete Steps 0a → 6 in order. Do not skip steps.

Step 0 — Validity check

If the test is skipped (.skip, describe.skip, it.skip), verify the feature still exists:

1. git log --oneline -15 -- path/to/test.ts
2. Search for the feature implementation — has it changed since the test was skipped?
3. Verify relevant API endpoints / UI components still exist in the codebase.

Step 1 — Environment context

Determine which environment(s) the test runs in. Fleet uses config files, not in-file tags (@ess/@serverless tags do not apply).

A test can be broken in stateful but pass in serverless, or vice versa — always note which config surfaced the failure. If the failing issue title references a serverless pipeline (e.g. Serverless Oblt or Serverless Security), look in the solution serverless paths first.

Step 2 — Duplicate coverage check

Search for existing coverage of the same behavior before proposing a fix:

1. FTR API integration tests covering the same endpoint/behavior
2. Jest server integration tests
3. Jest unit tests co-located with source
4. Scout Playwright tests

Don't rely on test names — check what the test actually asserts.

If lower-layer coverage exists: recommend moving to the cheaper layer (unit > integration > FTR API > Scout). Lower layers are faster and more reliable.

Step 3 — Layer analysis (type-dependent)

Step 4 — Bug vs flakiness classification

When a test fails after a code change but test code is unchanged, investigate the application code first.

Step 5 — Fix proposal

Follow Fleet test conventions from references/conventions-and-deletion.md for the test type identified in Step 0a.

Before proposing a fix, audit data & cleanup:

• Identify all resources the test creates (agent policies, package policies, packages, SO documents, indices)
• Verify every resource has explicit cleanup in before/after or beforeEach/afterEach
• Ensure setup handles crashed previous runs (clean before, not only after)

For flakiness: provide root cause, before/after code, and why the fix works. For bugs: describe the bug, affected environments, and next steps. For deletion candidates: document which lower-layer coverage makes deletion safe.

Step 6 — Issue disposition

Determine the verdict, then act:

Decision rubric (first match wins):

1. close-stale — test file/name gone; feature removed; duplicate of newer issue.
2. flaky-rerun — passes locally 3× AND issue >60 days old AND no recent test-file commits. OR before all/after each hook failure.
3. fix — fails locally and reproduces the signature.
4. delete-test — .skip'd >180 days AND no champion AND lower-layer coverage exists.
5. escalate — otherwise.

---

Mode 1: Response format

Always produce the investigation report first. For fix and delete-test verdicts, follow it by opening a PR.

## Analysis: #<N> — <title>

**Test type:** FTR API integration | Jest server integration | Jest unit | Scout Playwright | Cross-team
**Environment:** stateful | serverless | both
**Classification:** Bug | Flakiness | Environment Issue | App Bug
**Verdict:** close-stale | flaky-rerun | fix | delete-test | escalate
**Confidence:** High | Medium | Low

## Findings

### Duplicate Coverage
[Found / Not Found — details]

### Layer Analysis
[Appropriate / Should Move — details]

### Root Cause
[What's causing the failure]

### Reproduction
[Ran locally N×, M failed — or: not run, reason]

### Sibling issues
[Other open issues referencing the same test file, from check_fleet_test_status.sh output]

## Recommendation

[Primary recommendation]

### Option A: [Fix / Delete / Close / Rerun]
[Diff or command or closing comment text]

### Option B: [Alternative, if applicable]
[Details]

## Related Files
[List of files to check or modify]

After the report — verdict-dependent next step:

---

Mode 2: Backlog audit

Input

gh issue list --repo elastic/kibana \
  --label "Team:Fleet" --label "failed-test" \
  --state open --limit 200 \
  --json number,title,labels,createdAt,updatedAt

Or a list of issue numbers provided by the user.

Per-issue skim (Mode 2 only — not the full Mode 1 analysis)

For each issue:

1. Extract test file path from the issue title (Kibana auto-reporter format: <Pipeline>.<path> - <describe chain> <test name>).
2. Check if file/test still exists: grep -r "<test name>" "<file>" (or NOT FOUND).
3. Check last commit on the file: git log -1 --format="%ai %s" -- "<file>".
4. Check issue's updatedAt — if only bumped by the auto-reporter, note it.
5. Check issue labels — note blocker if present, and record any version labels (e.g. v9.3.0, v9.4.0) for use in PR backport labels later.
6. Look for sibling issues: base=$(basename "<file>" .ts); gh issue list --repo elastic/kibana --label "Team:Fleet" --label "failed-test" --state open --search "$base".
7. Assign a quick verdict: close-stale | flaky-rerun | fix | delete-test | escalate.

Output format

First, the tracking table:

| # | Title (short) | Test type | File exists? | Last file commit | Labels | Quick verdict | Cluster |
|---|---|---|---|---|---|---|---|
| 224362 | Fleet preconfig reset — all | Jest server int. | ✅ | 2025-06-18 | — | fix | A |
| 246983 | EPM KB returns content | FTR API | ✅ | 2025-12-10 | 🔴 blocker `v9.3.0` `v9.4.0` | fix | B+F+M |
| 247566 | APM input_only_package | Cross-team | ✅ | 2026-01-05 | — | escalate | O |
...

The Labels column must capture:

• 🔴 blocker — if the issue has the blocker label (higher priority, test is currently skipped and blocking a release branch)
• ⏭ skipped — if the issue has the skipped-test label
• Version labels (v9.3.0, v9.4.0, etc.) — record all of them; the PR fixing this issue must carry the same version labels for backports

Then, the cluster summary:

## Cluster summary

| Cluster | Issues | Suggested action |
|---|---|---|
| A — Preconfig/setup (Jest server int.) | 14 | Mode 3 fix — one shared cause |
| B+F+M — Knowledge Base (cross-layer) | 5 | Mode 3 fix — same KB regression at 3 layers |
| J — Policy secrets (FTR API) | 6 | Mode 3 fix |
...

## Recommended next steps

Lane 1 (batch close): #NNN, #NNN, ... (~N issues)
Lane 2 (flaky-rerun): #NNN, #NNN, ... (~N issues) — note jest issues: local only
Lane 3 (Mode 3 clusters, ordered by highest issue number in the cluster — most recently filed first): <cluster> → <cluster> → ...
Lane 4 (handoffs): #247566 → APM team; #246569, #265666 → Security team

---

Mode 3: Cluster fix

Input

A cluster name (e.g. A) or a list of issue numbers copy-pasted from the Mode 2 cluster summary.

Analysis

1. Collect all test files referenced by the cluster's issues.
2. Read all of them together — look for the shared fixture, shared setup call, shared assertion, or shared helper that is common across all failures. This is what's invisible from any single issue.
3. Run check_fleet_test_status.sh on the most representative test (usually the oldest issue's test).
4. Reproduce one representative test locally to confirm the failure.
5. Identify the fix at the shared layer (the fixture, helper, setup hook, or shared config that affects the whole cluster), not at the individual test level.
6. If the shared-cause hypothesis fails (each test fails for a different reason), return: "This cluster doesn't share a root cause — run Mode 1 on these issues individually: #N, #N, ..."

Output format

Always produce the investigation report first, then open a PR with the fix.

## Cluster fix: <cluster name>

**Issues:** #N, #N, ... (N total — this fix should close all of them)
**Test type:** <type>
**Shared root cause:** <one paragraph>
**Reproduced locally:** yes (N of N runs failed) | no (reason)

## Fix

**File to change:** <path>

**Before:**
```ts
// existing code

After:

// fixed code

Why this fixes the whole cluster:

Verification matrix

Run all of the following and confirm green:

• <reproduction command for test 1>
• <reproduction command for test 2> ...

Issues closed by this PR

Fixes #N Fixes #N ...

**After the report:** Apply the fix, present the diff, run the verification matrix locally, report the result, then **ask for approval before opening a PR**. Use a short summary of the investigation report as the PR body (root cause in 1–2 sentences, what changed and why). Add a `Fixes #N` line for **every** issue in the cluster — GitHub auto-closes each linked issue on merge when this keyword is present (use `Fixes` not `Closes` to match Kibana convention). Apply standard Fleet PR labels (`Team:Fleet`, `release_note:skip`, `backport:version`) **plus the union of all version labels from the cluster's issues** — if any issue has `blocker`, those version labels are especially important as they indicate release branches that need the fix backported.

---

## Flaky Runner Support

The Buildkite flaky-test pipeline supports only FTR and Scout. Jest has **no CI runner** — local looping is the only option.

| Test type | Buildkite flaky runner | Re-run mechanism |
|---|---|---|
| FTR API integration | ✅ `type: "ftrConfig"` | [ci-stats.kibana.dev/trigger_flaky_test_runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner) with `config.*.ts` path |
| Scout Playwright | ✅ `type: "scoutConfig"` | Same UI, Scout config path |
| Jest server integration | ❌ Not supported | `for i in {1..25}; do node scripts/jest_integration --testPathPattern=<path> && echo "PASS $i" || echo "FAIL $i"; done` |
| Jest unit | ❌ Not supported | `for i in {1..10}; do yarn jest <path> && echo "PASS $i" || echo "FAIL $i"; done` |

For jest: if it passes 9/10 or more runs locally after a long silence, that's sufficient evidence to unskip. If it consistently fails, it's a `fix` verdict, not `flaky-rerun`.

---

## Information gathering strategy

**Self-investigate first — don't ask the user:**

| Information | How to find it |
|---|---|
| Feature still valid? | Search codebase, check recent commits |
| Duplicate coverage? | Grep for the assertion/behavior in other test files |
| When was test skipped? | `git log -S '.skip' -- "<file>"` |
| Which FTR config runs it? | `base=$(basename "<file>" .ts); grep -rl -- "$base" x-pack/platform/test/fleet_api_integration/` |
| Sibling open issues? | `check_fleet_test_status.sh` output, or `base=$(basename "<file>" .ts); gh issue list --search "$base"` |
| Is the issue stale? | Compare `git log -1 --format=%ai -- "<file>"` with issue `updatedAt` |
| Was it already fixed? | `base=$(basename "<file>" .ts); gh pr list --repo elastic/kibana --search "$base" --state merged` |
| Linked GitHub issue? | Look at the `// Failing:` or `// FLAKY:` comment above the skip |

**Ask the user (can't self-determine):**

| Information | Why |
|---|---|
| Which environment failed? | CI links are private |
| Error message from CI | Not in code |
| Consistent or intermittent? | Requires multiple runs |
| Additional CI context | Private infrastructure |

**Guidelines:**
1. Self-investigate first — exhaust what you can learn before asking.
2. Ask efficiently — combine related questions into one message.
3. Never assume you can access CI dashboards, build logs, or screenshots.

---

## Boundaries

- **Always:** analyze test code, search for duplicates, produce the investigation report before taking any action.
- **Always:** self-investigate before asking the user.
- **For `fix` and `delete-test` verdicts:** apply the changes, present the diff, run the affected tests locally, report the result, and **ask for approval before opening a PR**.
- **Ask first:** before suggesting moving a test to a different layer (may affect coverage intentionally).
- **Never:** open a PR without explicit human approval — the report is the gate, approval is the trigger.
- **Never:** assume the problem is with the test — it might be a real app bug.
- **Never:** close GitHub issues directly — draft the comment, human posts it.

---

## Steady-state hygiene (after burn-down)

Once the backlog is cleared, use Mode 2 monthly:

1. **Monthly Mode 2 run** on open `Team:Fleet` + `failed-test` issues (~30 min).
2. **Skip budget:** no `.skip` older than 90 days — fix or delete.
3. **DoD for new skips** (PR review checklist):
   - Comment above the skip links a GitHub issue.
   - Issue has `Team:Fleet` + `skipped-test` labels.
   - Issue has a **named assignee** (not just the team label — unassigned issues rot).

---

## After completing a fix

Always verify with the appropriate re-run mechanism (see §Flaky Runner Support) before merging.

When the fix is verified or PR is ready, request feedback:

> "If you have a moment, please share feedback to help improve this skill — what was useful, what was off, anything it missed."

---

## Continuous learning

When you identify a root cause or fix pattern not documented here, tell the user and offer to add it to `references/common-flaky-patterns.md`.

**Signs you've discovered something new:**
- The root cause doesn't match any pattern in `references/common-flaky-patterns.md`.
- You needed a technique not mentioned in `references/conventions-and-deletion.md`.
- A cluster turned out not to share a root cause despite appearing to.
- Environment-specific Fleet behavior (e.g. space-awareness affecting test isolation) that wasn't documented.

---

## References

Open only what you need:

- Fleet-specific failure patterns (shared SO state, package registry timing, policy secrets race conditions, etc.): `references/common-flaky-patterns.md`
- Fleet test conventions, cleanup audit, deletion guidelines, flaky process: `references/conventions-and-deletion.md`
- FTR test execution: `fleet-ftr-testing` sub-skill
- Local Kibana setup: `kibana-local-dev` sub-skill