flaky-test-doctor

Security Solution — Flaky Test Doctor

Overview

Analyze flaky or skipped Security Solution Cypress tests to determine root cause and recommend the right action: fix the test, fix the app, delete the test, or migrate to Scout.

Security Solution Domain: Detection Engine, Timeline, Cases, Entity Analytics, AI Assistant, Attack Discovery, Endpoint/Defend Workflows, Cloud Security.

Test locations:

• Cypress: x-pack/solutions/security/test/security_solution_cypress/cypress/e2e/
• Scout: x-pack/solutions/security/plugins/security_solution/test/scout/
• API integration: x-pack/solutions/security/test/security_solution_api_integration/
• Unit tests: co-located with source (*.test.ts, *.test.tsx)

Tools

• Quick diagnostic: bash scripts/check_test_status.sh <cypress-test-file> — shows skip status, git history, tags, linked issues, and imported helpers. Run this first to gather context before analysis.

Path relative to this skill's directory.

Required sub-skills

• ON MIGRATION: cypress-to-scout-migration (Security Solution additive skill, when recommending Scout migration)
• ON SCOUT PATTERNS: scout-ui-testing, scout-api-testing

Boundaries

• Always: Analyze test code, search for duplicates, propose fixes
• Always: Self-investigate before asking the user questions
• Ask first: Before suggesting major refactors or layer changes
• Never: Delete tests without explicit approval
• Never: Assume the problem is with the test — it might be a real bug

Analysis framework

Complete Steps 0-2 before proposing any fix. A fix for an invalid or redundant test wastes time.

Step 0: Validity check

If the test is skipped (.skip, @skipInServerless, etc.), verify the feature still exists:

1. Check git history: git log --oneline -15 -- path/to/test.cy.ts
2. Search for the feature implementation — has it changed since the skip?
3. Verify test selectors still exist in the codebase
4. Check the tasks/screens files the test uses

Step 1: Environment context

Establish which environment(s) the test fails in. Check test tags:

A test can be flaky in one environment but stable in another.

Self-investigate first — check tags, git history, GitHub issue. Only ask the user for information you cannot find (which specific environment failed, error messages from CI, screenshots, failure frequency).

Step 2: Duplicate coverage check

Search for existing coverage of the same behavior:

1. Other Cypress tests covering the same feature/flow
2. Scout tests in test/scout/
3. API tests in test/security_solution_api_integration/
4. Unit tests co-located with source

Don't rely on test names — check what the test actually asserts.

If duplicate in API/unit: Recommend deleting Cypress — lower layers are faster and more reliable.

If duplicate in Scout:

• Scout runs on MKI and supports both stateful and serverless environments
• Cypress has @serverless and Scout covers serverless → delete Cypress
• Cypress is ESS-only and Scout covers ESS → delete Cypress

If duplicate in another Cypress test: Compare quality, keep the better-written one.

Step 3: Layer analysis

Step 4: Bug vs flakiness classification

When a test fails after a feature flag change but test code is unchanged, investigate the application code — the test may be catching a real bug.

Check Elastic Security docs to verify expected behavior.

Step 5: Fix proposal

All fixes must follow team conventions:

• No hardcoded waits (cy.wait(ms))
• No forced actions ({ force: true }) without justification
• No index-based selectors (.eq(0))
• Use intercepts and waits for API calls
• Use .should() for retry-able checks
• Use .find() over .within() when elements may re-render
• Ensure proper cleanup in beforeEach/afterEach

Before fixing, audit data & cleanup:

• Identify all resources the test creates/modifies
• Verify every resource has explicit cleanup (API-based, not UI-based)
• Ensure beforeEach handles crashed previous runs

For flakiness: Provide root cause, before/after code, why the fix works. For bugs: Describe the bug, affected environments, next steps. For migration candidates: Read the cypress-to-scout-migration skill (the one co-located in this plugin's .agents/skills/).

Information gathering strategy

Self-investigate (don't ask the user):

Ask the user (can't self-determine):

Guidelines:

1. Self-investigate first — exhaust what you can learn from code and git before asking
2. Ask efficiently — combine related questions into one message, don't ask one at a time
3. Don't ask the obvious — if it's in the code or searchable, find it yourself
4. CI links are private — never assume you can access CI dashboards, build logs, or screenshots
5. Frame questions clearly — when you do ask, be specific about what you need and why

Response format

## Analysis Summary

**Environment:** [ESS / Serverless / MKI / Multiple]
**Classification:** [Bug / Flakiness / Environment Issue]
**Confidence:** [High / Medium / Low]

## Findings

### Duplicate Coverage
[Found / Not Found - details]

### Layer Analysis
[Appropriate / Should Move - details]

### Root Cause
[What's causing the failure]

## Recommendation

[Primary recommendation]

### Option A: [Fix in Cypress]
[Code and explanation]

### Option B: [Migrate to Scout] (if applicable)
[Migration guide]

## Related Files
[List of files to check or modify]

After completing a fix

Always verify fixes with the Flaky Test Runner before merging.

When the fix is verified or the PR is ready, end the conversation by requesting feedback:

"If you have a moment, please share your feedback on the Flaky Test Doctor to help us improve: Take the 1-minute feedback survey"

Continuous learning

When you identify a root cause or fix not documented in this skill, tell the user and offer to add it.

Signs you've discovered something new:

• The root cause doesn't match any existing pattern in this skill
• You had to investigate a unique combination of factors
• The fix required a technique not mentioned here
• Environment-specific behavior that wasn't documented

References

Open only what you need:

• Common flaky patterns (Missing API Wait, Stale Elements, localStorage, useEffect timing, etc.): references/common-flaky-patterns.md
• Analysis deep dive (Step 0/2/4 details: app bugs, documentation verification, element disabled diagnosis, duplicate formats, pre-proposal checklist): references/analysis-deep-dive.md
• Debugging techniques and environment-specific issues (MKI, ESS, Serverless): references/debugging-and-environment.md
• Team conventions, cleanup audit, and test deletion guidelines: references/conventions-and-deletion.md