retry_flaky_tests

Retry Flaky Tests Skill

Automatically handles flaky CI tests by intelligently retrying only failed required checks on GitHub PRs. Saves time by avoiding manual monitoring and selective retrying.

Usage

/retry-flaky-tests [--pr_number 123] [--wait_for_completion true] [--max_retries 3] [--poll_interval 30]

Key Features

• Smart Filtering: Only retries required checks (ignores optional ones)
• Wait Logic: Can wait for in-progress runs to complete before retrying
• Batch Processing: Handles multiple failed checks in one command
• Status Awareness: Distinguishes between failed, in-progress, cancelled states
• Retry Limiting: Prevents infinite retry loops with max attempts
• Real-time Feedback: Shows progress and results throughout process

Process

Step 1: Identify Target PR

If --pr_number provided:

gh pr view {pr_number} --json number,headRefName,baseRefName

If not provided, detect from current branch:

# Get current branch
current_branch=$(git branch --show-current)

# Find PR for this branch  
gh pr list --head "$current_branch" --json number,title,headRefName

Validation:

• Ensure PR exists and is open
• Confirm we have appropriate permissions
• Extract PR number, head branch, base branch for later use

Step 2: Get Current Check Status

# Get all check runs for the PR
gh pr checks {pr_number} --json name,status,conclusion,detailsUrl

# Get workflow runs with more detail
gh run list --pr {pr_number} --json databaseId,name,status,conclusion,workflowName

Parse results to categorize checks:

• ✅ Passed: status: "completed", conclusion: "success"
• ❌ Failed: status: "completed", conclusion: "failure"
• ⏸️ In Progress: status: "in_progress" or status: "queued"
• ⏭️ Skipped/Cancelled: conclusion: "skipped" or conclusion: "cancelled"

Step 3: Identify Required vs Optional Checks

Get branch protection rules:

# Get protection rules for base branch (usually main/master)
gh api repos/{owner}/{repo}/branches/{base_branch}/protection \
  --jq '.required_status_checks.contexts[]'

Cross-reference with current checks:

• Match check names from Step 2 with required contexts from protection rules
• Build list of required checks that failed
• Exclude optional checks from retry logic

Fallback if no protection rules:

# If branch protection API fails, use heuristics:
# - Checks with "required" in description
# - Common required check patterns: "build", "test", "lint", "security"
# - Exclude obvious optional ones: "codecov", "sonar", "deploy-preview"

Step 4: Wait Logic (if enabled)

If --wait_for_completion true:

while [[ $in_progress_count -gt 0 ]]; do
    echo "⏳ Waiting for $in_progress_count checks to complete..."
    sleep {poll_interval}
    
    # Re-check status
    gh pr checks {pr_number} --json name,status,conclusion
    # Update in_progress_count
done

echo "✅ All checks completed. Proceeding with retry logic..."

Progress indicators:

• Show which checks are still running
• Display estimated time remaining (if available from API)
• Allow user to interrupt waiting with Ctrl+C

Step 5: Smart Retry Logic

For each failed required check:

# Get the specific workflow run ID
run_id=$(gh run list --pr {pr_number} --workflow "{workflow_name}" \
  --json databaseId --jq '.[0].databaseId')

# Retry only the failed jobs (not successful ones)  
gh run rerun $run_id --failed

echo "🔄 Retried {workflow_name} (run #$run_id)"

Retry strategy:

• Only retry checks that both:
  1. Are required for merge
  2. Have conclusion: "failure" (not cancelled/skipped)
• Track retry attempts to respect --max_retries
• Wait between retries to avoid rate limiting

Step 6: Post-Retry Monitoring

After triggering retries:

echo "🚀 Triggered retries for $retry_count required checks:"
for check in "${retried_checks[@]}"; do
    echo "  - $check"
done

echo ""
echo "📊 Current status summary:"
echo "  ✅ Passed: $passed_count"  
echo "  🔄 Retrying: $retry_count"
echo "  ⏸️ In Progress: $in_progress_count"
echo "  ❌ Still Failed: $still_failed_count"

This skill transforms flaky CI from a time-sink into an automated background task! 🚀