run-e2e-test

Run E2E Test

When To Use

Use this skill when the user wants to manually replay a cloud-provider-azure e2e test case against a live Kubernetes cluster. The skill parses the Go test source, translates each step to CLI commands (kubectl / az), and guides interactive execution.

Prerequisites

• az login completed (active Azure session)
• Valid KUBECONFIG pointing at the target cluster
• kubectl available on $PATH
• python3 available on $PATH

Inputs

Workflow

Replace <SKILL_DIR> with the path to this skill directory.

Step 1 — Confirm Cluster Context

Unless --skip-context-check is passed:

python3 <SKILL_DIR>/scripts/check_context.py

This displays the current kube context, cluster endpoint, and Azure subscription. Ask the user to confirm this is the correct cluster before proceeding. If the user says no, stop and ask them to switch context.

Step 2 — Analyze the Test

python3 <SKILL_DIR>/scripts/analyze_test.py <test-file-path> [--test-name "<test name>"] [--json]

This parses the Go test file and outputs a structured plan with:

• variables: Go const/var declarations (deployment names, service names, labels, ports) extracted from the file — use these to resolve ${VAR} placeholders in emitted commands
• setup: BeforeEach steps (namespace, deployment, Azure client init)
• test_cases: list of It(...) blocks, each with ordered steps
• teardown: AfterEach cleanup steps

Each step includes:

• type: k8s_create, k8s_update, k8s_delete, k8s_wait, az_create, az_read, az_delete, az_wait, connectivity_check, assertion, phase
• description: human-readable explanation
• command: suggested CLI command(s)
• wait: polling parameters if applicable
• verify: what to check in the output

Use --json to get machine-readable JSON output instead of the human-readable format:

python3 <SKILL_DIR>/scripts/analyze_test.py tests/e2e/network/ensureloadbalancer.go --json

If --test-name is provided, only that test case is included. Otherwise all test cases are listed.

Variable Resolution

The variables field in the JSON output lists Go constants and variable declarations extracted from the test file header. Use them to populate ${VAR} placeholders in the emitted commands. For example, if the output contains "testBaseName": "service-lb", substitute ${DEPLOYMENT_NAME} with the value derived from that constant. Some variables may reference other Go symbols — in that case, read the test source to resolve the chain.

Step 3 — Review the Plan

Present the extracted plan to the user. For each test case show:

1. Setup steps (namespace, deployment, pre-created Azure resources)
2. Test body steps in order
3. Verification checks
4. Cleanup steps

Ask the user to confirm before executing. They may choose to skip certain steps or modify parameters.

Step 4 — Execute Setup

Run the setup steps:

1. Create namespace: kubectl create namespace <generated-name>
2. Create deployment: generate YAML from the test's deployment manifest and kubectl apply -f -
3. Wait for pods: kubectl wait --for=condition=ready pod -l <labels> -n <ns> --timeout=5m
4. Pre-create Azure resources (if needed): PIPs, subnets, IP prefixes via az network ... commands

Track all created resources for cleanup.

If any setup step fails:

1. Run the diagnostic command from the Error Handling section
2. Report the error with diagnostic output
3. Ask the user whether to clean up now (jump to Step 6) or keep the environment for manual investigation
4. Do not proceed to Step 5

Step 5 — Execute Test Steps

For each step in the test body:

1. Create/update K8s resources: generate YAML, kubectl apply -f -
2. Wait for conditions: poll with the timeouts from the test
   • Service IP: kubectl get svc -n <ns> <name> -o jsonpath='{.status.loadBalancer.ingress[*].ip}'
   • Poll every 10s, timeout per test constants (typically 5m for Standard LB)
3. Verify Azure state: run az commands and check output matches assertions
4. Connectivity checks: kubectl exec <exec-pod> -- nc -vz -w 4 <ip> <port>
5. Report results: for each assertion, show PASS/FAIL with actual vs expected

Before executing, read references/INDEX.md (the patterns index). For each step in the plan, match the step's description keywords and step_type against the index to identify which sections of references/patterns.md are needed. Read only those sections using the line ranges from the index. Do not read the entire patterns.md file.

On step failure, follow the Failure Flow waterfall in the Error Handling section. Key behaviors:

• Transient errors (429, 409, 5xx, timeout): retry per the retry policy
• Poll timeout (condition never met): record as assertion FAIL
• Assertion/verification mismatch: record FAIL with actual vs expected, run diagnostics, then ask the user whether to (a) continue, (b) pause for investigation, (c) abort to cleanup, or (d) re-check
• Non-recoverable errors: stop and jump to Step 6

Step 6 — Cleanup

Run cleanup in reverse order:

1. Delete K8s services
2. Delete K8s deployments
3. Delete pre-created Azure resources (PIPs, subnets, prefixes)
4. Delete namespace

Always offer cleanup even if a step failed. Track which resources were actually created to avoid deleting things that don't exist.

If a cleanup step fails:

1. Retry the deletion once after 30s
2. If it still fails, log the resource as orphaned and continue to the next cleanup step
3. Include all orphaned resources in the Step 7 report

Step 7 — Report

Summarize the execution:

Always include:

• Total steps executed: N
• Passed assertions: N
• Failed assertions: N

Include if any failures occurred:

• Skipped steps: N (with reason for each)
• For each failure:
  • Step description
  • Expected vs actual
  • Diagnostic output (if collected)
• Infrastructure errors encountered
• Re-checked assertions: N (list step name, attempts before pass or final fail)

Always include:

• Resources created: list
• Resources cleaned up: list
• Orphaned resources: list (if any cleanup failed)

Bundled Resources

• scripts/check_context.py — display and validate current kube/Azure context
• scripts/analyze_test.py — parse Go e2e test file into structured replay plan
• references/INDEX.md — generated keyword and step-type index into patterns.md; read this first to find which sections to load
• references/patterns.md — comprehensive mapping of Go e2e test patterns to kubectl and az CLI commands, including manifest templates and polling patterns
• scripts/gen_patterns_index.py — regenerate references/INDEX.md from keyword annotations in patterns.md

Error Handling

Error Categories

Retry Policy

Retry transient errors before classifying as a failure:

Diagnostics

When a step fails, run the relevant diagnostic before reporting:

When reporting diagnostic output, redact values of environment variables whose names contain SECRET, PASSWORD, TOKEN, KEY, or CONNECTION_STRING. Truncate log output to 20 lines.

Resource Tracking

Track all resources created during execution in your working memory:

Created resources:
  k8s: namespace/e2e-test-abc123
  k8s: deployment/deployment-lb-test (ns: e2e-test-abc123)
  k8s: service/svc-test (ns: e2e-test-abc123)
  k8s: pod/exec-agnhost (ns: e2e-test-abc123)
  az:  public-ip/pip-test (rg: mc_mygroup)

Include both directly created resources (from your commands) and indirectly created ones (discovered via az queries after Service creation — e.g., LB rules, NSG rules, PIP allocated by the cloud controller manager).

On early exit, use this list for targeted cleanup in Step 6.

If the agent session is interrupted during a pause, created resources become orphans. Re-run cleanup manually using the namespace name and resource names from the test's variable declarations.

Failure Flow

When a command fails, evaluate in this order:

1. Is it retryable? Check the retry table. If yes, retry.

2. Is it an infrastructure error? (auth, cluster unreachable, command not found) → Stop. Report error. Jump to Step 6.

3. Is it a setup step failure? (Step 4) → Stop. Run diagnostics. Report. Jump to Step 6.

4. Is it a poll timeout? (wait for condition that never became true) → Treat as assertion failure.

5. Is it an assertion/verification failure? → Record FAIL with actual vs expected. Run the matching diagnostic from the Diagnostics table. Ask the user:

   • (a) Continue: skip dependent steps, proceed to next independent step
   • (b) Pause: keep environment alive for manual investigation; suggest the diagnostic command, the failed step's verification command, and 1–2 exploratory commands for the resource under test; wait for user to say "continue", "re-check", or "abort"
   • (c) Abort: jump to Step 6 (Cleanup)
   • (d) Re-check: re-run this step's verification (including waits/polls); if it passes, clear the FAIL and proceed normally (do not skip dependent steps); if it fails again, re-present the same 4 options; after 3 re-check failures on the same step, warn that repeated re-checks are failing and suggest aborting

   Batch mode: if more than 2 assertion failures have occurred in the current test case and the user selects Continue, offer: "Multiple assertions are failing. (a) Keep pause-on-failure, (b) Switch to auto-continue (record all remaining FAILs without pausing), (c) Abort to cleanup."

   Pause warnings:

   • ⚠️ The cluster and Azure resources remain live during the pause. Cloud controllers and Azure reconciliation may modify resources (LB rules, NSG rules, endpoints).
   • Azure resources created by this test remain provisioned and billable during the pause.
   • If the pause exceeds 10 minutes, remind the user that the environment may have changed and suggest re-running diagnostics.

To identify dependent steps, check if a subsequent step uses a variable produced by the failed step. For example, if "Wait for service external IP" fails (no IP assigned), skip "Verify connectivity to service IP" since it depends on having a valid IP.

6. Is it a cleanup failure? → Warn, retry once, log orphan, continue.

Limitations

• The analysis script uses pattern matching on Go source, not a full Go parser. It handles the standard Ginkgo patterns used in this repo (Describe, When, Context, It, BeforeEach, AfterEach, By). Unusual or deeply nested test structures may need manual interpretation by the agent.
• By(...) annotations are emitted as phase type steps rather than merged into other steps. The agent should use them as context markers.
• Annotation resolution relies on a static map. If a test uses an annotation constant not in the map, the raw consts.XxxAnnotation name is emitted. Check pkg/consts/consts.go for the string value.
• The variables field captures simple const/var string assignments. It does not evaluate Go expressions or resolve cross-references.
• Commands use sed for POSIX-portable field extraction (no GNU grep -oP).

Notes

• The agent should read references/patterns.md when it encounters a pattern not covered by the analysis script output.
• Azure resource group is discovered from node providerID: kubectl get nodes -o jsonpath='{.items[0].spec.providerID}' → parse RG name.
• Some tests require specific environment conditions (Standard LB, AKS cluster, VMSS node pools). The analysis script flags these as skip conditions that the agent should check before executing.