Debugging E2e Failures
Testing & Qualitydebugging-e2e-failuresDebugging E2e Failures is an Agent Skill for testing & quality published in the tektoncd/catalog repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Diagnose and fix failed e2e tests in tektoncd/catalog. Use when a CI run fails, a local test hangs, or a TaskRun/PipelineRun reaches an error state. Covers triage from GitHub Actions output to root cause, common failure patterns, and local reproduction.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/tektoncd/catalog/blob/HEAD/.claude/skills/debugging-e2e-failures/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/debugging-e2e-failures, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Debugging e2e Test Failures
Triage Workflow
Step 1: Identify which matrix leg failed
CI runs e2e tests against multiple Tekton Pipeline versions. Check the GitHub Actions summary to find which version(s) failed:
# From a PR, list failed checks
gh pr checks <PR-number> | grep -i fail
# Or view a specific workflow run
gh run view <run-id> --log-failed
The matrix legs are:
latest/lts-latest→ Pipeline v1.9.0lts-latest-minus-one→ Pipeline v1.6.0lts-latest-minus-two→ Pipeline v1.3.0lts-latest-minus-three→ Pipeline v1.0.0
If the failure is only on older Pipeline versions, the task likely uses a
feature not available in that version. Check tekton.dev/pipelines.minVersion.
Step 2: Find the failing task
The CI log will show lines like:
STARTING: <task-name>/<version>
FAILED: <task-name> task has failed to comeback properly
Search the log for FAILED: or ERROR to locate the failing task.
Step 3: Read the diagnostic dump
When a test fails, show_failure() in test/e2e-common.sh dumps:
- All TaskRuns (YAML) in the test namespace
- All PipelineRuns (YAML)
- All pod container logs
- API resources
Look for:
.status.conditions[].reasonon the TaskRun/PipelineRun- Container logs from
step-*containers - Events showing scheduling or image pull issues
Step 4: Reproduce locally
# Minimal Kind cluster with Tekton
kind create cluster --name debug-catalog
kubectl apply -f https://github.com/tektoncd/pipeline/releases/download/v1.0.0/release.yaml
kubectl wait --for=condition=ready pod -l app=tekton-pipelines-controller \
-n tekton-pipelines --timeout=60s
# Run the failing test
./test/run-test.sh task <task-name> <version>
To match a specific CI matrix leg, use the matching Pipeline version:
kubectl apply -f https://github.com/tektoncd/pipeline/releases/download/<version>/release.yaml
Common Failure Patterns
TaskRun stuck in Pending
kubectl describe taskrun <name> -n <namespace>
kubectl get events -n <namespace> --sort-by='.lastTimestamp'
Causes:
- Missing workspace PVC: the test needs a
pre-apply-taskrun-hook.shto create a PVC or useemptyDir - Missing Secret/ConfigMap: check if the test references resources that the hook should create
- Resource quota: unlikely in Kind, but check node resources
ImagePullBackOff
kubectl describe pod <pod> -n <namespace> | grep -A5 "Events:"
Causes:
- Digest changed: the image was rebuilt upstream and the pinned digest is stale. Update the digest in the task YAML
- Registry rate limit: docker.io pulls are rate-limited. The Kind setup
uses a pull-through cache at
registry.local:5000, but it can fail if the cache is cold - Typo in image reference: check the
image:field carefully
Script failure (exit code non-zero)
kubectl logs <pod> -n <namespace> -c step-<step-name>
Causes:
- Missing tool in image: the script calls a binary not in the container
- Parameter interpolation bug: if
$(params.*)is used in ascriptblock, values with special characters will break the shell - Network issue: the script tries to reach an external service. Use fixtures instead (see below)
Fixture/mock API failures
If the task uses tests/fixtures/ for API mocking:
# Check the go-rest-api-test sidecar logs
kubectl logs <pod> -n <namespace> -c go-rest-api
Causes:
- Path mismatch: the fixture rule path doesn't match what the task actually calls. The sidecar logs show incoming requests
- Method mismatch: POST vs GET vs PATCH
- Missing fixture rule: a new API call was added but no fixture covers it
Timeout (10 minute limit)
The test runner polls for 60 iterations of 10 seconds (10 minutes total). If a TaskRun takes longer, it times out.
# Check what's still running
kubectl get taskrun -n <namespace> -o wide
kubectl get pods -n <namespace>
Causes:
- Waiting for input: the script has an interactive prompt or
read - Deadlock on workspace: two steps writing to the same file
- Slow image pull: large images on a cold Kind cluster cache
- Infinite loop in script: check the step logs for repeated output
test_yaml_can_install failure
This happens before any TaskRun is created — the Task YAML itself is invalid:
ERROR: task <name> can't be installed
Causes:
- Invalid YAML syntax: run
yamllint -c .yamllint <file> - Unknown API field: using a field from a newer Tekton version than the CI leg is testing against
- Duplicate resource name: another task with the same
metadata.name
Iterating Without Full Teardown
# Delete the test namespace and retry
kubectl delete ns <task>-<version> --ignore-not-found
./test/run-test.sh task <task-name> <version>
# Or just re-apply the task and re-run the TaskRun
kubectl delete taskrun --all -n <namespace>
kubectl delete task <task-name> -n <namespace>
kubectl apply -f task/<task-name>/<version>/<task-name>.yaml -n <namespace>
kubectl apply -f task/<task-name>/<version>/tests/run.yaml -n <namespace>
Cleanup
kind delete cluster --name debug-catalog
docker network prune -f