failure-diagnosis

Failure Diagnosis

When diagnosing a deployment issue, work through the relevant section based on the symptom. Use the pattern tables to match log output before hypothesizing.

Build Failure Patterns

After calling get_deployment_logs, scan the output for these patterns.

Node.js

Python

Go

Rust

Java

General Dockerfile

Container Runtime Failures

After a deployment succeeds (image built) but the container crashes or misbehaves.

Exit Codes

Exit codes 128+N mean the process was killed by signal N.

Container Inspect Signals

Use container_inspect to check these fields:

Common Runtime Patterns

Scan get_container_logs or get_application_logs for these:

Crash Loop Detection

A container is in a crash loop when:

1. restart_count > 3 in the last 10 minutes
2. Container logs show the same error repeating
3. Container uptime resets to 0 repeatedly

To diagnose a crash loop:

1. get_container_logs for the last 100 lines
2. container_inspect for oom_killed, exit code, restart count
3. If OOM: container_stats to see memory usage trend
4. If exit 1: search logs for the first error after startup
5. If exit 137 but not OOM: check host memory via machine-agent delegation

Networking Failures

When the app runs but is not reachable externally.

Port Mismatch Diagnosis

Four values must agree — a mismatch at any level causes unreachable apps:

If any disagree, the app is unreachable. The app listen port is the source of truth — all others must match it.

Reachability Matrix

Use this decision tree when the app is reported as unreachable:

Proxy and TLS Issues

Container-to-Service Connectivity

When the app can't reach its dependencies (database, cache, external API):

1. container_exec ["nslookup", "<hostname>"] — DNS resolution
2. container_exec ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "<url>"] — HTTP reachability
3. network_diagnostics with type port — TCP connectivity
4. Check if both containers are on the same Docker network via container_inspect → networks

Diagnostic Decision Tree

Start here. Match the symptom, follow the path.

Symptom: build_failed

1. get_application_deployments to find the failed deployment
2. get_deployment_logs for the full build output
3. Scan logs against Build Failure Patterns tables above
4. If match found: apply the documented fix
5. If no match: search for the first error or Error line — that's usually the root cause (earlier lines are often cascading failures)

Symptom: container exited / crash loop

1. get_application to confirm deployment status
2. list_containers to find the container (it may have been removed on crash)
3. container_inspect for exit code, oom_killed, restart_count, health_status
4. Map exit code using Exit Codes table
5. get_container_logs (or get_application_logs if container is gone) for the last error
6. Match log output against Common Runtime Patterns table
7. If OOM: container_stats to see current memory usage vs limit

Symptom: app unreachable

1. http_probe the public URL — if it responds, problem is intermittent or resolved
2. get_application to confirm the app exists and has a deployment
3. list_containers to check container is running
4. container_exec ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "localhost:PORT"] — internal reachability
5. If internal works but external doesn't: proxy_config for upstream mismatch, then Port Mismatch Diagnosis
6. If internal fails too: check container logs for startup errors
7. If container is running but not listening: container_exec ["ss", "-tlnp"] to see what ports are bound

Symptom: intermittent errors / slow responses

1. container_stats for CPU and memory pressure
2. get_container_logs with recent timeframe — look for error spikes
3. If memory > 80% of limit: approaching OOM, recommend increasing memory or fixing leak
4. If CPU > 90%: app is compute-bound, check for infinite loops or missing caching
5. http_probe multiple times to confirm intermittent pattern
6. Check container_inspect → restart_count for silent crash-restarts

Related Skills

• domain-tls-routing — For domain resolution, TLS certificate, and reverse proxy routing issues specifically
• pre-deploy-checklist — Run before deployment to catch issues that would cause the failures diagnosed here