k8s-debug-pending-pod

Interaction

Collect the following from the user in a single exchange before starting diagnosis:

If the user provides a cluster name instead of a context, fuzzy-match it against available contexts in the kubeconfig and confirm the match.

The skill automatically discovers and diagnoses all unhealthy pods (Pending, CrashLoopBackOff, ImagePullBackOff, Error) in the namespace. No pod name is needed.

Command Convention

All kubectl commands use shorthand below. Actual execution must always include explicit flags:

kubectl --kubeconfig=<kubeconfig> --context=<context> ...

Never rely on default kubeconfig or current-context.

Diagnostic Workflow

Step 1: Global Pod Overview

Get a full picture of all pod states in one command:

kubectl get pods -n <namespace> -o wide

Categorize pods into:

• ✅ Running/Completed — healthy, skip
• 🟡 Pending — go to Step 3
• 🔴 CrashLoopBackOff/Error — go to Step 4
• 🔴 ImagePullBackOff/ErrImagePull — go to Step 5
• ⚠️ Other abnormal states — note for report

All unhealthy pods are diagnosed. No manual pod selection needed.

Step 2: Event Triage

Quick scan of recent warning events for immediate signal:

kubectl get events -n <namespace> --field-selector type=Warning --sort-by=.lastTimestamp | tail -50

Use events to guide which deep-dive steps are most relevant. Common event→action mapping:

Short-circuit rule: If events clearly point to a single root cause (e.g., all events are FailedScheduling with the same message), mark it as the confirmed primary cause but still scan remaining steps at reduced depth for additional findings.

Step 3: Pending Pod Root Cause Analysis

For each Pending pod:

kubectl describe pod <pod-name> -n <namespace>

Parse the Events and Conditions sections. Common causes:

Step 4: CrashLoopBackOff / Error Diagnosis

Only execute if Step 1 found pods in CrashLoopBackOff or Error state.

A. Termination reason:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'

B. Logs (current + previous):

kubectl logs <pod-name> -n <namespace> --tail=100
kubectl logs <pod-name> -n <namespace> --previous --tail=100

C. Resource config:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}' | jq .

D. Restart count:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[*].restartCount}'

High restart count (>5) combined with short uptime indicates persistent failure — check probes, dependencies, and startup ordering.

Step 5: ImagePullBackOff Diagnosis

Only execute if Step 1 found pods in ImagePullBackOff or ErrImagePull state.

A. Image reference:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].image}'

B. Pull error from events:

kubectl describe pod <pod-name> -n <namespace> | grep -A10 "Events:"

C. imagePullSecrets check:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

If empty and the image is from a private registry, this is likely the root cause.

Step 6: Resource Quota Check (Conditional)

Only execute if Step 3 indicates quota exhaustion or resource issues.

kubectl describe resourcequota -n <namespace>

Analyze:

• Which resources are at or near limits (Used/Hard ratio > 80%)
• Whether the pending pod's requests would exceed remaining capacity
• Recommend: request quota increase or reduce pod resource requests

Optional — Resource allocation efficiency:

kubectl get pods -n <namespace> -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory'
kubectl top pods -n <namespace>

Flag:

• Requests >> actual usage → waste, can reduce to free quota
• No limits set → unbounded usage risk
• High requests with low limits → OOM risk

Step 7: Report

Generate a structured report:

Kubernetes Pod Troubleshooting Report
=====================================
Namespace: <namespace>
Cluster:   <context> (via <kubeconfig>)
Scope:     all unhealthy pods in namespace

Primary Issue
- <confirmed root cause with evidence>

Additional Findings
- <secondary issues discovered during scan>

Pod Status Summary
- Total: N | Running: N | Pending: N | CrashLoop: N | ImagePull: N

Detailed Findings
- [Per-pod breakdown with cause and evidence]

Recommended Actions (priority order)
1. <highest impact fix>
2. <next>
3. <next>

Skipped Checks
- <any steps skipped due to permissions, unreachable API, or missing metrics-server>