testing:kubectl-debugging

Kubectl Debugging Patterns

Common kubectl commands for debugging Kagenti components.

Context-Safe Execution (MANDATORY)

All kubectl/oc commands MUST redirect output to files. Commands below are shown in bare form for readability. When executing, always redirect:

export LOG_DIR=/tmp/kagenti/k8s/${CLUSTER:-local}
mkdir -p $LOG_DIR

# Pattern: kubectl <command> > $LOG_DIR/<name>.log 2>&1 && echo "OK" || echo "FAIL"
# Analyze in subagent: Task(subagent_type='Explore') with Grep

Table of Contents

• Setting Up Environment
• Helm Debugging
• ConfigMap and Secret Inspection
• Pod Debugging
• Service Debugging
• Keycloak Client Verification
• Job Debugging
• Istio Debugging
• Events
• Quick Reference

Setting Up Environment

Using Correct Kubeconfig

# HyperShift cluster
export KUBECONFIG=~/clusters/hcp/kagenti-hypershift-custom-mlflow/auth/kubeconfig

# Kind cluster
export KUBECONFIG=~/.kube/config
kubectl config use-context kind-kagenti

Verify Connection

kubectl cluster-info
kubectl get nodes

Helm Debugging

Check Rendered Values

helm get values kagenti-deps -n kagenti-system

Check All Values (Including Defaults)

helm get values kagenti-deps -n kagenti-system -a

Template Without Installing

helm template kagenti-deps charts/kagenti-deps -n kagenti-system \
  -f /tmp/values.yaml > /tmp/rendered.yaml

Check Release Status

helm list -n kagenti-system
helm history kagenti-deps -n kagenti-system

ConfigMap and Secret Inspection

Extract ConfigMap Content

kubectl get configmap otel-collector-config -n kagenti-system -o yaml

Extract Specific Key

kubectl get configmap otel-collector-config -n kagenti-system \
  -o jsonpath='{.data.otel-collector-config\.yaml}'

Decode Secret

kubectl get secret mlflow-oauth-secret -n kagenti-system \
  -o jsonpath='{.data.MLFLOW_CLIENT_ID}' | base64 -d

List All Secret Keys

kubectl get secret mlflow-oauth-secret -n kagenti-system \
  -o jsonpath='{.data}' | jq 'keys'

Pod Debugging

Check Pod Environment Variables

kubectl get pod otel-collector-xxx -n kagenti-system \
  -o jsonpath='{.spec.containers[0].env}' | jq

Check Pod Status

kubectl describe pod otel-collector-xxx -n kagenti-system

Get Pod Logs

kubectl logs -n kagenti-system otel-collector-xxx
kubectl logs -n kagenti-system otel-collector-xxx --previous  # After crash
kubectl logs -n kagenti-system otel-collector-xxx -f          # Follow

Exec Into Pod

kubectl exec -it otel-collector-xxx -n kagenti-system -- /bin/sh

Check Mounted Files

kubectl exec -it otel-collector-xxx -n kagenti-system -- \
  ls -la /etc/pki/ca-trust/extracted/pem/

Service Debugging

Check Service Endpoints

kubectl get endpoints mlflow -n kagenti-system

Check Service Labels

kubectl get svc mlflow -n kagenti-system --show-labels

Port Forward

kubectl port-forward svc/mlflow 5000:5000 -n kagenti-system

Keycloak Client Verification

Get Token

# Set variables
KEYCLOAK_URL="http://keycloak-service.keycloak.svc.cluster.local:8080"
CLIENT_ID="mlflow-client"
CLIENT_SECRET=$(kubectl get secret mlflow-oauth-secret -n kagenti-system \
  -o jsonpath='{.data.MLFLOW_CLIENT_SECRET}' | base64 -d)

# Get token
curl -X POST "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=$CLIENT_ID" \
  -d "client_secret=$CLIENT_SECRET"

Test From Inside Cluster

kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl -X POST "http://keycloak-service.keycloak.svc.cluster.local:8080/realms/master/protocol/openid-connect/token" \
  -d "grant_type=client_credentials" \
  -d "client_id=mlflow-client" \
  -d "client_secret=<secret>"

Job Debugging

Check Job Status

kubectl get jobs -n keycloak
kubectl describe job mlflow-oauth-secret -n keycloak

Get Job Pod Logs

kubectl logs -n keycloak -l job-name=mlflow-oauth-secret

Rerun Failed Job

kubectl delete job mlflow-oauth-secret -n keycloak
# Job will be recreated by Helm if still in chart

Istio Debugging

Check Waypoint Status

kubectl get gateway -n kagenti-system
kubectl describe gateway mlflow-waypoint -n kagenti-system

Check AuthorizationPolicy

kubectl get authorizationpolicy -n kagenti-system
kubectl describe authorizationpolicy mlflow-traces-from-otel -n kagenti-system

Check Pod Identity

istioctl proxy-config secret otel-collector-xxx -n kagenti-system

Check ztunnel Logs

kubectl logs -n istio-system -l app=ztunnel --tail=100

Events

Namespace Events

kubectl get events -n kagenti-system --sort-by='.lastTimestamp'

Pod Events

kubectl get events -n kagenti-system --field-selector involvedObject.name=otel-collector-xxx

Resource Usage

Pod Resources

kubectl top pods -n kagenti-system

Describe Resource Limits

kubectl get pod otel-collector-xxx -n kagenti-system \
  -o jsonpath='{.spec.containers[0].resources}'

Quick Reference

Related Skills

• tdd:hypershift
• k8s:live-debugging
• istio:ambient-waypoint