OpenParable
Skip to content
← Back to skills

Debug Session

Testing & Quality
debug-session

Debug Session is an Agent Skill for testing & quality published in the ai-dynamo/dynamo repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Sets up a structured debugging session for a Dynamo bug — pull the report from a Linear ticket, GitHub issue, or pasted text, capture the environment, create a persistent worklog markdown file, and walk through rebuild, reproduction, and investigation steps. Use when starting to investigate a reported Dynamo bug or regression and the investigation should be tracked in a worklog.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/ai-dynamo/dynamo/blob/HEAD/.agents/skills/debug-session/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/debug-session, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

Start Debug Session

Create a structured debugging session for an issue in the Dynamo ecosystem.

Step 1: Get the Bug Report

Ask the user how they want to provide the bug:

Option A: Linear ticket

  • User provides ticket ID (e.g., "DYN-123")
  • Fetch via Linear MCP tools
  • Extract: title, description, reproduction steps

Option B: GitHub issue

  • User provides issue URL
  • Fetch via gh issue view <url>
  • Extract: title, description, reproduction steps

Option C: Paste

  • Ask user to paste the bug report directly
  • Parse out the key details

Step 2: Discover Environment

Gather environment information:

!nvidia-smi --query-gpu=name,count --format=csv,noheader 2>/dev/null || echo "No GPU detected"

!uname -a

!which python && python --version

This tells you:

  • GPU type and count (L40s, H100s, etc.)
  • OS/platform
  • Python environment

Note: The user's ~/.claude/CLAUDE.md may have more details about their dev environment (paths, aliases, preferences). Check there for additional context.

Step 3: Create Worklog

Create a worklog file to track the investigation:

  • Filename: <issue-slug>.md in current directory
  • Template:
# Debug: [Issue Title]

**Date**: [today's date]
**Source**: [Linear ticket / GitHub issue / user report]
**Status**: investigating
**Environment**: [GPU type/count from nvidia-smi]

## Problem
[Description of the issue]

## Reproduction Steps
1. [Step to reproduce]
2. ...

## Expected vs Actual
- **Expected**:
- **Actual**:

## Investigation Log

### [timestamp]
[Notes on what you tried/found]

## Root Cause
[Fill in when found]

## Fix
[Fill in when implemented]

Step 4: Set Up Testing

Build Commands

Rebuild Dynamo after making changes:

cd lib/bindings/python && maturin develop --uv && cd ../../.. && uv pip install -e .

If a framework change is required (sglang, vllm, trtllm), check the user's ~/.claude/CLAUDE.md for rebuild instructions specific to that framework.

Running Examples

Examples are located at: /home/ubuntu/dynamo/examples/backends/

Available backends:

  • sglang/launch/ - SGLang backend examples
  • vllm/launch/ - vLLM backend examples
  • trtllm/launch/ - TensorRT-LLM backend examples

Based on the bug report, determine which backend is relevant:

  • If unclear, ask the user which backend/example to run
  • Run the example in the background
  • Wait for model to be ready

Verifying the Model is Up

curl localhost:8000/v1/models

Testing with a Request

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<model-name-from-above>",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'

Step 5: Begin Investigation

Dynamo Infrastructure Debugging

KV cache and routing issues:

  • Check KV event logs in lib/llm/src/block_manager/kv_consolidator/tracker.rs
  • Look at block manager state and consolidation behavior
  • Inspect routing decisions in the KV-aware router

ZMQ / networking issues:

  • Check ZMQ socket configuration and endpoint bindings
  • Look for connection timeouts or message drops
  • Verify nats/etcd connectivity for service discovery

Multi-node / disaggregated issues:

  • Check prefill/decode worker assignment
  • Verify DGD (disaggregated) status reporting
  • Inspect inter-node communication via nvidia-smi on each node
  • Check NCCL and GPU direct RDMA status

Process inspection:

  • ps aux | grep dynamo - check running processes
  • nvidia-smi - GPU utilization and memory
  • ss -tlnp | grep 8000 - check port bindings
  • journalctl -u dynamo - systemd logs if applicable

General Debugging Workflow

  1. Reproduce first - verify you can trigger the bug before attempting fixes
  2. Document as you go - update the worklog with findings
  3. Minimal changes - fix the bug, do not refactor surrounding code
  4. Verify the fix - confirm the reproduction case now passes

Performance-critical code - avoid unnecessary abstractions or comments.