Chatto Debugging
Testing & Qualitychatto-debuggingChatto Debugging is an Agent Skill for testing & quality published in the chattocorp/chatto repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
Production debugging tools for NATS streams, KV buckets, and protobuf events. Covers nats CLI commands, stream inspection, protobuf decoding, and message iteration.
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/chattocorp/chatto/blob/HEAD/.agents/skills/chatto-debugging/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/chatto-debugging, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
Debugging
Local Startup And Heap Profiling
When benchmarking projection startup or runtime memory locally, prefer the built-in diagnostics knobs before adding ad hoc instrumentation:
CHATTO_METRICS_ENABLED=true \
CHATTO_METRICS_PPROF=true \
CHATTO_METRICS_BIND_ADDRESS=127.0.0.1 \
CHATTO_METRICS_PORT=9090 \
CHATTO_DIAGNOSTICS_STARTUP_CPU_PROFILE=.context/bench/startup.pprof \
mise x -- go run -tags bootstrap . run
CHATTO_DIAGNOSTICS_STARTUP_CPU_PROFILEwrites a Go CPU profile from early process startup throughChattoCore.WaitForBoot, which captures embedded NATS startup and projection replay.CHATTO_METRICS_PPROF=trueexposes/debug/pprof/on the private metrics listener for heap, goroutine, trace, and under-load CPU profiles after boot.- Keep benchmark data and profiles under
.context/bench/so they remain gitignored and easy to compare across runs. - Use
go tool pprof -top ./bin/chatto .context/bench/startup.pproforgo tool pprof -http=:0 ...to inspect profiles. For heap profiles, fetchhttp://127.0.0.1:9090/debug/pprof/heapfrom the metrics port. - When a large imported instance is available in
cli/data/, build comparable binaries fororigin/mainand the branch withgo build -tags bootstrap, copycli/data/into a fresh.context/bench/.../datadirectory for each run, and alternate runs between revisions. On fast Apple Silicon, absolute startup times can understate Linux cluster costs; use repeated runs, pprof deltas, projection startup metrics, and retained heap profiles to identify the next target. - For projection replay work, inspect allocation and CPU profiles before changing code. Past wins came from removing duplicate consumer fanout, duplicate protobuf decode, repeated subject matching, and repeated JetStream metadata parsing while preserving stable stream sequence semantics.
NATS CLI for Production Debugging
This is the only supported path for raw NATS inspection. The ConnectRPC admin diagnostics surface intentionally exposes only aggregate operational metrics — it does not expose stream subjects, KV keys, or per-stream/per-bucket breakdowns. Those leak structural information (room IDs, user IDs, bucket names) without a use case the nats CLI doesn't already cover. If you need to inspect a specific stream or bucket, shell into the host with operator credentials and use the commands below.
The nats CLI can directly inspect streams and KV buckets to debug production issues. Useful commands:
# List all streams and their message counts
nats stream ls
# List subjects in a stream with message counts per subject
nats stream subjects EVT
# Get stream info (config, state, consumer count)
nats stream info EVT --json
# Get a specific message by sequence number
nats stream get EVT 51 --json
# Get the last message for a specific subject
nats stream get EVT -S "evt.room.{roomId}.message_posted" --json
# List all KV buckets
nats kv ls
# List keys in a KV bucket
nats kv ls RUNTIME_STATE
# Get a KV entry
nats kv get RUNTIME_STATE "session.{sessionId}"
Decoding Protobuf Events
Stream messages are base64-encoded protobuf. To decode:
# Get message and decode raw protobuf (shows field numbers and values)
nats stream get STREAM_NAME SEQ --json | jq -r '.data' | base64 -d | protoc --decode_raw
# Common field numbers in SpaceRoomEvent:
# - 300: RoomCreatedEvent
# - 310: UserJoinedRoomEvent
# - 311: UserLeftRoomEvent
# - Field 1: event ID
# - Field 2: timestamp
# - Field 3: actor ID
Iterating Through Stream Messages
To find all messages matching a pattern:
for i in $(seq 1 100); do
result=$(nats stream get STREAM_NAME $i --json 2>/dev/null)
if echo "$result" | jq -e '.subject | contains("pattern")' > /dev/null 2>&1; then
echo "=== Seq $i ==="
echo "$result" | jq -r '.subject, .time'
fi
done