crash-analysis

Crash Analysis

Systematically analyze a dd-trace-php crash report to identify the root cause.

Schema note: The event file you receive is the backend-enriched form, not the one libdatadog emits.

Input

The user provides a crash event JSON file (path via $ARGUMENTS, or pasted inline). If no path is given, ask for it. Save pasted JSON to a temporary file.

Phase 1 — Triage (extract all key facts before any analysis)

Extract and print a summary table with these fields. Use jq against the event file. Do all extractions in parallel where possible.

Note: parse_ucontext.py only supports amd64. Check .ucontext.arch first; on aarch64, skip the script and read register values directly from .ucontext.raw.

crash_diagnosis (schema 1.8+)

crash_diagnosis is computed server-side by the Datadog errors-worker (DataDog/dd-source, domains/evp-workers/apps/errors-worker/src/crashtracking/), not by libdatadog. It consumes sig_info, ucontext.registers, and /proc/self/maps from the event; if any is absent, the field is omitted. Use it to confirm (not skip!) manual triage steps:

DiagnosisCategory enum (complete)

SSI architecture

When the Datadog SSI package is installed, the process loads four binaries instead of one monolithic ddtrace.so:

(Sizes vary across versions, but should be of those orders of magnitude)

Loading order (the loader is a Zend extension and fires before any extension= module):

1. dd_library_loader.so MINIT fires
2. Loader calls dlopen(libdatadog_php.so, RTLD_NOW|RTLD_GLOBAL)
3. Loader calls zend_register_internal_module for the SSI ddtrace.so
4. PHP processes extension= directives; any extension=ddtrace.so pointing elsewhere is rejected as duplicate

If a non-SSI ddtrace.so (e.g. a vendor-bundled extension) is also listed in extension=, PHP rejects it as a duplicate after the loader already registered the SSI one — and vice versa (if the non-SSI extension loads first, the SSI loader detects the name already in module_registry and unregisters its own injected copy instead; see loader/dd_library_loader.c).

Tag reference

The following tags may be present alongside the event. Use them to attribute the crash and understand context:

Check whether any profiler counter (profiler_collecting_sample, profiler_unwinding, profiler_serializing) is nonzero — this attributes the crash to profiler activity.

From the mapped files, determine:

• Products loaded: look for ddtrace.so, ddappsec.so, datadog-profiling.so
• SSI mode: check for libdatadog_php.so and dd_library_loader.so — if present, the process is running the SSI (Single-Step Instrumentation) package. See SSI architecture below.
• OS/arch: prefer os_info.architecture (schema 1.8+) over host.arch (which may be empty), but fall back to reading the mapped ld-linux file name
• libc: GNU (ld-linux-x86-64.so) or musl (ld-musl-x86-64.so)

Finally, print the triage summary before continuing.

Phase 2 — Stacktrace correlation

Checkout the matching version tag in a worktree (tags are like 1.16.0). For PHP source, use the php-src repository next to this checkout; PHP tags are like PHP-8.1.33.

PHP runtime frames (experimental.runtime_stack.frames, format: "Datadog Runtime Callback 1.0") contain:

• file / function / line — source location
• type_name — class name when the frame is a method call (e.g. "Couchbase\\Collection")

Note: Ondřej Surý packages for Debian may be slightly modified relative to upstream PHP. If discrepancies appear, use apt-get source inside an appropriate Docker container to obtain the exact source.

Map each native stacktrace frame to source code:

1. Use .claude/scripts/find_map_region.py <address> <event.json> to identify which binary each instruction pointer belongs to. In schema 1.6+ events, frames include a module_base_address field that already identifies the binary — cross-check against the mapped files to confirm the binary name.
2. For frames in Datadog binaries, correlate to source using the worktree.
3. For frames in PHP, correlate to php-src at the matching tag.

To determine whether a frame in a PHP process belongs to a Datadog product: first check the function name (it may clearly identify a DD module); if inconclusive, check whether the instruction pointer falls within the mapped executable segments of ddtrace.so, ddappsec.so, or datadog-profiling.so.

If there are no Datadog product stack frames, still check whether a Datadog product could be the root cause — for example, by having modified the behavior of PHP functions that are clearly on the crashing call path. If this seems unlikely, ask the user before continuing with deep analysis.

If frames land in unknown binaries, note them but focus on Datadog frames first.

If you can identify the root cause at this point, stop and report. Only continue to Phase 3/4 if the analysis is ambiguous or low-confidence.

Note: the authoritative native stacktrace for the crashing thread is .error.stack.frames (format: "Datadog Crashtracker 1.0", always populated when a stack could be captured). The crashing thread name is in .error.thread_name.

error.threads is a per-thread snapshot array present in schema 1.8+. Each element carries a crashed boolean flag, name, state, and stack.{frames, incomplete}. In practice, crashed is often false on every thread and stack.frames is null with incomplete: true — the per-thread stacks are frequently unavailable. Use them as supplementary context only; do not rely on them as the primary frame source.

Phase 3 — Binary verification (if needed)

If the stacktrace correlation is ambiguous or the crash is in Datadog code:

Datadog binaries

1. Download the release binaries:

   • SSI (libdatadog_php.so in maps): fetches from ECR public, no credentials needed:

     .claude/scripts/dd_php_release_url --ssi '<version>' '<arch>'
     

   • Monolithic: fetches from GitHub releases:

     .claude/scripts/dd_php_release_url '<version>' '<php_minor>' '<arch>' '<gnu|musl>'
     

   Both print a temp directory with the extracted package. Use the version exactly as it appears in tracer_version (or reconstructed from library_version).

2. Verify the binary matches the crash by comparing:

   • Size of first mapped region (from /proc/self/maps) vs. p_memsz of the first PT_LOAD segment (readelf -l <binary>). The mapped region spans from the page floor of p_vaddr to the page ceiling of p_vaddr + p_memsz, so the mapped size is the segment's virtual address range rounded outward to page boundaries; mapped size ≥ p_memsz, they need not be equal.
   • Note the base address: map_start - (p_vaddr & ~(PAGE-1)), where PAGE is typically 0x1000 on x86-64 (aarch64 can be 0x1000, 0x4000, or 0x10000). In schema 1.6+ events, module_base_address in each frame is the load bias and can be used directly instead of computing it.

3. Disassemble around the crashing instruction pointer.

   SSI binaries from ECR are stripped but retain exported public symbols — pass the binary directly to GDB:

   gdb -batch -ex 'add-symbol-file <binary> -o <base_addr>' \
       -ex 'disassemble <ip_addr>-32,<ip_addr>+32'
   

   For full symbols (line numbers, inlined functions), try the S3 tarball — it includes co-located .debug files but requires a manual CI job to have been triggered. Check with:

   curl -sI https://dd-trace-php-builds.s3.amazonaws.com/<version>/dd-library-php-ssi-<version>-<arch>-linux.tar.gz
   

   If available, .debug files sit next to each .so in the tarball. Pass to GDB with add-symbol-file <binary>.debug -o <base_addr>.

   For monolithic builds (non-SSI), the binary is unstripped; pass it directly:

   gdb -batch -ex 'add-symbol-file <binary> -o <base_addr>' \
       -ex 'disassemble <ip_addr>-32,<ip_addr>+32'
   

   On macOS, run gdb/readelf/objdump inside a Docker container with --platform=linux/amd64. Build a Dockerfile with the tools you need rather than installing packages each time.

PHP binaries (when PHP frames need deeper analysis)

If source code alone is insufficient to understand PHP frames:

1. Identify and download the PHP binaries. Common sources:

   • Ondřej Surý packages (Debian/Ubuntu)
   • PHP Docker images (Alpine or Debian, binaries installed under /usr/local)

2. Extract the binaries into the same temp directory used for Datadog binaries.

3. Verify each PHP binary matches the crash the same way as Datadog binaries: compare the size of the first mapped region against p_memsz of the first PT_LOAD segment (the mapped region spans the segment's virtual address range rounded outward to page boundaries, so mapped size ≥ p_memsz), note the base address.

4. Disassemble PHP frames using gdb with add-symbol-file at the computed base address.

Phase 4 — Root cause

Based on the disassembly, register state, and source code:

• Identify what memory access or operation triggered the fault.
• Trace backward to find the bug (NULL deref, use-after-free, bad offset, etc.).
• Check if the fault address (si_addr) and the register state are consistent with the hypothesis.

Output

Present findings as:

### Crash Summary
- **Signal**: ...
- **Version**: ... | PHP ... | arch ... | libc ...
- **Products**: ddtrace / ddappsec / profiling
- **Profiler active**: yes/no

### Root Cause
<explanation with source file:line references>

### Evidence
<key register values, disassembly snippet, source correlation>

### Confidence
<high / medium / low — and what would raise it>

Rules

• If no Datadog frames are present, check for indirect causation (DD hooks on the crashing path) before deciding the crash is unrelated. If it still seems unrelated, ask the user whether to continue before doing deep analysis.
• Use git worktree (not git checkout) to inspect version-specific source.
• On macOS, use Docker with --platform=linux/amd64 for ELF tools.
• Parallelize extractions and lookups wherever possible.