Llmobs Integrations
Developmentllmobs-integrationsLlmobs Integrations is an Agent Skill for development published in the DataDog/dd-trace-py repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.
dd-trace-py LLMObs integration development guide. Use when creating, modifying, or debugging LLMObs integrations for LLM/AI libraries in the Python tracer. Covers BaseLLMIntegration, stream handling, message extraction, token counting, tool call parsing, and VCR-based testing patterns. Triggers: "llmobs", "LLMObs", "BaseLLMIntegration", "llmobs_set_tags", "_llmobs_set_tags", "BaseStreamHandler", "submit_to_llmobs", "integration.trace", "LLM span", "VCR", "cassette", "anthropic", "openai", "google_genai", "claude_agent_sdk", "generative-ai", "LLM integration", "llmobs_enabled".
How to use this skill
Bring this guide into your coding agent with a prompt tailored to the tool you use.
- Open your project in Codex.
- Copy and paste the prompt below.
- Review files and risks before approving any action.
Inspect the Agent Skill at https://github.com/DataDog/dd-trace-py/blob/HEAD/.claude/skills/llmobs-integrations/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/llmobs-integrations, but wait for my approval before writing files or running scripts.
Copying this prompt does not install or run the skill. Review third-party files before use.
dd-trace-py LLMObs Integrations
LLMObs integrations enable Datadog LLM Observability for AI/LLM libraries. They extract model inputs, outputs, token usage, and tool calls from traced spans. This skill should be used in addition to the apm-integrations skill.
Two-Layer Architecture
LLMObs integrations consist of two cooperating layers:
- Patch Layer (
ddtrace/contrib/internal/{name}/patch.py) -- wraps library functions. Standard request/response LLM integrations constructLlmRequestEventand usecore.context_with_event()so the LLM tracing subscriber owns span lifecycle and LLMObs tag extraction. - Integration Layer (
ddtrace/llmobs/_integrations/{name}.py) -- extendsBaseLLMIntegration, implements_set_base_span_tags()and_llmobs_set_tags()to extract and set provider-specific messages, tools, metadata, and token metrics.
Both layers must work together. The patch layer identifies the operation and passes request/response data through the event; the integration layer controls what data is extracted.
Active Patch Patterns
- Event-based request spans: Use
LlmRequestEventwithcore.context_with_event()for new standard request/response LLM integrations. Anthropic is the canonical reference. This is the preferred pattern. - Direct integration spans: Some existing or specialized integrations still call
integration.trace()andintegration.llmobs_set_tags()directly, especially for child spans, agent/tool spans, or integrations not yet migrated. Google GenAI, OpenAI tool spans, and Claude Agent SDK are useful references.
Key Files
| Purpose | File |
|---|---|
| Base LLM integration class | ddtrace/llmobs/_integrations/base.py (BaseLLMIntegration) |
| Stream handler base classes | ddtrace/llmobs/_integrations/base_stream_handler.py (BaseStreamHandler, StreamHandler, AsyncStreamHandler) |
| Shared utilities | ddtrace/llmobs/_integrations/utils.py |
| LLMObs annotation helper | ddtrace/llmobs/_utils.py (_annotate_llmobs_span_data) |
| LLMObs constants | ddtrace/llmobs/_constants.py |
| LLMObs types | ddtrace/llmobs/types.py (Message, AudioPart, ToolCall, ToolResult, ToolDefinition) |
| Integration registry | ddtrace/llmobs/_integrations/__init__.py |
Reference Integrations
Always read 1-2 references before writing or modifying LLMObs code.
| Provider | Patch File | LLMObs Integration | LLMObs Tests |
|---|---|---|---|
| Anthropic (canonical) | ddtrace/contrib/internal/anthropic/patch.py | ddtrace/llmobs/_integrations/anthropic.py | tests/contrib/anthropic/test_anthropic_llmobs.py |
| Claude Agent SDK (latest, agent pattern) | ddtrace/contrib/internal/claude_agent_sdk/patch.py | ddtrace/llmobs/_integrations/claude_agent_sdk.py | tests/contrib/claude_agent_sdk/test_claude_agent_sdk_llmobs.py |
| OpenAI | ddtrace/contrib/internal/openai/patch.py | ddtrace/llmobs/_integrations/openai.py | tests/contrib/openai/test_openai_llmobs.py |
| Google GenAI | ddtrace/contrib/internal/google_genai/patch.py | ddtrace/llmobs/_integrations/google_genai.py | tests/contrib/google_genai/test_google_genai_llmobs.py |
Use Anthropic as the canonical reference for standard LLM integrations. Use Claude Agent SDK for agent-pattern integrations (agent spans, tool child spans, thinking blocks).
Abstract Methods to Implement
Subclass BaseLLMIntegration and implement:
_set_base_span_tags(span, **kwargs)
Set provider-specific APM tags on the span (e.g., {name}.request.model).
_llmobs_set_tags(span, args, kwargs, response, operation)
Extract and annotate all LLMObs fields on the span:
| Field | Description |
|---|---|
kind | "llm" for LLM calls, "agent" for agent calls, "tool" for tool calls |
model_name | Model identifier (e.g., "claude-3-sonnet-20240229") |
model_provider | Provider name (e.g., "anthropic", "openai") |
input_messages | List of Message objects from request |
output_messages | List of Message objects from response |
metadata | Dict of sanitized request parameters (temperature, top_p, etc.) |
metrics | Token usage dict with INPUT_TOKENS_METRIC_KEY, OUTPUT_TOKENS_METRIC_KEY, TOTAL_TOKENS_METRIC_KEY |
tool_definitions | List of ToolDefinition objects if tools are passed |
Fields are usually set via _annotate_llmobs_span_data(...), not raw span._set_ctx_items(...).
Key Constraints
submit_to_llmobs=Truemust be set onLlmRequestEventfor event-based request spans or passed tointegration.trace()for direct LLMObs spansctx.dispatch_ended_event()must run on success and error paths for event-based patch wrappers- Streaming must use
BaseStreamHandler/AsyncStreamHandler-- never consume streams directly - Event-based patch wrappers should not call
span.set_exc_info(),span.finish(), orintegration.llmobs_set_tags()directly; the tracing subscriber handles that when the event ends - Direct integration spans must keep
integration.llmobs_set_tags()and span lifecycle handling aligned with the closest current reference - Integration instance must be stored on the module:
module._datadog_integration = MyLibIntegration(integration_config=config.mylib)
Message Types
from ddtrace.llmobs.types import AudioPart, Message, ToolCall, ToolResult, ToolDefinition
# Input/output messages
Message(content="text", role="user")
Message(content="response", role="assistant", tool_calls=[...])
# Audio attachments in multimodal messages
AudioPart(mime_type="audio/wav", content="<base64-audio>")
Message(content="", role="user", audio_parts=[...])
# Tool calls (in output messages)
ToolCall(name="get_weather", arguments={"city": "NYC"}, tool_id="toolu_123", type="tool")
# Tool results (in input messages)
ToolResult(result="72F sunny", tool_id="toolu_123", type="tool_result")
# Tool definitions (from request parameters)
ToolDefinition(name="get_weather", description="...", schema={...})
Debugging Quick Tips
- No LLMObs spans -- check
submit_to_llmobs=True,ctx.dispatch_ended_event(), andllmobs_enabled - Wrong messages -- check message extraction handles multi-part content and tool blocks
- Wrong tokens -- check field name mapping (libraries use different names for token counts)
- Streaming broken -- verify
BaseStreamHandlersubclass, checkfinalize_stream()dispatches the ended event or finishes direct-trace spans according to the reference pattern - DD_TRACE_DEBUG=true to see patching activity and span creation
See Failure Modes for detailed debugging guide.
Reference Files
- Implementation Guide -- LLM-specific steps (references apm-integrations guide for the full workflow)
- Failure Modes -- All 8 failure modes with causes and fixes
- Testing Guide -- LLMObs test patterns, VCR cassettes, suitespec