deepeval-otel

DeepEval OpenTelemetry Export

Use this skill to instrument an AI application — an LLM app, agent, RAG pipeline, or chatbot — with raw OpenTelemetry so its traces land in Confident AI's Observatory. No deepeval package is needed — it works with any OTLP-capable OpenTelemetry SDK. The job is exactly two things: export to the correct Confident AI OTLP endpoint, and set the confident.* attributes Confident AI reads off each span.

Scope: AI Applications Only

This skill instruments AI applications only. The confident.* attributes and span types — agent, llm, retriever, tool — describe AI components, and Confident AI's Observatory is built to evaluate and monitor AI behavior.

Instrument only the AI parts of the system: agent loops and planning, LLM calls, retrieval / vector search, and tool calls. Do not apply confident.* attributes to non-AI software (web servers, CRUD backends, database layers, infrastructure) or to non-AI spans inside an otherwise-AI app — that data does not belong in Confident AI and will not render meaningfully. If the target has no LLM, agent, retrieval, or tool-calling component, this skill does not apply.

When to Use vs the deepeval Skill

Use this skill for vendor-neutral OTLP export to Confident AI — pointing an OpenTelemetry exporter at Confident AI and setting confident.* attributes.

Use the deepeval skill when the user wants to build a Python pytest eval suite, generate datasets or goldens, write metrics, run deepeval test run, or instrument with the deepeval SDK's @observe decorator. The two skills are complementary, not alternatives.

Prerequisites

• A Confident AI account and a CONFIDENT_API_KEY.
• An OpenTelemetry SDK for the application's language. For Python: opentelemetry-sdk and opentelemetry-exporter-otlp-proto-http.
• The Confident AI OTLP endpoint accepts HTTP only — never gRPC.

How It Works

Confident AI exposes an OTLP/HTTP traces endpoint. Point any OpenTelemetry span exporter at it with the x-confident-api-key header. Confident AI's exporter then reads confident.* attributes off each span to build the trace and span structure. Parent/child nesting comes from native OpenTelemetry span context, not from any attribute.

Workflow

1. Confirm the target is an AI application (it has LLM calls, an agent loop, retrieval, or tool calls). If it has none of these, stop — this skill does not apply. Then inspect for an existing OpenTelemetry setup (a TracerProvider, span exporters, or an OpenTelemetry Collector) and prefer repointing what exists over adding a parallel pipeline.
2. Choose the endpoint from the API key's region prefix. Read references/endpoint-and-exporter.md.
3. Wire (or repoint) an OTLP/HTTP span exporter with the x-confident-api-key header. For Python, start from templates/confident_otel_setup.py.
4. If the process runs other OpenTelemetry instrumentation or an APM agent (auto-instrumentation for HTTP/DB, Datadog, etc.), isolate the Confident AI export so only AI spans reach it — a dedicated pipeline or a span filter. Read "Export Only AI Spans" in references/endpoint-and-exporter.md.
5. Set confident.span.* attributes on spans; set confident.trace.* for trace-wide fields. Read references/span-attributes.md and references/trace-attributes.md.
6. Honor the OTLP data-type rules: JSON-encode dicts/metadata, use native arrays for string lists. See the Data-Type Rules in span-attributes.md.
7. If the app already emits OpenTelemetry GenAI semantic conventions, read references/gen-ai-fallbacks.md before adding redundant attributes.
8. Verify traces appear in the Confident AI Observatory.

Core Principles

1. Instrument AI components only — agent, LLM, retriever, and tool spans. Never apply confident.* attributes to non-AI software or non-AI spans.
2. Export only AI spans. If the process has other OpenTelemetry instrumentation or an APM agent, isolate the Confident AI pipeline (a dedicated provider or a span filter) so non-AI spans — HTTP requests, DB queries, infra — are never exported to Confident AI.
3. Prefer repointing an existing OTLP exporter over adding a parallel one.
4. The confident.* attribute keys are the entire contract — they are the same in every language, so language choice is irrelevant.
5. Always use OTLP/HTTP. Confident AI's endpoint does not accept gRPC.
6. Honor OTLP data-type rules: attribute values must be primitives or homogeneous primitive lists; JSON-encode dicts and metadata.
7. Set confident.span.type explicitly when it is known; rely on gen_ai.* inference only as a fallback.
8. Never put secrets, credentials, or raw sensitive data into span attributes.

References

Templates