Validate OTel semantics for gen_ai attributes

[sentry-junior]

Give the plugin a way to check that instrumentation uses correct OpenTelemetry semantics — the right attribute names, and the right attributes paired with operations. gen_ai is the primary focus.

• Today the plugin can guide AI monitoring setup (sentry-setup-ai-monitoring) and OTel exporter wiring (sentry-otel-exporter-setup), but it has no structured way to verify emitted spans/attributes against OTel semantic conventions
• Incorrect or mismatched attributes (e.g. wrong gen_ai.* names, attributes on the wrong operation) break Sentry AI monitoring quality and make telemetry hard to query reliably
• Desired capability: inspect or guide instrumentation so attribute names and operation↔attribute pairings match OTel conventions, with gen_ai as the first high-value domain

Requested by David Cramer via Junior.

--

View Junior Session in Sentry

[sentry-junior]

Audit against current getsentry/sentry-conventions and upstream open-telemetry/semantic-conventions-genai:

Findings

The new canonical guidance in src/skills/sentry-setup-ai-monitoring/SKILL.md is directionally strong: it uses gen_ai.chat, pairs it with gen_ai.operation.name=chat, uses the current gen_ai.input.messages / gen_ai.output.messages shape, and uses gen_ai.tool.call.arguments / .result on gen_ai.execute_tool spans.

There is still substantial semantic drift in the hydrated SDK references:

• src/references/sdks/node/ai-monitoring.md and python/ai-monitoring.md teach op=gen_ai.request. request is not a canonical Sentry op or a well-known OTel gen_ai.operation.name; known LLM calls should use gen_ai.chat, gen_ai.embeddings, gen_ai.generate_content, or gen_ai.text_completion, with matching gen_ai.operation.name.
• Those Node/Python docs also teach deprecated attributes: gen_ai.request.messages → gen_ai.input.messages; gen_ai.response.text / .tool_calls → gen_ai.output.messages; gen_ai.tool.input / .output → gen_ai.tool.call.arguments / .result.
• gen_ai.request.reasoning_effort appears in the skill plus Node/Python/Next/Nest references, but the current canonical attribute is gen_ai.request.reasoning.level.
• Cached/cache-write/reasoning token aliases are stale throughout the skill and references: use gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, and gen_ai.usage.reasoning.output_tokens. The old input_tokens.cached, input_tokens.cache_write, and output_tokens.reasoning forms are explicitly deprecated in Sentry conventions.
• The skill says gen_ai.request.model is required and gen_ai.operation.name optional under “Common (all AI spans)”. This is backwards/over-broad. OTel requires gen_ai.operation.name; gen_ai.request.model is conditional for client calls and only recommended for an agent when the agent has one fixed model. It should not be required on tool/handoff spans.
• Likewise gen_ai.agent.name should not be generically recommended on tool spans. Operation-specific allowed/required sets are needed rather than one common bag.
• gen_ai.handoff is a Sentry-supported op, but it is not currently in the Sentry convention’s well-known gen_ai.operation.name list. We need to explicitly decide/document whether this is a Sentry extension and what attribute pairing is expected, rather than implying pure OTel parity.

This drift exists despite prose saying “do not set deprecated attributes,” which is exactly why a structured validator is worthwhile.

Suggested capability

Use a versioned, generated ruleset sourced from sentry-conventions (with upstream OTel provenance), not a hand-maintained markdown list. For each span, validate:

1. attribute existence/status/type (stable, deprecated, replacement, Sentry extension vs OTel);
2. span.op ↔ gen_ai.operation.name agreement;
3. operation-specific required/conditional/forbidden attributes;
4. structured content shape for input/output messages and tool calls;
5. cross-field invariants such as cached/reasoning counts ≤ totals;
6. hierarchy checks where observable (agent → model/tool/handoff children).

Diagnostics should distinguish errors from migration warnings because Sentry ingestion currently normalizes/backfills several deprecated aliases.

Eval feasibility

High for deterministic static evals; medium for runtime correctness.

A useful first eval suite does not need an LLM judge or OpenRouter: fixture snippets/span JSON + expected diagnostics can cover typos, deprecated aliases, op/name mismatches, required/conditional attributes, wrong types, and token invariants. Add adversarial near-misses and one clean fixture per operation. LLM-based evals are useful later for testing whether the skill repairs code correctly across JS/Python/PHP and explains ambiguity, but they should sit on top of the deterministic semantic oracle—not replace it.