diff --git a/rfcs/2026-04-29-25329-trace-data-model.md b/rfcs/2026-04-29-25329-trace-data-model.md new file mode 100644 index 0000000000000..a7643acafa527 --- /dev/null +++ b/rfcs/2026-04-29-25329-trace-data-model.md @@ -0,0 +1,1254 @@ +# RFC 25329 - 2026-04-29 - Internal Trace Data Model + +This RFC replaces the inner representation of Vector's `TraceEvent`, today a thin newtype over +`LogEvent`, with a strongly-typed, trace-homogeneous container derived from the wire-level +batching of OTLP and Datadog APM traces. Each `TraceEvent` carries one `TraceId`, one `Resource`, +one `Scope`, a `DatadogEventContext`, and the `Vec` belonging to that trace and grouping, +plus the existing `EventMetadata`. The Datadog context isolates source-native state that has no +format-independent semantic home; its default value represents no Datadog state. The container +shape, together with the wire-format mappings specified in the +two sub-RFCs below, yields zero-loss `OTLP -> Vector -> OTLP` and +`Datadog -> Vector -> Datadog` round trips, including across Vector's disk buffers, and gives +transforms a uniform typed surface across the two source formats. + +The full proposal is split across three documents: + +- This document defines the typed data model, the temporary + `TraceEventCompat` coexistence enum and shim mechanism, and Vector's internal + protobuf serialization. The VRL encoding of the typed model is out of scope. +- [Trace Data Model: Datadog Mapping](2026-04-29-25329-trace-data-model/datadog-mapping.md) + specifies the bidirectional mapping between `TraceEvent` and the Datadog agent-to-backend + protobuf, including the cross-format conformance rule for `OTLP -> Vector -> datadog_traces`. +- [Trace Data Model: OTLP Mapping](2026-04-29-25329-trace-data-model/otlp-mapping.md) + specifies the bidirectional mapping between `TraceEvent` and the OTLP wire format. + +The three documents are proposed together and share a single approval. + +## Context + +- [RFC 11851 -- OpenTelemetry traces source](2022-03-15-11851-ingest-opentelemetry-traces.md) + was accepted on the condition that an internal trace model be established before the work + was completed. The OTLP mapping sub-RFC completes that condition for the OTLP side. +- [RFC 9572 -- Accept Datadog traces](2021-10-15-9572-accept-datadog-traces.md) introduced the + `datadog_agent` trace ingest path, which the `datadog_traces` sink can consume but which + does not have a well-defined internal representation. The Datadog mapping sub-RFC supplies + that representation. +- An earlier draft of an internal trace model is available at + [2024-03-22-20170-trace-data-model](https://github.com/hdost/vector/blob/add-trace-data-model/rfcs/2024-03-22-20170-trace-data-model.md); + this RFC supersedes that draft. +- The current implementation in + [`lib/vector-core/src/event/trace.rs`](../lib/vector-core/src/event/trace.rs) is + `TraceEvent(LogEvent)` -- a thin newtype with no type structure. Transforms depend on the + ingesting source's key layout, and cross-format conversions are ad-hoc per sink. +- [vectordotdev/vector#22659 -- Transform between opentelemetry and datadog traces](https://github.com/vectordotdev/vector/issues/22659). + +## Glossary + +This RFC defines the OpenTelemetry-side and informational vocabulary the data model +depends on. Datadog-specific format definitions (`Datadog APM trace format`, `Datadog +Agent OTLP ingest`, `Datadog tracer-to-agent API`) live in the +[Datadog mapping sub-RFC's Glossary](2026-04-29-25329-trace-data-model/datadog-mapping.md#glossary); +the entries below are the format-agnostic shared vocabulary. + +- **OTLP (OpenTelemetry Protocol)**: the wire format the OpenTelemetry project defines for + traces, metrics, and logs. The traces schema lives in + [`opentelemetry/proto/trace/v1/trace.proto`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto), + with shared value types in + [`common/v1/common.proto`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto) + and resource types in + [`resource/v1/resource.proto`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/resource/v1/resource.proto). + When this document says "OTLP" it means that wire schema and the data model it defines + (`ResourceSpans`, `ScopeSpans`, `Span`, `AnyValue`, etc.). +- **OpenTelemetry**: the broader project under which OTLP is one component. References in + this RFC to "OpenTelemetry" name the project's non-wire artefacts: the + [specification](https://github.com/open-telemetry/opentelemetry-specification) and the + [semantic conventions](https://github.com/open-telemetry/semantic-conventions) (the + registry of attribute keys such as `service.name` and `http.request.method`). +- **W3C Trace Context** (informational): the W3C recommendation defining the + [`traceparent` and `tracestate` HTTP headers](https://www.w3.org/TR/trace-context/). The + proposed `TraceFlags` and `TraceState` types correspond to these headers. + +## Cross cutting concerns + +- First-class OpenTelemetry signal support + ([vectordotdev/vector#1444](https://github.com/vectordotdev/vector/issues/1444)). + +## Scope + +### In scope + +- Define `TraceEvent` as one `TraceId`, an array of spans, and supporting resource data, + replacing the current `TraceEvent(LogEvent)`. +- Define the typed surface that supports both wire formats: `TraceEvent`, `Span`, + `Resource`, `Scope`, `DatadogEventContext`, `DatadogAgentEnvelope`, + `DatadogTracerContext`, `DatadogChunkContext`, `DatadogSpanContext`, `Attributes`, + `SpanEvent`, `SpanLink`, `TraceId`, `SpanId`, the closed-with-escape-hatch enums (`SpanKind`, `SpanStatus`, + `SamplingPriority`), `TraceFlags`, and `TraceState`. +- Define the migration strategy that lets each trace-producing or trace-consuming component + migrate independently: an initial sink-local validation stage behind existing buffers, + the temporary + `enum TraceEventCompat { Legacy(LegacyTraceEvent), Typed(TraceEvent) }` + coexistence, the per-source `Legacy -> TraceEvent` shim mechanism keyed on + `vector.trace_legacy_layout`, and the compile-time gating that catches unmigrated + consumers. The permanent `TraceEvent` type is always the typed structure. +- Extend Vector's internal event protobuf with a `TypedTrace` variant alongside the renamed + `LegacyTrace` so trace events cross disk-buffer and `vector` source/sink boundaries + unchanged. +- Specify the effective-equivalence round-trip guarantee for `OTLP -> Vector -> OTLP` and + `Datadog -> Vector -> Datadog` as a model-level claim; the per-wire-format mappings that + satisfy this claim are in the OTLP mapping and Datadog mapping sub-RFCs respectively. + Effective equivalence means backend-observable identity, not byte-level identity; details + the backend does not observe (e.g. the relative ordering of spans from distinct trace + IDs in a non-conforming wire grouping, the grouping of such spans into chunks, or the + assignment of spans among equivalent `ScopeSpans`) may differ. The guarantee applies + to pure-relay pipelines only: any VRL write to a trace event is best-effort and + forfeits the round-trip claim for the modified event. + +### Out of scope + +- New trace sources/sinks (Zipkin, Jaeger, etc.). +- APM stats computation semantics (already covered by RFC 9862). +- Zero-loss cross-format round-trip (`Datadog -> OTLP -> Datadog` or + `OTLP -> Datadog -> OTLP`). +- The VRL language surface for the typed model: path encodings, `del()` + semantics, `trace_span_*` / `trace_flags` functions, enum spellings, duration + units at the VRL boundary, `AttrValue` ↔ `Value` conversion, and both the + former normative VRL subsections and any illustrative VRL sketch. Those + details will be specified when typed VRL is implemented. This RFC still + defines the `remap.trace_representation` opt-in, convert-before-program, and + the rule that VRL writes forfeit the round-trip guarantee. +- Implementation mechanics that do not change a specified contract. Rust API + shapes, crate and constructor internals, protobuf field layouts beyond the + discriminator tags and presence rules, review checklists, and similar + decisions remain implementation-time choices. This RFC and the mapping + sub-RFCs specify the invariants and wire mappings those choices must + satisfy. + +### Zero-loss round-trip exclusions + +The effective-equivalence guarantee does not cover the following model-level input shapes. +Each is justified by a paragraph in the Implementation or Rationale section below. +Wire-format-specific exclusions (the OTLP deprecated-environment rewrite, +Datadog `Span.error` normalization, `meta`/`metrics` producer-side disjointness, etc.) are +declared in the corresponding sub-RFC's Scope section. + +- **All-zero `TraceId` or `SpanId`** are rejected on every ingress path. A source-wire + span or link carrying the zero ID is dropped; an invalid typed event ID rejects that + event. See "Identifiers" for the drop granularity and the sub-RFCs for per-format + detection. +- **Wire-domain normalization**: durations and timestamps outside a destination field's + numeric domain are clamped to the nearest representable endpoint on encode and + reported. Derived timestamps follow the same rule. The mapping sub-RFCs identify + format-specific consequences. + +Every failure, drop, and normalization these documents describe is reported as described by the +[instrumentation specification](../docs/specs/instrumentation.md). These documents +note reporting only where the disposition itself is not the default one. + +Numeric boundary behavior follows one policy throughout the model and mappings: + +- Explicit writes must fit the target field's numeric domain and fail atomically + otherwise. +- Encoding and derived arithmetic never wrap or panic. Values outside the destination + field's domain are clamped to the nearest endpoint and reported. +- Mapping- or relay-generated dropped-count increments saturate and remain observable + after saturation. + +## Pain + +- Transforms written against today's `TraceEvent` depend on the exact key layout the + ingesting source produced. A remap that works for `datadog_agent` traces does not work + for OTLP traces, even when the semantic intent is identical. This is the opposite of how + `Metric` behaves and is the primary blocker to useful trace transforms. +- Cross-format routing (e.g. `opentelemetry` source -> `datadog_traces` sink) requires + bespoke translation reading undocumented magic keys. Each new sink duplicates this work. +- OTLP `ScopeSpans` may contain spans from several traces. Preserving that grouping as one + topology event would force filtering and sampling decisions for unrelated trace IDs to + share an event boundary. +- `TraceEvent` corrupts numeric ID precision via `trace_id as i64` on both the + `datadog_agent` source and the `datadog_traces` sink + ([#14687](https://github.com/vectordotdev/vector/issues/14687)). +- VRL programs authoring spans without typed events, links, or status can produce + structurally invalid output that is only discovered at sink encoding time. + +## Proposal + +### User Experience + +During coexistence, a `remap` transform must opt in to the typed representation: + +```yaml +transforms: + typed_traces: + type: remap + inputs: + - traces + trace_representation: typed + source: "." +``` + +`trace_representation` initially defaults to `legacy`. Typed mode accepts only trace input and +converts before running VRL; path access never converts. Conversion partitions a legacy event +into trace-homogeneous typed events: an OTLP grouping becomes one event per distinct wire +`Span.trace_id`, and a Datadog chunk becomes one event per distinct reconstructed wire trace +ID and `Span.service` pair. Each ID is stored at the resulting event root. The program then +runs independently on each result. VRL conditions do not convert, so typed conditions require +an upstream typed `remap`. Because conversion is one-way, legacy mode rejects typed input with +a representation-mismatch mapping error. + +The typed `TraceEvent` is uniform across both wire formats: the same fields are valid +regardless of source. `TraceEvent.datadog` and each `Span.datadog` isolate Datadog-native +wire state; their default values represent no Datadog-native state. `DatadogEventContext.agent` +and `DatadogEventContext.chunk` are `None` when their wire envelopes are absent. +Format-specific encoding details are documented in the OTLP mapping and Datadog mapping +sub-RFCs. The VRL encoding of these fields is out of scope. + +This uniformity is also the ingest invariant for every trace source: when the source wire +format carries data that has a typed home in the model, ingest stores it in the +corresponding typed struct field rather than leaving it encoded only under source-specific +attribute keys. The Datadog contexts hold Datadog-native wire state with no +format-independent semantic home. Attribute maps hold common semantic attributes rather +than source-envelope sub-objects. Sink egress projects from that typed surface, not from +source-specific ingest layouts. + +The `trace_to_log` transform is retained and emits exactly one `LogEvent` per +`TraceEvent`; it does not fan out spans. The log root is the canonical typed object +with exactly the `trace_id`, `resource`, `scope`, `datadog`, and `spans` fields. +`EventMetadata` and finalizers transfer to the log as event metadata rather than being +inserted into its fields. During coexistence, converting one legacy input may first +produce several trace-ID partitions and therefore several logs, one per resulting +`TraceEvent`. This defines one source-independent output shape; the user migration +guide provides the old-to-new field mapping and examples. + +### Implementation + +#### `TraceEvent` + +```rust +pub struct TraceEvent { + trace_id: TraceId, + resource: Resource, + scope: Scope, + /// Datadog-native state. The default value means that none is present. + datadog: DatadogEventContext, + /// Spans belonging to `trace_id` within this resource/scope grouping. + spans: Vec, + metadata: EventMetadata, +} +``` + +Each `TraceEvent` carries a single `TraceId` and `Resource` -- meaning a single service. +Every span belongs to that event-level trace ID. + +The mapping to wire-level structures differs by format: + +- **OTLP**: each `ScopeSpans` is partitioned by the wire `Span.trace_id`, yielding one + `TraceEvent` per distinct ID with that ID stored in `TraceEvent.trace_id`. The + enclosing `ResourceSpans` provides `Resource`. See the OTLP mapping sub-RFC for the + grouping algorithm and per-field mapping. +- **Datadog**: each `TraceChunk` is partitioned by the full trace ID reconstructed from + the wire `Span.traceID` and `_dd.p.tid`, and by wire `Span.service`. The Datadog + mapping sub-RFC specifies the grouping and re-coalescence rules. + +#### `Span` + +```rust +pub struct Span { + pub span_id: SpanId, + pub parent_span_id: Option, + pub trace_state: TraceState, + pub flags: TraceFlags, + + pub name: String, + pub kind: SpanKind, + + pub start_time: DateTime, + /// Span duration with nanosecond precision. + pub duration: Duration, + pub status: SpanStatus, + + /// Datadog-native state. The default value means that none is present. + pub datadog: DatadogSpanContext, + + /// Per-span attribute map. + pub attributes: Attributes, + + pub events: Vec, + pub links: Vec, + + pub dropped_attributes_count: u32, + pub dropped_events_count: u32, + pub dropped_links_count: u32, +} +``` + +`TraceEvent.datadog` and `Span.datadog` isolate Datadog-native wire state while remaining +part of the single typed schema. An event with no Datadog wire state lifted on ingress +carries default contexts. Typed consumers read and write the same fields regardless of +source, and Datadog egress treats explicitly authored namespace +state as authoritative without consulting event provenance. Typed namespace slots are +preferable to format-discriminated event variants because common consumers still +operate on one `TraceEvent` shape. + +#### Datadog-native contexts + +```rust +#[derive(Clone, Debug, Default)] +pub struct DatadogEventContext { + pub agent: Option, + pub tracer: DatadogTracerContext, + pub chunk: Option, +} + +#[derive(Clone, Debug, Default)] +pub struct DatadogAgentEnvelope { + pub host_name: String, + pub env: String, + pub agent_version: String, + pub target_tps: f64, + pub error_tps: f64, + pub rare_sampler_enabled: bool, + pub tags: Attributes, +} + +#[derive(Clone, Debug, Default)] +pub struct DatadogTracerContext { + pub tags: Attributes, +} + +#[derive(Clone, Debug, Default)] +pub struct DatadogChunkContext { + pub priority: Option, + pub origin: Option, + pub dropped: bool, + pub tags: Attributes, +} + +#[derive(Clone, Debug, Default)] +pub struct DatadogSpanContext { + /// Datadog `Span.resource`. + pub resource_name: Option, + /// Datadog `Span.type`. + pub span_type: Option, + /// Datadog `Span.meta_struct`. + pub meta_struct: BTreeMap, +} +``` + +The namespace contains only source-native Datadog wire state. Normalized trace semantics, +including trace and span identifiers, status, common resource fields, events, links, and +ordinary attributes, remain in the format-independent model. `DatadogEventContext` and +`DatadogSpanContext` are logically always present and default-valued. + +#### `Resource` and `Scope` + +```rust +pub struct Resource { + pub service: Option, // service.name + pub environment: Option, // deployment.environment.name + pub host: Option, // host.name + pub attributes: Attributes, + pub schema_url: Option, + pub dropped_attributes_count: u32, +} + +pub struct Scope { + /// `None` carries the OTLP "instrumentation scope name unknown" semantics. + pub name: Option, + pub version: Option, + pub attributes: Attributes, + pub schema_url: Option, + pub dropped_attributes_count: u32, +} +``` + +#### Identifiers + +```rust +pub struct TraceId(NonZeroU128); +pub struct SpanId(NonZeroU64); +``` + +Wire mappings must reject malformed structural input rather than truncate, pad, or +guess its meaning unless that mapping explicitly defines a normalization. The mapping +sub-RFCs define the rejection granularity and telemetry. + +Unless a mapping defines a stricter disposition, an unrepresentable malformed leaf drops +the smallest independently countable model item that contains it: an invalid attribute +value or nested element drops that attribute entry; an invalid link or event field drops +that link or event; and an invalid required span field drops the span. The corresponding +in-band dropped count is saturating-incremented when the receiving or destination +representation carries one, and the drop is reported. Sibling items continue through +the mapping. +If a non-empty wire grouping or candidate partition yields no accepted spans, it +produces no event and the drop is reported. + +Zero `TraceId` and `SpanId` values are unrepresentable in a well-formed event by the +`NonZero` types above. Every construction site rejects and reports zero inputs. On +source-wire ingress, an invalid trace ID drops the span from which the event-level ID +would have been derived; an invalid span or link ID drops that span or link. On internal +`TypedTrace` proto decode, a zero or wrong-length top-level `trace_id` rejects the event +as corruption, while the corresponding invalid `span_id` or link ID follows the +smallest-item disposition above. + +A `TraceEvent` whose `spans` vector becomes empty retains its top-level `trace_id` and +passes through unchanged. Sinks emit the corresponding empty wire shape, fire +finalizers on successful delivery, and report the condition. OTLP and Datadog ingress +produce no event from an empty wire grouping because neither format supplies an ID for +it. The internal proto and typed JSON retain the ID on empty events. + +#### Status, kind, and sampling priority + +```rust +pub enum SpanKind { + Unspecified, + Internal, + Server, + Client, + Producer, + Consumer, + /// Unrecognized enum number from a newer OpenTelemetry version. Stored + /// verbatim so an OTLP -> Vector -> OTLP relay emits the original wire + /// value unchanged. See "Closed-with-escape-hatch enum invariant" below + /// for the construction-time normalization rule. + Other(i32), +} + +pub enum SpanStatus { + Unset, + Ok, + Error(String), + /// Unrecognized status code from a newer OpenTelemetry version. The raw + /// code integer and any status message are stored verbatim so an + /// OTLP -> Vector -> OTLP relay emits the original wire values unchanged. + /// See "Closed-with-escape-hatch enum invariant" below. + Other(i32, String), +} + +pub enum SamplingPriority { + UserReject, // -1 + AutoReject, // 0 + AutoKeep, // 1 + UserKeep, // 2 + /// Out-of-range value. Datadog tracing libraries may uncommonly emit + /// these. See "Closed-with-escape-hatch enum invariant" below. + Other(i32), +} +``` + +##### Closed-with-escape-hatch enum invariant + +`SpanKind`, `SpanStatus`, and `SamplingPriority` share a single invariant: a value +matching a known variant's wire number is always carried as that variant, never as +`Other(n)`. Every construction site must normalize raw values through a shared validated +constructor. + +The typed model and wire codecs can transport `Error("")`, which is required for +Datadog `error != 0` spans that carry no `error.message`. + +#### Empty strings in optional string slots + +`Option` typed slots preserve `Some("")` when the source mapping supplies an +empty value; `None` means absent or unset. There is no model-wide empty-string +normalization. A mapping may collapse empty to `None` only where its wire format makes +the values indistinguishable or defines them as equivalent; each such case is specified +in the corresponding sub-RFC. + +#### `TraceFlags` and `TraceState` + +`TraceFlags` is the OTLP `Span.flags` / `Link.flags` bitfield: a 32-bit word whose low +byte is the W3C trace-flags byte and whose remaining bits carry OTLP- and Datadog-defined +context information. Its implementation must retain the full raw word, including unknown +bits, so OTLP's reserved bits 10-31 and future additions round-trip unchanged. It exposes +the low W3C trace-flags byte and the OTLP parent- / link-target-remoteness tristate as +derived views without narrowing the stored value. + +`TraceState` stores the W3C `tracestate` header verbatim. Sources copy the header in +unchanged; sinks emit it unchanged +unless a transform mutated it. Vector does not validate the header against the W3C +grammar, enforce the 32-entry or 512-byte limits, reject invalid members, or deduplicate +keys: the raw string is preserved and re-emitted as-is. Validation is the responsibility +of the producing tracing SDK, not the relay. This is consistent with `TraceFlags`, which +also preserves unknown bits without validation. Any structured Rust accessor API is an +implementation-time choice. + +#### Events and links + +```rust +pub struct SpanEvent { + pub name: String, + /// Epoch (`1970-01-01T00:00:00Z`) represents "timestamp unknown" per OTLP. + /// On both OTLP and Datadog egress, epoch round-trips as `time_unix_nano = 0` + /// (the proto3 default). + pub time: DateTime, + pub attributes: Attributes, + pub dropped_attributes_count: u32, +} + +pub struct SpanLink { + pub trace_id: TraceId, + pub span_id: SpanId, + pub trace_state: TraceState, + pub flags: TraceFlags, + pub attributes: Attributes, + pub dropped_attributes_count: u32, +} +``` + +`SpanLink.trace_id` remains on the link because it identifies the linked trace target, +which may differ from the enclosing `TraceEvent.trace_id`. + +#### `Attributes` and `AttrValue` + +```rust +pub struct Attributes(BTreeMap); + +/// Mirrors OTLP `AnyValue`. +pub enum AttrValue { + String(String), + Bytes(Bytes), + Bool(bool), + Int(i64), + Double(f64), + Array(Vec), + Map(BTreeMap), + Null, +} +``` + +`AttrValue` is the storage type for every attribute leaf in the model +(`Span.attributes`, `SpanEvent.attributes`, `SpanLink.attributes`, +`Resource.attributes`, `Scope.attributes`, Datadog context tag maps, and recursively +into nested `Map` / `Array` values). The `Attributes` newtype exists so future invariants +(key validation, size bounds) can be added without requiring a migration. +Per-format wire mappings live in the sub-RFCs. + +#### Typed slot/attribute-map pairs + +Several common typed slots correspond to attribute-map keys that wire formats also use: + +- `Resource.service` versus `Resource.attributes."service.name"`. +- `Resource.environment` versus `Resource.attributes."deployment.environment.name"`. +- `Resource.host` versus `Resource.attributes."host.name"`. +- `Span.status` (`Error` / `Other` message) versus `Span.attributes."error.message"`. + +The in-memory model permits both forms to coexist. The typed slot is the canonical +field; the matching attribute-map key exists only as a wire-shape +detail. Ingress lifting (when an attribute key populates a typed slot) and egress +synthesis (when a typed slot populates an attribute key or a canonical wire location) +are wire-format-specific and are specified by each mapping sub-RFC. + +#### Retention of `TraceEvent` and `Event::Trace` + +The outer `Event::Trace` variant is retained. During coexistence its payload temporarily +changes from today's `TraceEvent(LogEvent)` to `TraceEventCompat`; after coexistence +`TraceEventCompat` becomes an alias for the typed `TraceEvent`. This avoids adding a +second trace dispatch arm to `Event`. + +#### Migration: coexistence of `LogEvent` and typed representations + +`TraceEvent` is permanently the typed struct defined above. During migration, the public +temporary compatibility enum carried by `Event::Trace` and `TraceArray` is: + +```rust +pub struct LegacyTraceEvent(LogEvent); + +pub enum TraceEventCompat { + /// Pre-migration source output: an untyped `LogEvent` whose key layout is + /// defined by the producing source. The producing source identifies its + /// layout in `LogEvent.metadata().value()` under the reserved sub-key + /// `vector.trace_legacy_layout` so the correct shim can be selected + /// after fan-in, disk-buffer round-trips, or `vector` source/sink hops. + Legacy(LegacyTraceEvent), + /// The permanent typed container. + Typed(TraceEvent), +} +``` + +Trace event-producing sources each set the reserved sub-key `vector.trace_legacy_layout` +in `EventMetadata.value` to a static string identifying themselves on every +`TraceEventCompat::Legacy` trace they emit. A source that flips to +`TraceEventCompat::Typed` continues setting the same hint until the coexistence and +deprecated-proto window ends, so independently migrated producers retain durable +origin across disk buffers and `vector` hops. Conversion reads this sub-key to select +the corresponding `LegacyTraceEvent -> TraceEvent` shim. For a pre-precursor record +with no hint, conversion runs each registered shim's format-shape detector. Exactly +one match selects that shim; zero or multiple matches return a reported error. +A present but unrecognized hint returns the same error without attempting detection. + +`TraceEventCompat` owns only migration-boundary behavior: + +- Metadata, finalizer, allocation-size, and event-count operations dispatch to either + variant so generic topology and buffering code can operate without converting. +- Temporary untyped forwarding methods preserve the legacy component API while every + producer still emits `TraceEventCompat::Legacy`. They are removed before any producer + flips to typed output. +- Typed field accessors exist only on `TraceEvent`. `TraceEventCompat` does not expose + them, so typed code cannot access an unconverted legacy event. This type separation, + rather than a runtime check, is the enforcement mechanism. +- A converting consumer (a protocol sink, `remap` in typed mode, or `trace_to_log`) + consumes `TraceEventCompat` at every trace-event intake path through a fallible + conversion yielding zero or more `TraceEvent`s. The conversion must be one-to-many: + each typed event carries a single `Resource.service` and one `TraceId`, so a Datadog + legacy event becomes one event per distinct reconstructed wire trace ID and + `Span.service` pair, and each `ScopeSpans` in an OTLP legacy event becomes one event + per distinct wire `Span.trace_id`. Each group ID becomes `TraceEvent.trace_id`; + converted spans carry no duplicate. The + `Legacy` arm selects its shim from the hint, or from the unique format-shape detector + match for a hintless record. A conversion error retains the original + `LegacyTraceEvent` with its metadata and finalizers so the caller can report and drop + it under normal acknowledgement semantics. There is no reverse conversion. +- Pass-through transforms that emit traces (`sample`, `delay`, and similar) are + representation-preserving: they emit the same variant they received and never convert + `Legacy` to `Typed`. On typed input, the default `.sample_rate` write has no carrier + and is omitted; `Legacy` still receives it. +- Successful conversion clones the original `EventMetadata`, including its + `EventFinalizers`, onto every resulting `TraceEvent`, so acknowledgements wait for + every result as they do for Vector's existing fan-out. A conversion that yields no + events consumes the original without failing the batch. Per-span drops reported while + converting a well-formed layout are not conversion failures; the remaining spans still + produce typed events. +- `remap.trace_representation` selects the VRL representation before program execution. + Typed mode converts before constructing `VrlTarget` and runs the program once per + resulting typed event; legacy mode accepts only `Legacy`. Pre-execution conversion + failure uses the normal mapping-error, `drop_on_error`, and `reroute_dropped` + behavior against the original event, while a conversion yielding no events is not a + mapping error and emits nothing. Once converted, an event remains typed even when a + later VRL failure forwards it unmodified. + +Per-component shims are unidirectional (`LegacyTraceEvent -> TraceEvent` only). The +`datadog_agent` source ships with a shim and format-shape detector that know the +source's `LogEvent` key layout and produce a typed container; the OTLP source ships +with the equivalent shim and detector for its layout. + +Removing the temporary untyped forwarders catches unmigrated Rust consumers at compile +time. It does not migrate user VRL, which must set +`remap.trace_representation = "typed"` before producers flip. + +After every source, sink, and transform has been migrated, the compatibility enum and +`LegacyTraceEvent` are removed and `TraceEventCompat` becomes a type alias for +`TraceEvent`. This typed-only release changes the `remap` default to `typed`, rejects +`legacy`, and retains the option for a deprecation window. The legacy converters and +detectors remain temporarily as deprecated-proto wire decoders. + +#### Wire serialization + +Trace events cross internal-wire boundaries through disk buffers and the `vector` +source/sink, so Vector's event protobuf (`lib/vector-core/proto/event.proto`) needs wire +shapes for both migration variants. `EventWrapper` and `EventArray` retain field tag 3 +for the renamed `LegacyTrace` / `LegacyTraceArray` variants and add sibling +`TypedTrace` / `TypedTraceArray` variants at field tag 4. The legacy message field tags +and shapes remain unchanged. + +The typed payload encodes the `TraceEvent` defined above. +Proto presence preserves every `Option` distinction and +`datadog.chunk = None` versus `Some(DatadogChunkContext::default())`; an unset +`AttrValue` oneof represents +`AttrValue::Null`. Identifiers use their non-zero unsigned wire domains, trace IDs are +16-byte big-endian values, timestamps and durations are fixed-width unsigned +nanoseconds, and flags retain the full 32-bit word. Timestamp encoding applies the +numeric boundary policy defined in Scope. + +A decoded `TypedTrace` whose top-level `trace_id` is zero or not exactly 16 bytes is +treated as internal-wire corruption and rejected. + +The oneof tag is the discriminator. The fallible decode boundary (see Plan of Attack) is +a hard prerequisite for the typed proto step and must ship first. An older Vector that +has the fallible decode boundary but not the `TypedTrace` variant receives a +`typed_*`-tagged message and surfaces a controlled unknown-variant error; the consumer +drops and reports the affected message. The pipeline continues running. All-Legacy +traffic decodes correctly on any Vector version that supports field tag 3, since +`LegacyTrace` / `LegacyTraceArray` keep that tag. `vector` source/sink chains that span +the typed migration roll out receiver-first: every downstream `vector` receiver must +support `TypedTrace` before an upstream peer may send typed trace records to it. The +migration-boundary release (fallible decode plus the legacy-layout hint precursor) is +crash-safe but not lossless for typed traffic; it does not satisfy the receiver +prerequisite. + +Single-event encoding via `EventWrapper` is 1:1: +`TraceEventCompat::Legacy` encodes as `LegacyTrace` at tag 3 and +`TraceEventCompat::Typed` encodes as `TypedTrace` at tag 4. Array encoding is +1:1-or-N: during coexistence, in-memory `TraceArray` (a +`Vec`) can hold a mix of variants when a source that emits typed +events natively and one that still emits legacy events fan in to the same downstream +component, but the wire `EventArray.events` oneof must select one variant. A mixed +array is therefore encoded as several homogeneous wire arrays. Encoding must preserve +the original event order and each event's finalizer and acknowledgement behavior, so +that a mixed array is observationally equivalent to a homogeneous one for every +downstream stateful component. Decoders see only homogeneous wire arrays; mixing +reappears at fan-in points downstream. + +Retiring the `TraceEventCompat` enum does not immediately retire the proto: +`LegacyTrace`, +`LegacyTraceArray`, and the `legacy_*` oneof variants are first marked +`deprecated = true` for a release window so records written by older Vector instances +continue to decode when their layout can be identified safely. The per-component +conversion routines and detectors persist alongside the deprecated proto as wire +decoders. During coexistence, a tag 3 record decodes as +`TraceEventCompat::Legacy` and converts at a typed consumer boundary. After the +compatibility enum is retired, the tag 3 decoder performs that same fallible conversion +immediately and returns a `TraceEvent`; a tag 4 record decodes directly to +`TraceEvent` after the typed proto's own validation. A +`vector.trace_legacy_layout` hint (which travels in `EventMetadata.value` inside +`LegacyTrace`) selects the decoder directly; a pre-hint record is accepted only when +exactly one registered detector recognizes its legacy `LogEvent` shape. Zero or multiple +matches produce a controlled, reported error rather than guessing. +Deployments carrying a legacy layout that cannot be detected uniquely must drain or +segregate those pre-hint disk buffers and older-peer streams before installing the +typed-only release. + +Legacy wire support may retire only after all of the following hold: + +- The minimum compatibility interval required by Vector's supported rolling-upgrade + policy has elapsed. If no such policy is documented, retirement remains blocked. +- No Vector release still inside that supported upgrade set emits legacy trace records. +- The published deadline for draining or segregating pre-hint buffers and older-peer + streams has elapsed. + +Only then are the proto messages removed, field tag 3 reserved in both oneofs, and the +conversion routines and detectors deleted. + +#### JSON encoding + +The `json` and `native_json` codecs serialize `TraceEventCompat` untagged: `Legacy` is +today's inner map and `Typed` is the canonical `{trace_id, resource, scope, datadog, spans}` +object. `EventMetadata` is omitted. `native_json` still wraps the payload as `{"trace": ...}`. +These codecs emit the representation they received and do not convert. + +The generic `json` decoder remains log-only. Trace decode uses `native_json`, whose +outer `{"trace": ...}` tag is the discriminator. The payload under that tag is +untagged: the canonical `{trace_id, resource, scope, datadog, spans}` object is +`Typed`; any other object is `Legacy`. Matching both the typed shape and a Datadog or +OTLP detector is a reported error. + +## Rationale + +### Architectural choices + +- The container shape refines the wire-level batching of OTLP and Datadog by partitioning + each wire grouping into trace-homogeneous events. This makes the event a safe unit for + trace-ID-local transforms while retaining resource-, scope-, and chunk-scoped sharing + within each partition. A complete distributed trace may still span several events, so + trace-wide transforms remain stateful. Partitioning at source or legacy-conversion + ingress ensures every downstream consumer sees the invariant and lets fan-out retain + the original metadata, finalizers, and wire order. +- The always-present Datadog namespaces separate native wire state from normalized trace + semantics without introducing a format-discriminated event union. Their default values + mean no Datadog state. `Option` still distinguishes no chunk state + from an explicitly default-valued Datadog `TraceChunk`; protobuf message presence carries + that distinction across hops. +- Typed fields, with a single shape regardless of source, let transforms be written once. + `Metric` already demonstrates this in Vector; extending it to traces unblocks RFC 11851. + Typed-first ingest keeps cross-format egress a projection from that shared model rather + than a translation from source-specific layouts. +- Keeping the outer `Event::Trace` variant unchanged and temporarily changing only its + payload to `TraceEventCompat` avoids a second dispatch arm at every topology, buffer, + and finalizer site. Separating the wrapper from `TraceEvent` keeps the permanent type + statically typed throughout the migration. + +### Per-type design choices + +- `Resource` promotes only the three semantic-convention fields both wire formats agree + on (`service.name`, `deployment.environment.name`, `host.name`). Promoting more would + force Vector to track upstream convention evolution or ossify a stale subset; promoting + fewer would force every cross-format transform to read source-specific keys for common + metadata. Format-specific consequences live in the corresponding mapping sub-RFC. +- Non-zero `TraceId` / `SpanId` types eliminate a class of malformed values by + construction: OTLP defines all-zero IDs as invalid, and Datadog uses zero only as the + "no parent" sentinel (already `None`). Unsigned integers also fix the existing + `i64`-coercion precision bug + ([#14687](https://github.com/vectordotdev/vector/issues/14687)). +- `TraceFlags` is sized to the OTLP wire field (`u32`), not the W3C `traceparent` byte + (`u8`), so an `OTLP -> Vector -> OTLP` relay round-trips the full `Span.flags` / + `Link.flags` word: the W3C trace-flags byte (bits 0-7), OTLP's parent- / link-target- + remote tristate (bits 8-9), and OTLP's reserved bits 10-31. The same width is required + on the Datadog link path, where `SpanLink.flags` is `uint32` and bit 31 is a + "flags-are-meaningful" sentinel; a `u8` storage would clear that bit on every Datadog + link. Unknown bits, including forward-compat W3C additions such as the Level 2 + `random` flag, must round-trip without changing the type. A bitfield representation + that dropped undefined bits would lose that data. +- `Span.duration` is stored as integer nanoseconds because both wire formats carry + duration that way, while conversion to floating-point seconds progressively loses + nanosecond precision as magnitude grows. Wire-domain corner cases are clamped on the + corresponding boundary and declared as exclusions in the relevant sub-RFC. +- `Attributes` stores leaves as `AttrValue` rather than VRL `Value` so the wire + string-versus-bytes discriminator and NaN doubles are preserved structurally. That + avoids several round-trip exclusions from both sub-RFCs at the cost of one conversion + layer at a later VRL boundary. The reuse alternative and the wider VRL `Value` fix are + recorded under Alternatives and Future Improvements. + +### Migration approach + +- Before changing the topology event payload, the sink-local stage defined below uses the + canonical model and shared per-format routines inside trace protocol sinks after their + existing buffers. This validates the model, mappings, and egress contracts while the + rest of the system remains legacy. +- The migration uses a temporary + `enum TraceEventCompat { Legacy(LegacyTraceEvent), Typed(TraceEvent) }` as the + `Event::Trace` payload after that sink-local validation so each trace source, sink, + and transform can migrate in its + own PR while the rest of the system continues to operate against the representation + it expects. `TraceEvent` itself is always typed. See "Wholesale migration" under + Alternatives for why a single atomic replacement was rejected. +- Per-component shims convert `LegacyTraceEvent -> TraceEvent` only, never the reverse: + neither the migration hint nor the typed model can reconstruct a source-specific + legacy layout after arbitrary transforms. Converting consumers (protocol sinks, typed `remap`, + `trace_to_log`) therefore migrate before producers; pass-through transforms accept + both variants without converting so a `source -> sample -> remap` pipeline keeps + working while `remap` still defaults to `legacy`. Removing the untyped forwarders + makes omissions compile errors. User VRL migrates through the explicit `remap` mode, + available for an announced interval before producers flip. +- Shim selection is keyed on a reserved sub-key `vector.trace_legacy_layout` in + `EventMetadata.value` set by the producing trace source. The `vector` metadata + namespace is read-only to VRL, so transforms between source and sink cannot + accidentally delete or overwrite the hint. The metadata `Value` is serialized with + every event record and passes through fan-in, disk buffers, and `vector` source/sink + hops unchanged (unlike `EventMetadata.source_type`, which the topology rewrites on + every emission and so cannot serve as the selector across a serialised hop). + Conversion is explicit outside VRL; only a resulting `TraceEvent` exposes typed + accessors. Hintless records written before the precursor use format-shape detectors + and convert only on exactly one match; this fallback does not rely on + `EventMetadata.source_type`. The convention lives only for the duration of the + migration and deprecated-proto window; no new permanent struct field or wire-format + extension is needed. + +## Drawbacks + +- The eventual typed-source flip remains a breaking change for VRL configurations against + today's key layout. Explicit `remap` opt-in provides a migration window but cannot + preserve legacy programs after typed-only producers ship. +- The `trace_to_log` transform's output also changes; downstream VRL programs against + its output must update. +- Pipelines that encode traces with `json` or `native_json` (for example `file` and + `kafka`) also change when the payload becomes typed. During coexistence those codecs + remain untagged, so legacy output is unchanged until a source flips or a typed + `remap` sits upstream. Downstream readers of that JSON must update. +- Topology granularity is coarser than per-span: each event carries the spans for one + `TraceEvent.trace_id` within one resource/scope or Datadog chunk/service partition. Buffer-size + limits expressed in events still bound span counts less directly than the previous + `LogEvent`-per-span design. Event-count accounting continues to return `1` per + `TraceEvent` (not per span), so `component_received_events_total` and related + accounting count container events, not individual spans. This is consistent with how + `Metric` reports (one event per metric, not per sample); dashboards that expect + per-span event counts will read lower values than the actual span throughput. +- Partitioning duplicates the event's `Resource`, `Scope`, Datadog context, and metadata + once per emitted trace-ID/resource partition (or trace-ID/service partition for + Datadog). The duplication persists across disk serialization even if the in-memory + implementation shares immutable values. +- A wire grouping containing `K` distinct trace IDs produces `K` or more typed events + when service partitioning also applies. Event-count telemetry and typed `remap` + execution both increase with that fan-out. A program that intended to aggregate over + an entire `ScopeSpans` must use stateful aggregation rather than one remap invocation. +- Per-span operations (filter, sample, mutate one span) require iteration over the + event's spans vector rather than per-event treatment. A future topology-level expand-on-input / + collapse-on-output shim could let single-span transforms operate unchanged. +- The internal `event.proto` gains a new `TypedTrace` variant alongside the renamed + `LegacyTrace`. `vector` source/sink chains spanning the typed migration must upgrade + receivers before senders. A receiver at the migration-boundary release but without + the `TypedTrace` variant drops and reports the message; the pipeline continues running, + but the hop is not lossless. See "Wire serialization" for details. This ordering is + documented in the release notes alongside the remap-mode migration. +- Pre-hint legacy records depend on unique format-shape detection after consumers become + typed-only. Any unrecognized or ambiguous historical layout must be drained or + segregated before the typed-only release rather than being converted heuristically. +- Every trace source and sink must be rewritten to produce/consume the typed container. + The Plan of Attack sequences this so each component migrates independently, but it is + non-trivial work. +- The sink-local validation stage temporarily converts replayed legacy records after they + leave each protocol sink's buffer. Conversion therefore repeats after replay, and the + post-buffer adapter is removed once the shared coexistence boundary reaches the sink. +- The temporary public `TraceEventCompat` type and its wrapping and unwrapping at + component boundaries add migration-only API and mechanical changes. The foundational + migration step must rename today's untyped `TraceEvent` representation to + `LegacyTraceEvent` and adapt existing producers to wrap it, and the cleanup step must + replace the compatibility enum with an alias to `TraceEvent`. +- Wire-format-specific drawbacks (Datadog producer-side keyset-disjointness convention, + Datadog `Span.error` normalization, OTLP `deployment.environment` legacy-key rewrite, + etc.) are listed in the corresponding mapping sub-RFC. + +## Prior Art + +- [OTLP traces protocol](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto) + -- the primary shape this RFC adopts. The container `TraceEvent` is structurally one + trace-ID partition of a `ScopeSpans` plus its `Resource`; always-present Datadog event + and span namespaces carry the source-native delta that has no common semantic home. +- Each mapping sub-RFC's own Prior Art section covers the wire format it targets and the + reference implementations for it, including the Datadog APM protobuf and the Datadog + Agent's OTLP ingest. +- [2024-03-22-20170 draft](https://github.com/hdost/vector/blob/add-trace-data-model/rfcs/2024-03-22-20170-trace-data-model.md) + -- an earlier draft that modelled the event as a `ResourceSpans` (batch of multiple + scope/spans groupings). The current RFC adopts a similar container shape but at finer + granularity (one event per trace-ID partition within a `ScopeSpans` rather than per + `ResourceSpans`). + +## Alternatives + +Wire-format-specific alternatives are documented in the corresponding mapping sub-RFC. + +### OTLP-only schema with Datadog round-trip via import/export encoding + +Adopt only OTLP-derived fields in the internal model -- `TraceEvent` carries one +`TraceId`, one `Resource`, one `Scope`, and a `Vec`, with no Datadog-specific typed fields -- and +achieve `Datadog -> Vector -> Datadog` round-trip transparency through an +import/export layer that encodes every Datadog-specific concept under reserved attribute +keys. This moves the proposal's event- and span-level Datadog contexts into internal +attribute maps rather than limiting reserved attributes to the OTLP wire bridge. + +The appeal is OTLP's status as the de facto industry trace schema. A single canonical +container removes `TraceEvent.datadog`, `Span.datadog`, and the `SamplingPriority` enum +from the API surface, leaving only `TraceId` and the OpenTelemetry-shaped +`Resource` / `Scope` / `Span`. +Cross-format consumers see one schema. + +Rejected because the encoding required to carry all Datadog-specific concepts under OTLP +attributes without data loss is not uniform with how OTLP-sourced data sits in the same +attribute maps, and the non-uniformity is observable to every transform on the typed +surface: + +- **Chunk-scoped state has no faithful per-span encoding.** `TraceChunk.{priority, + origin, droppedTrace, tags}` apply uniformly to every span in the chunk; the only + place to carry them under a pure-OTLP schema is on every `Span.attributes` map in the + chunk. Per-span duplication encodes a structural invariant -- every span in a chunk + shares the same value -- as an arithmetic coincidence that any single-span attribute + mutation silently breaks, inflates the wire by a factor proportional to chunk size, + and forces Datadog egress to recover the chunk grouping by attribute comparison rather + than by container traversal. Promotion to `Resource.attributes` is not a workaround: a + Datadog `TracerPayload` may contain multiple chunks against the same resource, so the + resource grouping does not coincide with the chunk grouping. The proposed + `TraceEvent.datadog.chunk` field reflects the structural fact directly when present; the + encoding is one slot per chunk-scoped value rather than + `N spans × one entry per chunk-scoped value`. +- **Datadog-native span fields lose typed access.** + `Span.datadog.resource_name` and `Span.datadog.span_type` are core inputs to Datadog + routing and APM stats aggregation. Encoding them as internal reserved attributes is + mechanically lossless but forces every Datadog-aware transform, sink, and VRL program + to read them as string-keyed attribute lookups rather than typed accessors. The same + loss applies to `SamplingPriority`: typed as an enum with an `Other(i32)` escape hatch + in the proposal, it degrades to a string-encoded integer under the alternative, + surrendering both the well-known-values ergonomic and construction-time validation. +- **Reserved-key partitioning becomes an internal cost.** The proposal uses reserved + `datadog.*` attributes only at the OTLP wire boundary and lifts them into typed contexts + on ingress. A pure-OTLP internal design retains that encoding in every event, so every transform + walking `Span.attributes` must partition the map into user attributes and Datadog + wire-state encoding to avoid mishandling either, and every sink must do the same on + egress. The proposal's typed fields make the partition once at the type level. +- **The round-trip guarantee weakens from structural to conventional.** The proposal's + `Datadog -> Vector -> Datadog` guarantee rests on structural identity: + every trace/service partition carries its `TraceEvent.datadog.chunk = Some(...)` + context directly, and egress re-coalesces partitions with equal context and + `TraceEvent.trace_id`. Under the alternative, the guarantee rests on every transform respecting + the reserved-key convention; any transform that drops `datadog.chunk.priority` from + a span's attributes silently loses the chunk's sampling priority on egress. Today's + `TraceEvent(LogEvent)` exhibits the same convention-dependent failure mode and is + part of why this RFC exists. + +The proposal already adopts OTLP as the primary shape: `TraceEvent.trace_id` promotes +the wire span grouping key, `Resource`, `Scope`, and the remaining `Span` fields follow +OTLP, semantic conventions name the typed resource fields, and attribute keys follow +OpenTelemetry naming. The Datadog mapping is expressed as projections onto that primary +shape. The explicit `TraceEvent.datadog` and `Span.datadog` namespaces are the minimal +extension that keeps Datadog-native concepts typed and chunk-scoped state structurally +distinct from per-span state without presenting them as common semantics. The pure-OTLP +alternative trades that extension for attribute conventions on every internal consumer. + +### One wire grouping per event + +Map each OTLP `ScopeSpans` or Datadog `TraceChunk` directly to one `TraceEvent`. This +minimizes copies of resource, scope, chunk, and metadata state and makes source ingest +and sink egress nearly mechanical. + +Rejected because OTLP permits one `ScopeSpans` to contain spans from several trace IDs, +and malformed Datadog input can do the same. A filter or sampler operating on such an +event would receive unrelated traces as one topology unit. Stable partitioning by +`trace_id` provides one invariant across both formats and stores that identity directly +on every `TraceEvent`. The cost is duplicated event-level context when one wire grouping +contains several traces; a complete distributed trace can still span several events +across resource, scope, service, or chunk boundaries. + +### One span per event (`TraceEvent { trace_id, span, metadata }`) + +Carry a single span per event. This shape offers two ergonomic advantages: the internal memory usage +of a single span (with the resources shared) is more consistent and granular, and per-span +operations (filter, sample, mutate one span) work directly without iteration. The container shape +does not provide that directly; a future topology-level shim could. The single-span shape, however, +requires the `Resource`, `Scope`, and `DatadogEventContext` to either be duplicated for each span or shared +via `Arc`. + +Rejected because Vector's disk buffers serialize each event as one record: `Arc` sharing collapses +on serialization, every span on disk gets a full inline copy of resource/scope/chunk, and on read +every span gets an independent allocation, thus costing Vector both extra costs in serialization and +deserialization as well as the associated memory expansion and sink-level reassembly mechanics. The +container shape limits the inflation by sharing context across all same-trace spans in one +resource/scope or chunk/service partition. Partitioning by trace ID duplicates context when a wire +group contains several traces, but avoids duplicating it once per span. + +### Parallel `Event` variants for new and old trace formats + +Introduce `Event::NewTrace` alongside `Event::Trace`, leaving the existing `TraceEvent` +untouched. Rejected because it splits trace handling across two `Event` variants for the +duration of the migration, forcing every topology-level dispatch site to handle both. +The tagged-inner approach contains the duality inside `TraceEvent`, leaving +`Event::Trace` as the single dispatch arm. + +### Discriminated union (`Trace::{Otel, Datadog}` or `Span::{Otel, Datadog}) + +Carry each format as-is and dispatch at every consumer. Rejected because it directly +inverts the stated pain -- every transform and every cross-format sink would handle two +shapes with the possibility of more later. This is effectively the status quo over +`LogEvent` just with predefined fields. + +### Single merged `attributes` map with richer typed fields + +Promote additional concepts (service, env, host *and* all semantic-convention +equivalents) to typed fields. Rejected because the semantic convention space is large +and evolving; fixing it in typed fields either forces Vector to track upstream releases +or ossifies a stale subset. The proposal types only the three resource fields both +formats agree on; the rest stay in source-native attributes where users already expect +them. + +### Reusing VRL `Value` for attribute storage + +Store attribute leaves directly in VRL's `Value`, reusing the existing accessor API and disk-buffer +encoding. The trace surface gains no new types and trace VRL programs share their type system with +`LogEvent` and `Metric`. Rejected because `Value::Bytes` collapses the wire string-versus-bytes +discriminator onto a single variant and `Value::Float` (`NotNan`) cannot carry NaN. A number of +round-trip exclusions (OTLP `bytes_value`-as-UTF-8, OTLP `double_value = NaN`, and Datadog `metrics` +NaN handling) therefore become unavoidable. The proposal's `AttrValue` preserves both axes +structurally and limits the conversion cost to the VRL boundary. The wider fix -- a +`Value::String` variant and a NaN-admitting float carrier in `Value` itself -- is in scope for VRL +rather than the trace data model and is recorded under Future Improvements. + +### Timing as `start_time` + `end_time` (OTLP-native) + +OTLP stores `start_time_unix_nano` and `end_time_unix_nano` as two independent +`fixed64`s. Datadog stores `start` plus `duration`. The driving factor for adopting +`start + duration` is that `duration` is more useful than `end_time` in realistic +transforms (filtering slow spans, computing percentiles, classifying long-running +requests), so the chosen representation also matches transform access patterns. + +### Duration as `f64` seconds + +Storing `duration` as `f64` was considered for VRL ergonomics. Rejected because both +OTLP (`fixed64` nanoseconds) and Datadog (`int64` nanoseconds) carry duration as integer +nanoseconds, while a floating-point seconds view progressively loses nanosecond +precision as duration magnitude grows. Storing `duration` as `std::time::Duration` +preserves the wire domain for all non-negative values. Datadog's `int64` wire field +permits negative values that `std::time::Duration` cannot represent; these are clamped +to zero on ingress and declared in the Datadog mapping sub-RFC. + +### `SpanStatus` as a closed enum + +Defining `SpanStatus` without an escape hatch would silently coerce any unrecognized +status code introduced by a future OpenTelemetry version to `Unset` (the proto3 +default), breaking the `OTLP -> Vector -> OTLP` relay guarantee for those spans. The +`Other(i32, String)` variant stores the raw code and message verbatim and egresses them +unchanged, preserving relay fidelity by the same mechanism used for `SpanKind`. The +Datadog egress path has no status-code wire field; `Other` values follow the +`Span.error = 1` rule. + +Among the known status variants, only `Error` carries a string because the OpenTelemetry +trace specification's +[Set Status](https://opentelemetry.io/docs/specs/otel/trace/api/#set-status) rule states +"Description MUST only be used with the Error StatusCode value." A wire `Status.message` +paired with `code = UNSET` or `OK` is non-conformant and is dropped on ingest. `Other` +carries its raw message solely as the forward-compatibility escape hatch described +above. + +### Parsed `TraceState` + +Storing `TraceState` as `IndexMap` would let transforms operate +on entries without an accessor layer. Rejected because every source and sink would have +to invoke the parser/serializer even for pure-relay pipelines, and because the W3C- +imposed bounds (32 entries, 512 bytes total) and typical real-world headers (a single +short entry) mean per-entry allocation costs more than re-parsing the raw header per +accessor call. + +### Wholesale migration + +Replace `TraceEvent(LogEvent)` with the typed container in one PR. Rejected because the +resulting PR would touch every trace source, every trace sink, the APM stats +aggregator, every trace-aware transform, and a large body of tests simultaneously. The +chosen temporary `TraceEventCompat` coexistence design lets each component migrate in +its own PR while keeping the permanent `TraceEvent` typed, subject to a partial-order +constraint that consumers migrate before producers (see "Plan Of Attack"). + +### Feature-flagged switch + +Gate the new representation behind a Cargo feature or runtime flag until all components +are migrated, then flip the default. Rejected because feature combinations proliferate +quickly across every trace source/sink and VRL, and because a runtime flag would +require duplicate code paths in performance-sensitive components. + +### Automatic conversion on typed VRL path access + +Detect a typed path such as `.resource` or `.spans` and convert +`TraceEventCompat::Legacy` in place on its first access. Rejected primarily because a +read should not silently and permanently change the representation emitted downstream. +This violates the principle of least surprise, and the conversion point would depend on +per-event control flow: a path in an untaken branch would leave one event legacy while +the same program could convert another event after earlier statements had already run. + +Typed and legacy paths also are not disjoint. In particular, the existing Datadog +layout already exposes `.spans`, so path spelling cannot distinguish an old program +from an intentional request for the typed surface. VRL's immutable target-read operation +cannot perform an in-place conversion, and its external target type is fixed when the +program is compiled rather than when a path executes. Performing conversion during a +read-only condition would additionally mutate the event merely to inspect it. Selecting +the representation in `remap` configuration and converting before execution avoids +path ambiguity, control-flow-dependent output, partial execution before conversion +failure, and disagreement between compile-time and runtime target shapes. + +### Wire serialization shape + +The chosen design is selected against two imperatives: incompatibility with older +Vector instances must surface loudly (not as silent data drops), and the post-migration +wire schema should carry no vestiges of the migration. Encoding mixed in-memory arrays +as one or more homogeneous wire arrays, while preserving event order, is the cost paid +for both. Each rejected alternative fails at least one: + +- **Extend `Trace` with a typed-fields field**, discriminator by field-presence. Fails + loud-incompatibility: an older Vector ignores the unknown field and decodes the rest + as a legacy event with empty `fields`, silently corrupting the receiver's view of the + batch. + +- **Per-element oneof inside a `MixedTraceArray`**, each array element internally + discriminating between `LegacyTrace` and `TypedTrace`. Fails post-conversion-vestige: + the end-state oneof has a single remaining variant once `LegacyTrace` is retired, and + flattening it to a plain `repeated TypedTrace traces = 1` requires a second wire- + format migration. + +- **Two repeated fields inside a single `TraceArray`**, encoder always 1:1. Fails + loud-incompatibility: an older Vector silently drops the unknown second field, + decoding a typed-only message as an empty `TraceArray`. Loud failure on older Vector + requires the discriminator to live at the oneof level, where the older Vector + recognizes "unknown variant"; a sibling field at a known message level is invisible + to it. + +## Outstanding Questions + +- **Physical representation of optional Datadog state.** The optional + `DatadogAgentEnvelope` and `DatadogChunkContext` values reserve their full inline size + even when absent. Should either use indirection after measuring representative event + sizes and allocation behavior? Sparsely populated maps may likewise benefit from a + compact optional carrier where empty and absent are semantically equivalent. + Any sparse representation must preserve the always-present logical Datadog + contexts, optional child distinctions, mutation semantics, and internal-wire contract. + +## Plan Of Attack + +This Plan of Attack covers the format-agnostic data-model and migration work owned by +this RFC. Per-format shims, encoders, and source and sink flips are sequenced in the +[OTLP mapping](2026-04-29-25329-trace-data-model/otlp-mapping.md) and +[Datadog mapping](2026-04-29-25329-trace-data-model/datadog-mapping.md) sub-RFCs. +Format-agnostic prerequisites land first, followed by sink-local validation and then +system-wide coexistence. Consumer and VRL work and the compile-time gate follow; source +flips are independent after that gate; cleanup is last. Once coexistence begins, +converting consumers must accept +`TraceEventCompat` and fallibly convert it to `TraceEvent` before any source emits typed +events, because shims are unidirectional. Pass-through transforms must accept both +variants without converting. + +The work is organized into seven stages. + +1. **Establish the migration boundary.** Make internal proto decoding fallible so an + unknown oneof variant is reported and drops only the affected message rather than + panicking the task. In the same release line, add the + `vector.trace_legacy_layout` precursor to both trace sources. This release must both + tolerate future event variants and identify every newly written legacy trace layout, + and must precede every `TypedTrace` producer. +2. **Validate the canonical model and mappings behind sink buffers.** Add the supporting + types and the permanent typed `TraceEvent` definition under an internal module path, + without replacing the current public `TraceEvent(LogEvent)` payload. Implement both + format mappings as shared conversion and encoding routines, including format-shape + detection for historical pre-hint records. Migrate each trace protocol sink at its + intake boundary: after reading the existing `EventArray` from its configured buffer, + convert the untyped trace into zero or more canonical `TraceEvent`s and use only that + typed model for partitioning, APM stats aggregation, and wire encoding. Generic + representation-preserving sinks do not convert. Topology channels, disk buffers, + `vector` source/sink hops, VRL, and JSON encoding continue to carry and expose only + the legacy representation. Use this stage to validate the canonical model and both + mappings against their round-trip, cross-format, metadata, finalizer, and stable-order + contracts before making either representation a system-wide alternative. +3. **Introduce coexistence without changing topology behavior.** Promote the typed model + to its permanent public `TraceEvent` export, rename today's untyped representation to + `LegacyTraceEvent`, and make the temporary + `TraceEventCompat::{Legacy, Typed}` enum the payload of `Event::Trace` and + `TraceArray` while all components continue to use `Legacy`. Adapt existing producers + mechanically to wrap their unchanged output, then add the typed internal-wire + variant. Before any source emits typed events, legacy behavior must remain + unchanged, including untagged `json` / `native_json` encoding of `Legacy` as today's + inner map. Both compatibility variants must survive disk-buffer and `vector` + source/sink boundaries, and optional-value and Datadog child-presence distinctions must + survive those boundaries. +4. **Integrate the validated mappings with coexistence.** Register the shared format + converters and format-shape detectors from stage 2 as the two + `LegacyTraceEvent -> TraceEvent` shims, add the typed-shape detector for + `native_json` input, and adapt the already-migrated protocol sinks to accept + `TraceEventCompat` at every intake path. A `Typed` input passes through directly; a + `Legacy` input uses the same conversion validated behind the sink buffer in stage 2. + Remove the temporary direct conversion from the old public trace payload. Do not + expose the typed `remap` mode until both legacy layouts can be converted explicitly. +5. **Establish typed VRL and migrate remaining consumers.** Implement the trace-only + `remap.trace_representation = "typed"` mode. The transform converts legacy input + before constructing `VrlTarget` and runs once per resulting typed event; paths never + convert. The VRL encoding of typed fields is out of scope for this RFC. + Then migrate `sample` and `trace_to_log`; `trace_to_log` converts at intake + and uses only `TraceEvent`. Protocol sinks already satisfy this contract from stage 4. + `json` and `native_json` encoding remain untagged and representation-preserving. + Pass-through transforms accept both variants without converting `Legacy` to `Typed`. + `sample` remains per-event, + so typed input receives one decision per trace-homogeneous resource/scope or + chunk/service partition. This does not make sampling trace-stable across every event + carrying the same `TraceEvent.trace_id`; + `trace_to_log` emits a uniform source-independent layout. + Before the next stage, every migrated consumer must accept + both legacy and typed input and the remap opt-in and round-trip contracts must hold. +6. **Use the compile-time gate, publish migration guidance, and flip producers.** Remove + untyped forwarding methods from `TraceEventCompat` and migrate every resulting Rust + call site. Publish the VRL migration guide, then leave the typed mode available for + the announced migration interval. The OTLP and Datadog sources may begin emitting + `TraceEventCompat::Typed` independently after that interval, once + no built-in trace-aware consumer depends on the legacy key layout and both source + formats have typed production paths. Each typed + source retains its `vector.trace_legacy_layout` hint through the coexistence and + deprecated-proto window. Across `vector` network hops, each downstream receiver must + support `TypedTrace` before its upstream sender emits typed trace records. +7. **Retire coexistence.** After both source flips, remove the compatibility enum and + `LegacyTraceEvent`, replacing `TraceEventCompat` with an alias to `TraceEvent`. Change + the `remap` default to `typed`, reject an explicitly configured `legacy` mode, and + retain the option through a deprecation window. Legacy proto decoding now converts + directly to `TraceEvent`. Retain those proto decoders, shape detectors, and + typed-source hints until the wire retirement criteria above hold; then reserve the + legacy tags and stop emitting the migration hint. + +## Future Improvements + +- Explicit trace-conversion transform: a `convert_trace` transform could provide a + graph-visible boundary before transforms with read-only VRL conditions, avoiding a + no-op typed `remap`. +- Topology-level per-span shim: a transform mode that fans out a `TraceEvent` into per- + span events, runs a downstream transform once per span, and collapses results back + into the container. Lets single-span transforms be authored without explicit iteration + while keeping the trace-homogeneous partition as the source of truth. +- VRL helpers for trace-state parsing/encoding: `parse_trace_state`, + `encode_trace_state`, `merge_span_attributes`. Format-specific decode helpers + (`decode_otlp_span`, `decode_datadog_span`) are listed in the respective mapping sub- + RFCs. +- Link-based routing: a trace-aware router transform that emits to different sinks + based on `SpanLink` targets. +- Stateful trace-aggregator transforms: tail-based sampling, per-trace APM-stats + aggregation, and similar trace-scoped operations that gather trace-homogeneous event + fragments by `TraceEvent.trace_id`. +- Trace- or chunk-stable sampling: an intentional sampling guarantee that makes a + single keep/drop decision per `TraceEvent.trace_id` (or per chunk identifier) and applies it + consistently to every event derived from that trace/chunk. Today's `sample` transform + has no such guarantee; it operates per-event, and any per-chunk atomicity is + incidental to the pre-migration `LogEvent`-per-chunk layout. This may be added to + `sample` as a configurable mode or shipped as a separate component (e.g. + `trace_sample`); either approach remains a future design decision. +- Distinct `Value::String` variant in VRL's `Value`, plus a NaN-admitting float carrier. + With `AttrValue` in place the trace round-trip no longer drives this work. The remaining + motivation is `LogEvent` / `Metric` UTF-8 handling and letting the trace VRL boundary + surface NaN as a typed `Float` rather than `Null`. The change is cross-cutting: every + `Value` consumer is affected, and admitting NaN requires a `Value` equality/ordering + redesign. diff --git a/rfcs/2026-04-29-25329-trace-data-model/datadog-mapping.md b/rfcs/2026-04-29-25329-trace-data-model/datadog-mapping.md new file mode 100644 index 0000000000000..632f0e8a6d67c --- /dev/null +++ b/rfcs/2026-04-29-25329-trace-data-model/datadog-mapping.md @@ -0,0 +1,904 @@ +# RFC 25329 - 2026-04-29 - Trace Data Model: Datadog Mapping + +This sub-RFC of [RFC 25329 -- Internal Trace Data Model](../2026-04-29-25329-trace-data-model.md) +specifies the bidirectional mapping between the typed `TraceEvent` defined in the parent RFC +and the Datadog agent-to-backend trace protobuf. It establishes the Datadog ingress and +egress paths, the effective-equivalence round-trip guarantee for +`Datadog -> Vector -> Datadog`, the trace-ID-and-service chunk split/coalesce rules, the +reserved Datadog event and span contexts that carry agent-payload, tracer-payload, chunk, and +`meta_struct` state, and +the cross-format conformance rule for `OTLP -> Vector -> datadog_traces`. + +## Context + +- The parent RFC defines the typed data model, migration mechanics, and internal wire + serialization. This sub-RFC assumes those definitions and the parent's Glossary, In/Out + scope clauses, and User Experience as background. The Datadog-specific wire-format and + cross-format references this sub-RFC depends on are defined in the Glossary below; OTLP + and other shared vocabulary is defined in the parent's Glossary. +- [RFC 9572 -- Accept Datadog traces](../2021-10-15-9572-accept-datadog-traces.md) introduced + the `datadog_agent` trace ingest path, which the `datadog_traces` sink can consume but + which today has no well-defined internal representation. This sub-RFC, together with the + parent and the OTLP mapping sub-RFC, supplies that representation. + +## Glossary + +This sub-RFC defines the Datadog-specific format vocabulary. OTLP, OpenTelemetry, W3C +Trace Context, and other informational entries are defined in the +[parent RFC's Glossary](../2026-04-29-25329-trace-data-model.md#glossary). + +- **Datadog APM trace format**: Vector targets exactly one hop in the Datadog tracing + pipeline -- the agent-to-backend protobuf served at `/api/v0.2/traces`. When this + sub-RFC says "Datadog" unqualified, it means that format. The schema lives in three + protobuf files in the Datadog Agent repository: + - [`agent_payload.proto`](https://github.com/DataDog/datadog-agent/blob/main/pkg/proto/datadog/trace/agent_payload.proto) + -- `AgentPayload` (`tracerPayloads[]`, agent-level `tags`, `agentVersion`, + `targetTPS`, `errorTPS`). + - [`tracer_payload.proto`](https://github.com/DataDog/datadog-agent/blob/main/pkg/proto/datadog/trace/tracer_payload.proto) + -- `TracerPayload` (`chunks[]`, tracer-level fields) and `TraceChunk` + (`priority` / `origin` / `droppedTrace` / `tags`, `spans[]`). + - [`span.proto`](https://github.com/DataDog/datadog-agent/blob/main/pkg/proto/datadog/trace/span.proto) + -- the per-span shape (`service`, `name`, `resource`, `traceID`, `spanID`, + `parentID`, `start`, `duration`, `error`, `meta`, `metrics`, `type`, `meta_struct`, + `spanLinks`, `spanEvents`). +- **Datadog Agent OTLP ingest**: the OTLP-to-Datadog conversion implemented by the + Datadog Agent in + [`pkg/trace/api/otlp.go`](https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/api/otlp.go) + (and supporting code under + [`pkg/trace/transform/`](https://github.com/DataDog/datadog-agent/tree/main/pkg/trace/transform)). + Current `main` is the normative reference for the enumerated Agent-aligned + derivations below and governs their details if a prose summary lags an upstream + change. The RFC's explicit mapping remains normative for all other fields; exact + reproduction of the Agent's complete OTLP converter output is not required. +- **Datadog tracer-to-agent API** (informational): the SDK-to-agent hop upstream of the + agent-to-backend format Vector targets. Vector does not consume these endpoints + directly. + +## Cross cutting concerns + +- APM stats aggregation in the `datadog_traces` sink, today reading magic keys from + `TraceEvent`, will read typed fields after this RFC and its parent land. Batch-scoped + re-coalescence before aggregation is specified under grouping. +- The OTLP-side reservation of `datadog.*` resource and span attributes for synthesis + of Datadog-native context on OTLP egress is the OTLP mapping sub-RFC's concern; this + sub-RFC owns the context values those keys carry on cross-format relay. + +## Scope + +### In scope + +- The bidirectional mapping between `TraceEvent` and Datadog `AgentPayload` / + `TracerPayload` / `TraceChunk` / `Span` messages. +- The parent RFC's effective-equivalence round-trip guarantee, applied to + `Datadog -> Vector -> Datadog`. Stable partitioning preserves span order within each + `(trace_id, service)` group. The relative order of spans from distinct services in + one trace, and for a non-conforming multi-trace chunk the relative positions of + spans from distinct trace IDs and their resulting egress chunk grouping, are details + the Datadog backend does not observe and so may differ. +- The three Datadog span-attribute partitions (`meta`, `metrics`, `meta_struct`) and how + the two scalar partitions map into `Span.attributes` by `AttrValue` variant while + `meta_struct` is preserved in `Span.datadog.meta_struct`. +- The agent-payload envelope and tracer-payload tags in + `TraceEvent.datadog.{agent,tracer}`. +- The chunk-scoped typed state in `TraceEvent.datadog.chunk = Some(...)` + (`priority`, `origin`, `dropped`, `tags`). +- The stable partition of each chunk by its wire spans' reconstructed trace ID and + `Span.service` on ingress, storing the ID in `TraceEvent.trace_id`, and the + corresponding re-coalescence by trace ID on egress. +- The authoritative reconstruction policy for explicitly present Datadog namespace + state and the common-field fallback when that state is absent. +- The cross-format conformance rule for `OTLP -> Vector -> datadog_traces`: a + trace whose Datadog namespace is absent/default follows the enumerated Datadog-Agent + derivations below and is backend-effectively equivalent under Vector's typed mapping. + It need not reproduce the Agent converter's exact wire shape. + +### Out of scope + +- The OTLP wire mapping (see the OTLP mapping sub-RFC). +- Zero-loss cross-format round-trip (`Datadog -> OTLP -> Datadog`); see the parent RFC's + Out of scope. +- `TracerPayload.containerDebug` (Datadog-internal container-tag-resolution diagnostic); + dropped on ingest, not synthesized on egress. +- Typed decoding of `AgentPayload.idxTracerPayloads = 11` (the indexed/deduplicated + tracer-payload form). Until support lands, standard `tracerPayloads` entries are + processed while indexed entries are discarded; see "Ingress and egress + mapping." +- Typed decoding of the pre-`tracerPayloads` `/api/v0.2/traces` shape + (`traces` / `transactions` with empty `tracerPayloads`). The typed mapping discards + those spans and reports the drop. Removing that decode path may land independently + of the typed migration. +- Exact wire parity with the Datadog Agent's complete OTLP converter, including its + legacy metadata encodings and configuration-dependent synthetic tags. + +### Zero-loss round-trip exclusions + +The effective-equivalence guarantee for `Datadog -> Vector -> Datadog` does not cover the +following input shapes. Each is justified by a paragraph in the Implementation or +Rationale section below. + +- **`Span.error` values other than `0` or `1`** ingest as `SpanStatus::Error(...)` and + egress as `Span.error = 1`, normalizing the specific integer to the conforming + bivalent representation. +- **Spans that omit `_dd.p.tid` when siblings reconstruct to more than one 128-bit + `trace_id` from the same wire `Span.traceID`** are dropped as malformed identity. + Explicitly tagged spans keep their reconstructed ID. +- **Datadog wire-domain normalization**: negative wire durations are clamped to zero on + ingress; durations and timestamps outside their destination fields' domains are + clamped to the nearest endpoint on egress. +- **`meta`/`metrics` producer-side non-disjointness**: the round-trip guarantee is conditional on + these two scalar maps being keyset-disjoint. If a producer emits the same key in both, the Datadog + source resolves the collision deterministically (`metrics` wins) and + saturating-increments `Span.dropped_attributes_count`; the dropped scalar + is not recoverable on egress. +- **Unknown `AttributeAnyValue` or array-element type discriminators** drop the + containing span-event attribute and saturating-increment + `SpanEvent.dropped_attributes_count`. +The Datadog-side consequences of the parent RFC's malformed-input rule, zero-ID +rejection, and wire-domain normalization also apply. + +## Pain + +- Today's `datadog_agent` source produces an untyped `TraceEvent(LogEvent)` whose key + layout encodes the wire field locations directly. The `datadog_traces` sink reads these + magic keys, and the APM stats aggregator does the same. Transforms written against this + layout are tightly coupled to source-side decisions and break under cross-format relay. +- The wire `Span.traceID` is 64 bits but Datadog's 128-bit traces extend the high half via + `meta["_dd.p.tid"]`. The current sink coerces `trace_id as i64` + ([#14687](https://github.com/vectordotdev/vector/issues/14687)), corrupting precision + for non-i64-representable values; a typed `TraceId(NonZeroU128)` (parent RFC) + eliminates the coercion by construction. +- Datadog chunk-scoped state (`priority`, `origin`, `droppedTrace`, `tags`) applies + uniformly to every span in a chunk, but the current `LogEvent`-per-chunk shape forces + the sink to recover this from per-span attribute keys, encoding a structural invariant + as a positional convention. + +## Proposal + +### User Experience + +The Datadog wire mapping is a projection onto the typed `TraceEvent` defined in the +parent RFC. The Datadog-specific fields are the `TraceEvent.datadog` and `Span.datadog` +namespaces. The event namespace carries the optional agent envelope, tracer tags, and +optional chunk; the span namespace carries `resource_name`, `span_type`, and +`meta_struct`. This sub-RFC specifies how those typed fields map to and from the Datadog +wire format. + +### Implementation + +#### Ingress and egress mapping + +Until indexed decoding is implemented, the source discards every non-empty +`idxTracerPayloads` entry. Standard `tracerPayloads` in the same `AgentPayload` continue +through the mapping below. An indexed-only payload consequently produces zero events and +is nonetheless acknowledged, avoiding retries that cannot succeed on the same Vector +version. + +After indexed entries are handled as above, a modern `AgentPayload` whose +`tracerPayloads` repeated field is empty and whose historical `traces` and +`transactions` fields are also empty, or a `TracerPayload` whose `chunks` repeated +field is empty, produces zero `TraceEvent`s: there is no +`TraceChunk` from which to populate `TraceEvent.datadog.chunk` and no event grouping is +well-defined in isolation, so the wire input has no `TraceEvent` representation. That +standard-empty case is lossless because it carries no span data the Datadog backend +would observe. + +A payload whose `tracerPayloads` is empty but whose `traces` or `transactions` fields +carry spans is the pre-`tracerPayloads` shape. The typed mapping does not ingest that +shape: those spans are discarded and the request is acknowledged so it is not retried. +This is not a lossless empty payload. Independent removal of that path must still report +an empty-`tracerPayloads` payload. + +An `AgentPayload` with at least one `TracerPayload` carrying at least one `TraceChunk` +with at least one span expands into one `TraceEvent` per distinct +`(reconstructed wire trace ID, Span.service)` pair within each +`(TracerPayload, TraceChunk)`. + +The grouping rules are: + +- Before partitioning, reconstruct each span's 128-bit ID from `Span.traceID` and + `_dd.p.tid` (absent or empty means high bits 0). If siblings that share a wire + `Span.traceID` reconstruct to exactly one distinct ID among those that supplied a + tag, untagged siblings receive that ID. If they reconstruct to more than one, keep + the tagged spans and drop untagged siblings as malformed identity. +- Partition each `TraceChunk` into one `TraceEvent` per distinct `(trace_id, service)` + pair in first-seen pair order, storing the pair's ID in `TraceEvent.trace_id` and + preserving span order within each group. A conforming single-trace, single-service + chunk remains one event; a multi-service or non-conforming multi-trace chunk splits + as needed. Egress re-coalesces service groups for the same trace (see below). A + `TraceChunk` whose `spans` repeated field is empty produces zero `TraceEvent`s. +- The enclosing `TracerPayload`'s metadata (`hostname`, `env`, `containerID`, + `languageName`, `tracerVersion`, etc.) populates the event's `Resource`. Per-span + `Span.service` populates `Resource.service`. +- The enclosing `AgentPayload`'s envelope (`hostName`, `env`, `agentVersion`, `targetTPS`, + `errorTPS`, `rareSamplerEnabled`, and `tags`) populates + `TraceEvent.datadog.agent = Some(...)`; + `TracerPayload.tags` populates `TraceEvent.datadog.tracer.tags` (see "Datadog + event-scoped state" below). +- `TraceChunk.{priority, origin, droppedTrace, tags}` populate + `TraceEvent.datadog.chunk = Some(DatadogChunkContext { ... })`, including when every + field is default. + Every decoded `priority` value is present internally: `0` maps to + `Some(SamplingPriority::AutoReject)` and every other `i32` maps through the parent + RFC's canonical constructor. +- `Scope` is left default; Datadog has no scope concept. + +The `datadog_agent` legacy shim applies the same grouping. Pre-flip source output is +one event per `TraceChunk`, so existing legacy VRL is unchanged. The shim splits +that chunk into the same typed events native ingest would have produced. The current +layout merges or omits agent, tracer, and chunk distinctions; those slots are not +recovered after a buffer. +Metadata, finalizers, and acknowledgements on the resulting sequence follow the +parent RFC's conversion contract. An empty-spans +legacy chunk converts to zero typed events, matching native ingest. + +The `Datadog -> Vector -> Datadog` effective-equivalence guarantee is a mapping from +decoded `AgentPayload` through the typed model and back. It is not a claim that the +default pre-flip source layout is lossless. Pipeline relay of the scopes that layout +merges or omits waits until the source emits typed events. + +| Datadog | Internal | +| ------------------------------------------------------------- | ----------------------------------------------------- | +| `TracerPayload.hostname` | `Resource.host` | +| `TracerPayload.env` | `Resource.environment` | +| `Span.service` (per span) | `Resource.service` of the event holding the span | +| `AgentPayload` envelope (whole message; see below) | `TraceEvent.datadog.agent = Some(...)` | +| `TracerPayload.tags` | `TraceEvent.datadog.tracer.tags` | +| `TraceChunk.{priority, origin, droppedTrace, tags}` | `TraceEvent.datadog.chunk = Some(...)` | +| `TracerPayload` non-host/env scalar fields (see below) | `Resource.attributes` under defined keys | +| `Span.traceID` (u64) | `TraceEvent.trace_id.low_u64` | +| `Span.meta["_dd.p.tid"]` (hex u64) if present (see below) | `TraceEvent.trace_id.high_u64` | +| `Span.spanID`, `Span.parentID` | `Span.span_id`, `Span.parent_span_id` | +| `Span.name` | `Span.name` | +| `Span.resource` | `Span.datadog.resource_name` | +| `Span.type` | `Span.datadog.span_type` | +| `Span.start`, `Span.duration` | `Span.start_time`, `Span.duration` (ns-exact) | +| `Span.error` and `Span.meta["error.message"]` | `Span.status` (see below) | +| `Span.meta` | `Span.attributes` (`AttrValue::String`, see below) | +| `Span.metrics` | `Span.attributes` (`AttrValue::Double`) | +| `Span.meta_struct` | `Span.datadog.meta_struct` (`Map`) | +| `Span.spanEvents[*].{time_unix_nano, name}` | `SpanEvent.{time, name}` | +| `Span.spanEvents[*].attributes` (`AttributeAnyValue`) | `SpanEvent.attributes` (typed `AttrValue` per variant)| +| `Span.spanLinks[*].traceID` (u64) | `SpanLink.trace_id.low_u64` in `Span.links` | +| `Span.spanLinks[*].traceID_high` (u64) | `SpanLink.trace_id.high_u64` | +| `Span.spanLinks[*].spanID` | `SpanLink.span_id` | +| `Span.spanLinks[*].tracestate` | `SpanLink.trace_state` (verbatim) | +| `Span.spanLinks[*].flags` (u32) | `SpanLink.flags` (full u32 verbatim) | +| `Span.spanLinks[*].attributes` | `SpanLink.attributes` (`AttrValue::String`) | + +The cross-format derivation rules later in this section (`datadog.span_type` from +`Span.kind` / `Span.attributes`, `datadog.resource_name` from `Span.attributes` / `Span.name`, the +`TracerPayload` semantic-convention key set, and the flattening of unmapped +`Resource.attributes` into per-span `meta`) are projections of the Datadog Agent OTLP +ingest reference in the Glossary. The precise key sets and lookup orders track current +upstream behavior without requiring a specification amendment. + +#### `TracerPayload` semantic-convention key mapping + +`TracerPayload` fields mapped to `Resource.attributes` under OpenTelemetry +semantic-convention keys: `containerID`, `languageName`, `languageVersion`, +`tracerVersion`, `runtimeID`, `appVersion`. The specific OpenTelemetry attribute key each +wire field maps to is defined by the Datadog Agent OTLP-ingest reference, which the +Vector mapping mirrors. The `TracerPayload`-envelope-equivalence rule in the egress +section consequently keys off the attribute set defined by the current reference. + +`TracerPayload.hostname` and `TracerPayload.env` map to typed +`Resource.host` / `Resource.environment` directly and are not part of the deferred +attribute set. `TracerPayload.containerDebug` is a Datadog-internal diagnostic with no +Vector consumer and is dropped on ingest (see "Out of scope"). + +#### Empty Datadog string fields + +Datadog's string scalars do not expose proto3 presence, so decode cannot distinguish an +omitted field from one explicitly encoded as empty. The mapping nevertheless preserves +the decoded empty value as `Some("")` for `Resource.{service,environment,host}`, +`Span.datadog.{resource_name,span_type}`, and `DatadogChunkContext.origin`; `None` means +no source value was available. Datadog egress therefore reproduces the empty value, and +the `resource_name` / `span_type` fallback applies only to `None`. + +#### Signed timing fields on ingress + +Datadog's `Span.duration` and `Span.start` wire fields are `int64` and can therefore +carry negative values, which no documented Datadog tracing SDK produces. The two +disagree on ingress disposition, so neither follows from the parent RFC's numeric +boundary policy: + +- A negative `Span.duration` is clamped to zero, because `std::time::Duration` cannot + represent it. +- A negative `Span.start` is preserved as a pre-epoch timestamp, which the typed model + can represent. It remains subject to the parent RFC's pre-epoch clamp if it later + crosses a disk buffer or `vector` hop, so a round trip preserves it only within a + single instance. + +Egress clamping for these fields and for `SpanEvent.time` follows the parent RFC's +policy against the concrete wire domains. The negative-duration clamp is declared as a +Datadog-side round-trip exclusion above. + +#### `Span.error` and `Span.status` + +`Span.error != 0` maps to `Error(meta["error.message"].cloned().unwrap_or_default())`, +else `Unset`. The `error.*` meta entries also flow into `Span.attributes` per the meta +merge rule below, keeping `error.type` / `error.stack` accessible alongside the typed +status. Datadog's wire `Span.error` is `int32`; values other than `0` and `1` are +non-conformant with the field's documented bivalent semantics. Such values ingest as +`SpanStatus::Error(...)` and egress as `Span.error = 1`, normalizing the specific +integer to the conforming bivalent representation; this is declared as a Datadog-side +round-trip exclusion above. + +#### `_dd.p.tid` (128-bit trace-ID high half) + +On ingress, `meta["_dd.p.tid"]` is consumed into `TraceEvent.trace_id` and is not +retained as a span attribute. An absent `_dd.p.tid`, or a key present with an empty +value, is a valid 64-bit ID (high bits 0) and does not drop the span. + +The tag is sink-owned: Datadog egress derives it exclusively from +`TraceEvent.trace_id.high_u64()`. If the high half is non-zero, egress writes +`meta["_dd.p.tid"]` as a zero-padded 16-character lowercase hex string; if zero, the +tag is omitted. A transform-authored `attributes["_dd.p.tid"]` does not win. Removing +such an entry is reported; Datadog has no wire dropped-attribute count to update. + +#### `SpanLink.traceID_high` + +Unlike `Span` itself -- whose proto carries only a 64-bit `traceID` and stores the high +half out-of-band in `meta["_dd.p.tid"]` -- `SpanLink` carries the high 64 bits in a +dedicated wire field, `traceID_high`. Combining `traceID` and `traceID_high` into the +typed 128-bit `SpanLink.trace_id` on ingest, and splitting it back on egress, is required +for the `Datadog -> Vector -> Datadog` round trip to preserve links to 128-bit trace +IDs. A `traceID_high` of zero on the wire is equivalent to absent and yields a +`SpanLink.trace_id` whose high half is zero; on egress, a zero high half is emitted as +field-absent (or zero, which is byte-identical under proto3). The link-target +`_dd.p.tid` is not consulted on either direction: links may reference a different trace +than the enclosing span, and the wire field is the canonical carrier. + +#### Zero-ID detection + +Datadog ingress applies the parent RFC's zero-ID drop rule, including its drop +granularity and reporting, with one format-specific qualification: because Datadog +splits a 128-bit trace ID across two wire fields, the rule is evaluated against the +*combined* ID rather than either field alone. A `Span.traceID` of zero paired with a +non-zero `_dd.p.tid` high half is a valid ID and is not rejected; the same applies to a +`SpanLink` whose `traceID` is zero but whose `traceID_high` is not. + +Datadog `Span.parentID == 0` is a "no parent" sentinel and is not a zero-ID failure: it +maps to `Span.parent_span_id = None` rather than to a zero `SpanId`. On egress, a `None` +parent emits `parentID = 0` to match the agent's convention. + +#### `SpanLink.flags` + +Datadog's `SpanLink.flags` is `uint32`, and the Datadog convention is that bit 31 must +be set whenever the field is meaningful (the proto comment: "If set, the high bit (bit +31) must be set"). Storing the full word in `TraceFlags(u32)` preserves both bit 31 and +the W3C / OTLP-defined low bits so the round trip is bit-exact. + +Datadog `Span` itself has no flags wire field and no trace-state wire field; on +cross-format Datadog egress, OTLP-sourced `Span.flags` and `Span.trace_state` are +dropped (in line with cross-format zero-loss being out of scope in the parent RFC). For +Datadog-sourced events, `Span.flags` and `Span.trace_state` are always their default +values on ingress, so this drop has no effect on a Datadog round trip. The same +constraint applies asymmetrically on the link path: on Datadog egress, `SpanLink.flags` +is emitted verbatim. For OTLP-sourced events bit 31 is not set, so the Datadog backend +treats the field as not meaningful and the W3C trace-flags byte plus the OTLP +`CONTEXT_HAS_IS_REMOTE` / `CONTEXT_IS_REMOTE` tristate carried by the link are not +surfaced through the Datadog wire. The sink does not synthesize bit 31. This is the +link-path analogue of the `Span.flags` cross-format drop and is out of scope for the +cross-format guarantee. + +#### `SpanEvent.attributes` typed value mapping + +Datadog `SpanEvent.attributes` is `map`, where +`AttributeAnyValue` carries an explicit type tag (`STRING_VALUE`, `BOOL_VALUE`, +`INT_VALUE`, `DOUBLE_VALUE`, `ARRAY_VALUE`). This is distinct from the flat +`Span.meta` / `Span.metrics` partitions and maps directly to `AttrValue` variants. + +Datadog's `AttributeAnyValue` has no native `bytes` or `kvlist` form. On Datadog egress, +`AttrValue::Bytes` is stringified to `STRING_VALUE` via `dd_value_to_string` (defined +below) and `AttrValue::Map` is stringified to a JSON `STRING_VALUE`. `AttrValue::Null` +entries are dropped from the wire map (the wire has no representation for "key present, +value absent"), parallel to the `Null` handling on the `meta` / `metrics` +and `SpanLink` egress paths. An `AttrValue::Array` maps scalar elements directly. +Elements without an +array-scalar representation (`Bytes`, nested `Array`, `Map`, or `Null`) become +`STRING_VALUE` using `dd_value_to_string`; `Null` uses the literal `"null"`. + +On ingress, an unknown `AttributeAnyValue.type` or an unknown type on any element of its +`array_value` makes the containing map attribute unrepresentable. The source drops only +that attribute, saturating-increments `SpanEvent.dropped_attributes_count`, and reports +the drop; other attributes and the event continue through the mapping. + +#### Datadog attribute partitions: convention versus invariant + +Datadog spans carry attributes in three independent wire-level maps: + +- `meta`: keys to UTF-8 strings. +- `metrics`: keys to IEEE-754 doubles. +- `meta_struct`: keys to opaque bytes (msgpack-encoded structured payloads). + +Datadog ingress maps the partitions into the common and Datadog span surfaces: + +- `meta` entries become top-level entries with `AttrValue::String`. +- `metrics` entries become top-level entries with `AttrValue::Double`. +- `meta_struct` entries populate `Span.datadog.meta_struct`, which maps each key + directly to its opaque `Bytes` payload. + +The protobuf decoder rejects invalid UTF-8 in `map` keys or values at +the payload boundary, before this attribute mapping runs; no raw-string fallback decoder +is introduced. + +If a producer emits the same key in both `meta` and `metrics`, the Datadog source +resolves the collision deterministically (`metrics` wins), saturating-increments +`Span.dropped_attributes_count`, and reports the drop. A key emitted in `meta_struct` +and either scalar map retains both values because `Span.datadog.meta_struct` and +`Span.attributes` are structurally separate. This includes the literal scalar key +`_dd.meta_struct`; it does not collide with the structured partition. + +Datadog egress, in order: + +1. Copy `Span.datadog.meta_struct` into the wire `meta_struct` map. +2. Partition the remaining attributes by `AttrValue` variant: `String` and `Bytes` to + `meta` (the latter as a UTF-8-lossy string), `Double` and `Int` (coerced to `f64`) + to `metrics`. `Null` is dropped (the wire has no representation for "key present, + value absent"). Variants with no native Datadog partition (`Bool`, + `Array`, `Map`) are stringified into `meta` via `dd_value_to_string`. + +The result is one entry per non-`Null` key in exactly one wire partition. +A sink-owned write evicts that key from every partition first. + +**`dd_value_to_string` rule.** Several Datadog wire fields are `map` and use one shared coercion, +named `dd_value_to_string` throughout this document. Per-variant rendering tracks the +Datadog Agent OTLP converter cited in the Glossary. A top-level `Null` map entry has +no wire representation, so it is omitted rather than coerced. + +#### Datadog event-scoped state + +The `AgentPayload` envelope populates `TraceEvent.datadog.agent`; `None` means that no +agent envelope is present. Its typed scalar fields are `host_name`, `env`, +`agent_version`, `target_tps`, `error_tps`, and `rare_sampler_enabled`, and its `tags` +field carries the wire-level `AgentPayload.tags` map. The double fields preserve NaN +payloads unchanged. + +`TracerPayload.tags` populates `TraceEvent.datadog.tracer.tags`. +`TracerPayload.hostname` and `TracerPayload.env` map to the common +`Resource.host` / `Resource.environment` fields directly. The tracer context is +always present because protobuf cannot distinguish an omitted map from an empty map. + +#### Datadog chunk context + +Datadog `TraceChunk.priority`, `origin`, `droppedTrace`, and `tags` apply uniformly to +every span in the chunk. Every Datadog-sourced `TraceEvent` corresponds to exactly one +chunk by construction and therefore carries `TraceEvent.datadog.chunk = Some(...)`, including +for an explicitly all-default chunk. An OTLP-sourced event with no recovered Datadog +chunk state carries `None`. + +Because proto3 provides no scalar presence for `TraceChunk.priority`, every decoded +wire value maps to `Some(SamplingPriority)`: zero is `Some(AutoReject)`, known non-zero +values use their canonical variants, and other `i32` values use `Other`. `None` is +reserved for events with no source priority, such as an OTLP event without recovered +Datadog chunk state. On Datadog egress, that `None` emits `AutoKeep` (wire 1), matching +today's `datadog_traces` sink and the Datadog Agent OTLP ingest at its default sampling +rate. Wire `0` is reserved for an explicit `AutoReject`. + +#### Datadog egress derivation rules + +When `Span.datadog.span_type` is `None` on Datadog egress (the normal case for OTLP-sourced +spans), the sink derives the wire `Span.type` from `Span.kind` and `Span.attributes`, +following the Datadog Agent's +[`SpanKind2Type`](https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/transform/otelutil.go) +logic. Per the deferral above, the individual kind-to-type cases track that upstream +reference and are not fixed by this document. + +If `Span.datadog.span_type` is `Some(v)`, the value is emitted as-is (Datadog-sourced spans +carry it directly). Because Datadog has no span-kind wire field, `Span.kind` is always +`Unspecified` for Datadog-sourced events on ingress; the `SpanKind2Type` derivation +therefore never fires on a pure `Datadog -> Vector -> Datadog` round trip. + +When `Span.datadog.resource_name` is `None` on Datadog egress (the normal case for OTLP-sourced +spans), the sink derives the wire `Span.resource` from `Span.attributes` following the +Datadog Agent's OTLP ingest reference implementation, falling back to `Span.name` when +no matching attribute is present. If `Span.datadog.resource_name` is `Some(v)`, the value is +emitted as-is. + +On Datadog egress, the sink: + +- Sets each wire `Span.error` from `Span.status`: `Error(_)` or `Other(_, _)` -> `1`; + `Unset` / `Ok` -> `0`. (`Other(code, _)` enforces `code` outside the known set + `{0, 1, 2}` by construction, so every `Other` value represents a non-zero status + code.) Datadog spans whose original wire `Span.error` was not `0` or `1` lose the + specific integer on round trip; see the Datadog-side exclusions above. +- Flattens unmapped `Resource.attributes` entries into each span's wire `meta` under + the attribute key. + - Scope: applies to keys other than the typed-slot promotions (`service.name`, + `deployment.environment.name`, `host.name`) and the TracerPayload-mapped semantic- + convention keys per "`TracerPayload` semantic-convention key mapping" above. + - Tie-breaker: a per-span `Span.attributes` entry at the same key wins over a + `Resource.attributes` entry; the wire format has no resource-attribute scope, so + the per-span duplication is the wire shape's nature, not Vector's choice. Each + discarded resource-scoped value is reported; Datadog has no wire dropped-attribute + count to update. + - For Datadog-sourced events these unmapped keys are empty by construction; native + agent/tracer state lives in `TraceEvent.datadog`, so the round-trip is unaffected. +- Copies `Span.datadog.meta_struct` into the wire `meta_struct` map and partitions + `Span.attributes` into `meta` / `metrics` by `AttrValue` variant per "Datadog + attribute partitions" above. +- Reconstructs each `SpanEvent.attributes` entry as an `AttributeAnyValue` from the + `AttrValue` variant per "`SpanEvent.attributes` typed value mapping" above, not the + `meta` / `metrics` partitioning rule. +- Emits `TraceEvent.trace_id.low_u64()` as every contained wire `Span.traceID`; writes + `meta["_dd.p.tid"]` from `TraceEvent.trace_id.high_u64()` if non-zero, omits it if zero + (see "`_dd.p.tid`" above). +- Resolves the typed slot/attribute-map pair `Span.status` versus + `Span.attributes."error.message"`: + - When `Span.status` is `Error(message)` or `Other(_, message)` and `message` is + non-empty, `meta["error.message"]` is set to the typed message. + - When `Span.status.message` is empty (`Unset`, `Ok`, `Error("")`, or `Other(_, "")`), + the sink does not synthesize a `meta["error.message"]` tag and any value the + attribute partitioning step placed there is left in place. This empty-message guard + preserves the round trip for the conforming input `error = 1, no + meta["error.message"]`, which ingests as `Error("")` with no attribute and must + egress identically. +- Emits `SpanLink.attributes` as the wire `map`: non-`Null` + `AttrValue` variants are stringified via `dd_value_to_string`, and `Null` entries are + omitted. + - The `meta` / `metrics` partitioning rule used for `Span.attributes` does not apply + to links because the Datadog `SpanLink.attributes` wire type is a flat string map, + not the `meta` / `metrics` / `meta_struct` triple. + - For Datadog-sourced events, `SpanLink.attributes` values are already + `AttrValue::String` on ingress due to the wire type, so the `dd_value_to_string` + conversion on egress is lossless for `Datadog -> Vector -> Datadog` round trips. + +Datadog `Span`, `SpanEvent`, and `SpanLink` provide no dropped-attribute count fields. +Consequently, Datadog-egress attribute losses are reported but cannot propagate an +updated in-band count on that wire. + +#### Envelope reconstruction and chunk re-coalescence + +Before grouping, Datadog egress normalizes namespace state into its wire shape. +Agent-envelope scalar fields are already typed. Entries in agent, tracer, and chunk tag +maps use `dd_value_to_string` and omit top-level `Null`. Grouping keys and wire +serialization both use this normalized result, so transform-authored state cannot +produce inconsistent grouping or a sink failure. + +All equality used by the nested Datadog grouping steps is structural. Double values are +compared by their IEEE-754 `to_bits()` representation, so a NaN equals the same preserved +NaN payload and positive and negative zero remain distinct. This applies to normalized +envelope doubles and doubles nested in resource or chunk attributes. + +Datadog egress groups events into wire `AgentPayload` / `TracerPayload` / `TraceChunk` +structures by nested grouping keys. Request partition includes +`EventMetadata.datadog_api_key` outside those keys, including for APM stats, so +fan-in of different tenants cannot share a payload. + +**`AgentPayload` grouping.** Groups events by their effective envelope and emits one +`AgentPayload` per group. `TraceEvent.datadog.agent = Some(...)` is authoritative +regardless of which source or transform populated it. When it is `None`, the effective +envelope is the proto3 defaults. This is the +outermost grouping step, so every downstream `TracerPayload` and `TraceChunk` is by +construction confined to a single `AgentPayload`. + +- Scalar reconstruction: each `AgentPayload`'s `hostName`, `env`, `agentVersion`, + `targetTPS`, `errorTPS`, `rareSamplerEnabled`, and `tags` are read from the matching + fields in a present `datadog.agent`; tags use the normalization above. +- Fallback for an absent agent envelope: events with `datadog.agent = None` + emit proto3 defaults for every `AgentPayload` envelope field, including `hostName` + and `env`. Application identity remains on `TracerPayload` via `Resource.host` / + `Resource.environment`. No `datadog_traces` sink configuration + governs these fields. Two such events therefore share the same default envelope and + land in the same `AgentPayload`. +- Grouping on the full envelope preserves the partitioning Vector applies today, so + two sets of events coming from different agent hosts or envs cannot be coalesced + into the same `AgentPayload` and relayed traffic stays attributed to its originating + agent. + +**Namespace authority.** Datadog egress does not inspect +`EventMetadata.source_type` or `vector.trace_legacy_layout` when reconstructing typed +events. A source, OTLP bridge, or transform that explicitly populates `.datadog` +expresses an intent to control Datadog-native egress state. Ordinary resource and span +attributes never acquire that authority merely because their keys resemble Datadog +internals. Typed namespace state therefore survives disk buffers and intermediate +`vector` source/sink hops without an origin-specific passthrough option. The migration +hint remains solely the temporary legacy-shim selector defined by the parent RFC. + +**`TracerPayload` grouping.** Within each `AgentPayload`, gather events with a +TracerPayload-envelope-equivalent `Resource` into one `TracerPayload`, with each span's +`Span.service` reconstructed from its event's `Resource.service`. + +- Equivalence: two events are TracerPayload-envelope-equivalent when their normalized + `datadog.tracer.tags` are equal and every `Resource` field that maps to a + `TracerPayload` wire field in the ingest table above is equal. + `Resource.schema_url`, `Resource.dropped_attributes_count`, and any + `Resource.attributes` key not mapped to a `TracerPayload` field do not contribute to + the grouping key. The effective agent envelope is already pinned by the enclosing + `AgentPayload` step. +- Scalar reconstruction: the wire `TracerPayload`'s scalar fields are reconstructed by + inverting the ingress mapping. `Resource.host` populates `TracerPayload.hostname`, + `Resource.environment` populates `TracerPayload.env`, and the semantic-convention + attributes per "`TracerPayload` semantic-convention key mapping" populate the + corresponding `TracerPayload` scalars. +- Tags: `TracerPayload.tags` is reconstructed from + `TraceEvent.datadog.tracer.tags`; entries use the normalization above. The default + tracer context therefore emits an empty map. + +**`TraceChunk` grouping and re-coalescence.** Before grouping, map +`datadog.chunk = None` to a synthesized `DatadogChunkContext` whose other fields are +default and whose missing `priority` is treated as the egress value `AutoKeep`. +Grouping keys use that same effective priority, so `None` and +`Some(DatadogChunkContext::default())` share an egress group as `AutoKeep`. Within each +`TracerPayload`, group spans across events by the effective `DatadogChunkContext` plus +`TraceEvent.trace_id`, and emit one `TraceChunk` per group. This +re-coalesces the service partitions of one source `TraceChunk` while keeping a +non-conforming multi-trace chunk separated by ID. Distinct source chunks remain +distinct even when their reconstructed context and trace ID match. An explicit `Some(AutoReject)`, +including a Datadog-decoded all-default chunk whose wire priority was `0`, does not share +that group. + +The sink forms these effective chunk/trace groups before APM stats aggregation and +wire serialization when those partitions share a sink encoder batch. A partition +lacking the root span that does not share that batch may derive the default APM +weight; that split-batch case is outside the grouping guarantee. The computation +within a reconstructed group remains governed by RFC 9862. + +- Empty events: an event whose `spans` vector is empty contributes no spans to any + group; it emits one additional empty `TraceChunk` from the effective chunk context. + The Datadog wire shape has no carrier for that event's `TraceEvent.trace_id`. Such + events are outside the pure-relay guarantee. +- Tags comparison and serialization: the `tags` comparison is canonical structural + equality with deterministic key ordering. `DatadogChunkContext.tags` entries are serialized to + the wire `TraceChunk.tags` (`map`) via `dd_value_to_string`. For + Datadog-sourced events, chunk tags are always `AttrValue::String` on ingress so this + stringification is lossless on round-trip. +- Cross-grouping invariant: chunk grouping is nested inside `TracerPayload` grouping + which is nested inside `AgentPayload` grouping, so events in the same chunk group + are by construction in the same `TracerPayload` and `AgentPayload`. +- Round-trip shapes: a multi-service wire chunk that was split into multiple events on + ingest re-coalesces into one chunk on egress; a non-conforming multi-trace chunk, + including one with several services per trace, produces one egress chunk per + `trace_id`. Re-coalescence concatenates partitions in first-seen pair order, + preserving span order within each partition; it does not restore original + cross-service interleaving. Both shapes are equivalent to the input as observed by + the Datadog backend (see Scope). + +#### Cross-format conformance: `OTLP -> Vector -> datadog_traces` + +An event whose Datadog-native slots are absent or default follows the enumerated +Agent-aligned derivations below. Explicit Datadog state lifted from reserved OTLP bridge +attributes is authoritative instead. Conformance means backend-effective +equivalence under Vector's typed mapping, not exact reproduction of every field the +Datadog Agent OTLP converter would place on the wire. Current Agent `main`, as linked in +the Glossary, is the reference for these derivations: + +- The `Span.type` derivation (`SpanKind2Type` logic) and the `Span.resource` derivation + (attribute-key lookup with `Span.name` fallback) follow the upstream Agent code. +- The `TracerPayload` semantic-convention key set defining which `Resource.attributes` + keys populate which `TracerPayload` scalar fields is similarly upstream-tracking. +- Agent-envelope synthesis when `.datadog.agent` is absent uses proto3 defaults for + every `AgentPayload` field, including `hostName` and `env`. +- Chunk priority when `.datadog.chunk` is absent or `.datadog.chunk.priority` is `None` + emits `AutoKeep` (wire 1). Vector does not implement the Agent's probabilistic OTLP + sampler; this matches today's `datadog_traces` sink and the Agent converter at its + default sampling rate. + +The rest follows this RFC's explicit typed mapping. In particular, Vector emits +`SpanEvent` and `SpanLink` through their native Datadog protobuf fields and preserves a +128-bit trace ID through `_dd.p.tid`; it does not reproduce the Agent converter's legacy +`meta["events"]`, `meta["_dd.span_links"]`, `meta["otel.trace_id"]`, or +`meta["span.kind"]` encodings unless a separate mapping rule above requires the same +field. Those wire-shape differences are permitted only where the Datadog backend +observes equivalent trace data. + +`datadog_agent -> Vector -> OTLP` is the inverse of the forward mapping for fields the +reference covers. The OTLP mapping sub-RFC defines reserved `datadog.*` resource and +span bridge attributes for the agent envelope, tracer tags, chunk state, span resource +name and type, and `meta_struct`. OTLP egress synthesizes those attributes from the +typed Datadog namespaces and OTLP ingress lifts them back. This entire path is +best-effort and is explicitly out of the zero-loss round-trip guarantee. + +No upstream reference implementation for the reverse direction (Datadog wire -> OTLP) is +cited as normative; the OpenTelemetry Collector's +[`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) +exists in `opentelemetry-collector-contrib` and may be consulted as a secondary +reference for fields that the Datadog Agent OTLP ingest does not document, but it is +not authoritative. + +## Rationale + +- The `meta` / `metrics` merge into `Span.attributes` relies on a producer-side + disjointness convention rather than a wire-format invariant. The Datadog `Span` proto + does not constrain keysets across the two scalar maps, but every examined Datadog SDK + and the trace agent maintain disjointness by construction, and the two maps carry + distinct value types (`AttrValue::String` versus `AttrValue::Double`) even in the + rare collision case. The model treats the keyset disjointness as a contract the + Datadog source asserts. If the convention ever ceases to hold for production traffic, + the contained fallback (a separate `Span.datadog_attributes` field) is documented + under "Alternatives". +- Datadog's wire `Span.error` is `int32` but documented as bivalent (`0` or `1`). The + typed model normalizes non-conforming values to `Span.error = 1` on egress, diverging + from the pre-typed-model sink which preserved arbitrary `int32` values byte-exactly. The + normalized form is what every Datadog backend documents; the typed `SpanStatus` enum + has no carrier for non-bivalent values, and preserving the raw integer would require + an `Option` shadow field that no consumer would read. +- The `meta_struct` partition is preserved in `Span.datadog.meta_struct` rather than + merged into the common attribute map. `AttrValue` distinguishes `String` from `Bytes` + structurally, but the partitions are semantically distinct: `meta_struct` payloads + are msgpack-encoded structured records, not opaque scalars. A dedicated bytes map + also prevents a producer scalar named `_dd.meta_struct` from colliding with the wire + partition. +- Agent-payload- and tracer-payload-scoped state are kept as separate typed contexts + rather than merged because the two scopes collide on + known keys at both the tag-map level and the scalar level. The Datadog Agent's trace + writer + ([`pkg/trace/writer/trace.go`](https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/writer/trace.go)) + writes `_dd.apm_mode` into `AgentPayload.tags` from its own configuration, and the + Agent's processing pipeline + ([`pkg/trace/agent/agent.go`](https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/agent/agent.go)) + writes the same key into `TracerPayload.tags` from a span's `Meta`. The two values + are semantically distinct (Agent's claimed mode versus tracer-reported mode) and + appear in the same payload. The same collision class applies to the scalar fields: + `AgentPayload.hostName` / `env` describe the collector and routinely differ from + `TracerPayload.hostname` / `env` (which describe the application), and Vector's + existing egress sink already partitions on the agent-level values to keep the two + attribution domains distinct. `DatadogAgentEnvelope` holds the full agent envelope + while `DatadogTracerContext` carries only tags because the other tracer-payload + fields have common `Resource` slots. +- Namespace authority is an intent boundary, not a security boundary. Ordinary + attributes cannot accidentally become an agent envelope, while a transform that + explicitly writes `.datadog` is asking to control Datadog-native egress state. This + removes source-type heuristics and preserves the state across `vector` hops. +- The Datadog egress rule for the `Span.status` / `error.message` typed-slot/attribute + pair preserves the pure round-trip property by construction. A Datadog-sourced span + with `error = 1` and no `meta["error.message"]` ingests as `Error("")` with no + attribute, and egress emits no meta tag (the empty-message guard suppresses the + overwrite). A span with `error = 1` and `meta["error.message"] = "x"` ingests as + `Error("x")` with `attributes."error.message" = "x"`, and the typed overwrite writes + the same `"x"` back -- no divergence event fires because the values are equal. The new + behaviour relative to the pre-RFC sink appears only when a transform mutates one of the + two without the other; in that case the typed value is selected and the divergence is + observable, matching the precedent set on the other Datadog typed slot/attribute pairs + (`Resource.{service,environment,host}` and `TraceEvent.trace_id.high_u64`). +- The Datadog egress chunk-grouping rule + `(effective DatadogChunkContext, TraceEvent.trace_id)` + follows directly from the typed model's single-trace-ID invariant. For the conforming + case, multi-service chunks split on ingest re-coalesce into one egress chunk and + single-service chunks pass through unchanged; for a non-conforming multi-trace chunk, + egress emits one chunk per `trace_id`. Both shapes are effectively equivalent at the + Datadog backend, since chunk grouping is an ingestion-time transport detail rather + than a semantic primitive. All four `DatadogChunkContext` fields contribute to the + grouping key; `dropped` (`droppedTrace` on the wire) is chunk-scoped sampler state, + not a per-group attribute: two chunks that share the same + `(priority, origin, tags, TraceEvent.trace_id)` but differ on `droppedTrace` must + remain distinct egress chunks, otherwise the relay re-emits the second chunk's spans + with the wrong dropped flag. +- Attribute iteration order within `SpanEvent.attributes` is not preserved by the + parent RFC's key-sorted `Attributes` carrier. Although the upstream Datadog + `Span.proto` notes that this order "should be preserved," the comment is not honored + by Datadog's own producers, so the reordering falls under the Scope clause for details + the backend does not observe and needs no exclusion. + +## Drawbacks + +- The Datadog round-trip guarantee depends on a producer-side keyset-disjointness + convention between `meta` and `metrics`. The Alternatives below describe contained + fallbacks if this convention ever ceases to hold. +- Events without an explicit Datadog agent envelope reaching the `datadog_traces` sink + emit proto3 defaults for every `AgentPayload` field, including `hostName` and `env`. + Application identity remains on `TracerPayload`. This matches leaving collector + identity unset when none was supplied, but operators who expected the relay to copy + `Resource.host` / `Resource.environment` onto the agent envelope will be surprised. +- Datadog's `SpanLink.flags` bit 31 sentinel is not synthesized on egress for OTLP- + sourced events, so the W3C trace-flags byte plus the OTLP remote-context tristate + carried by such links are not surfaced through the Datadog wire. This is a cross- + format asymmetry consistent with cross-format zero-loss being out of scope in the + parent RFC. + +## Prior Art + +- [Datadog APM agent-to-backend + protobuf](https://github.com/DataDog/datadog-agent/tree/main/pkg/proto/datadog/trace) + -- the wire format this sub-RFC targets. +- [Datadog Agent OTLP + ingest](https://github.com/DataDog/datadog-agent/blob/main/pkg/trace/api/otlp.go) -- + the moving reference for the enumerated Agent-aligned derivations. Reusing those + derivations minimises attribution differences without requiring Vector to reproduce + the Agent converter's complete wire output. +- The OpenTelemetry Collector + [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) + -- a secondary reference for the reverse direction (Datadog wire -> OTLP), used only + for fields the Datadog Agent OTLP ingest does not document. Not authoritative. + +## Alternatives + +### Separate `Span.datadog_attributes` field preserving the three wire partitions verbatim + +Carry a `DatadogAttributes { meta, metrics, meta_struct }` field on `Span` alongside the +canonical `attributes`, populated only on Datadog ingest. This represents the wire +format exactly and preserves any cross-partition collision between `meta` and `metrics`. +Rejected because it splits the attribute surface in two, forces every attribute-aware +component to handle both, and is paid against a `meta` / `metrics` collision case no +examined Datadog SDK or agent emits. Listed as the contained mechanical fallback if the +producer-side disjointness convention ever ceases to hold for production traffic; the +change is local to `Span`, the Datadog source, the Datadog sink, and a unified read +helper, with no impact on the OTLP side. The `meta_struct` partition is already +preserved exactly under `Span.datadog.meta_struct` in the proposal and does not +motivate this alternative. + +### Namespace-prefixed unified map for span partitions + +Encode Datadog's two scalar span-attribute partitions inside `Span.attributes` itself by +prefixing each key with its partition name (`dd.meta.`, `dd.metrics.`), with +`meta_struct` similarly flattened under `dd.meta_struct.`. Rejected because the +prefixes leak Datadog-specific encoding into every transform regardless of source: an +OTLP-only pipeline has to know about the namespace to avoid colliding with it, and an +OTLP-sourced attribute that happens to use a `dd.meta.*` key is silently misclassified +on egress. The `AttrValue`-variant routing for `meta` / `metrics` and the typed +`Span.datadog.meta_struct` map achieve the same egress mapping without imposing any +naming constraint on the flat attribute namespace. + +### Reserved resource-attribute envelope objects + +Encode `DatadogEventContext.agent` and `.tracer` under +`Resource.attributes."_dd.payload"` / `"_dd.tracer"` and use event provenance to decide +whether Datadog egress should honor them. Rejected primarily because provenance signals +are rewritten at intermediate `vector` hops, losing envelope authority unless operators +configure a separate passthrough rule. Custom OTLP resource attributes may also +legitimately use those keys, requiring a dynamic origin check to distinguish common +attributes from Datadog wire state. The explicit `.datadog` namespace makes authoring +intent structural, survives internal serialization, and leaves ordinary `_dd.*` +resource attributes untouched. + +### `DatadogChunkContext.priority` as a raw `i32` + +Datadog's wire representation is a signed integer with four well-known values +(`UserReject = -1`, `AutoReject = 0`, `AutoKeep = 1`, `UserKeep = 2`). Storing the raw +`i32` directly is simpler. Rejected because transforms that condition on priority then +have to compare against magic numbers, and there is no way to surface "this is a +non-standard value" to the user. A strict enum with an `Other(i32)` escape hatch +(parent RFC) keeps typed ergonomics for the common path while preserving any out-of- +range value. + +## Outstanding Questions + +- N/A. + +## Plan Of Attack + +Removing the obsolete `tracerPayloads`-empty ingest branch and adopting the upstream +Datadog protos may land independently. That removal must report a payload whose +`tracerPayloads` is empty, including one that still carries `traces` / `transactions` +spans, rather than treating it as a lossless empty modern payload. + +The parent RFC's stages 2 and 4 define the common sink-local validation and subsequent +coexistence integration. This sub-RFC supplies the following Datadog-specific work: + +1. Implement the Datadog converter and unique detection of historical pre-hint layouts. + The converter must split each chunk by reconstructed trace ID and `Span.service` in + stable first-seen pair order and apply the parent RFC's metadata and acknowledgement + contract to the resulting sequence. +2. Implement `TraceEvent -> Datadog` encoding satisfying the mapping and egress contracts + above. +3. During the parent RFC's sink-local stage, migrate the `datadog_traces` sink and APM + stats aggregation. Reconstruct effective chunk/trace groups before invoking the + aggregator when those partitions share a sink encoder batch. Establish the + `Datadog -> Vector -> Datadog` effective-equivalence guarantee, validate every + declared exclusion, and establish the enumerated + `OTLP -> Vector -> datadog_traces` conformance rules against the current Datadog + Agent reference. +4. During coexistence integration, register the converter as the Datadog shim and adapt + every `datadog_traces` intake path to accept both variants. +5. Publish the user migration guide, then migrate the `datadog_agent` source after the + parent RFC's compile-time consumer gate and after typed input passes end-to-end + through Datadog export. Typed source events retain the migration hint for the parent + RFC's deprecation window. + +## Future Improvements + +- Datadog `AgentPayload.idxTracerPayloads = 11` (indexed / deduplicated tracer-payload form): map + the indexed shape through the same typed model. Until that support is added, indexed + entries are discarded while standard entries continue. A stricter mode may instead + reject any payload containing indexed entries when operators prefer all-or-nothing + acceptance. The indexed shape has been reviewed against the typed slots and fits + field-for-field with no structural changes required; remaining work is string-table + codec at the wire boundary. +- VRL helper for decoding raw Datadog `Span` protobuf into the typed surface + (`decode_datadog_span`), parallel to the OTLP mapping sub-RFC's `decode_otlp_span` + helper. +- Adopt the Datadog tracer-to-agent endpoints (`/v0.3/traces`, `/v0.4/traces`, + `/v0.5/traces`, `/v0.7/traces`) as additional Datadog ingress shapes. The endpoints + are upstream of the agent-to-backend hop this sub-RFC targets; the typed model + accommodates them with format-specific shims and no schema changes. diff --git a/rfcs/2026-04-29-25329-trace-data-model/otlp-mapping.md b/rfcs/2026-04-29-25329-trace-data-model/otlp-mapping.md new file mode 100644 index 0000000000000..36707494726bd --- /dev/null +++ b/rfcs/2026-04-29-25329-trace-data-model/otlp-mapping.md @@ -0,0 +1,462 @@ +# RFC 25329 - 2026-04-29 - Trace Data Model: OTLP Mapping + +This sub-RFC of [RFC 25329 -- Internal Trace Data Model](../2026-04-29-25329-trace-data-model.md) +specifies the bidirectional mapping between the typed `TraceEvent` defined in the parent RFC +and the OTLP wire format. It establishes the +[`OpenTelemetry Protocol`](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto) +ingress and egress paths, the effective-equivalence round-trip guarantee for +`OTLP -> Vector -> OTLP`, and the per-attribute encoding rules between OTLP's `AnyValue` and the +parent RFC's `AttrValue`. + +## Context + +- The parent RFC defines the typed data model, migration mechanics, and internal wire + serialization. This sub-RFC assumes those definitions and the parent's Glossary, In/Out + scope clauses, and User Experience as background. In particular, every failure, drop, + and normalization below is reported per the parent's global reporting rule and the + [instrumentation specification](../../docs/specs/instrumentation.md) it cites, whether + or not the rule here restates it. +- [RFC 11851 -- OpenTelemetry traces source](../2022-03-15-11851-ingest-opentelemetry-traces.md) + was accepted on the condition that an internal trace model be established before the work was + completed. This sub-RFC, together with the parent and the Datadog mapping sub-RFC, satisfies + that condition. + +## Cross cutting concerns + +- First-class OpenTelemetry signal support + ([vectordotdev/vector#1444](https://github.com/vectordotdev/vector/issues/1444)). +- Cross-format relay (OTLP source -> `datadog_traces` sink): the conformance rule is specified + by the Datadog mapping sub-RFC; this sub-RFC defines only OTLP-side ingress and egress. + +## Scope + +### In scope + +- The bidirectional mapping between `TraceEvent` and OTLP `ResourceSpans` / `ScopeSpans` / + `Span` messages. +- The stable partition of each `ScopeSpans` by its wire spans' `trace_id`, producing one + `TraceEvent` per distinct ID with that ID stored at `TraceEvent.trace_id`. +- The parent RFC's effective-equivalence round-trip guarantee, applied to + `OTLP -> Vector -> OTLP`. Attribute iteration order within `Span.attributes`, when the + producer-side ordering was non-canonical, and the partition of spans among otherwise + equivalent `ScopeSpans` messages are details an OTLP backend does not observe and so + may differ. +- The promotion rule for the three semantic-convention attributes (`service.name`, + `deployment.environment.name`, `host.name`) into typed `Resource` slots, including the + legacy-key acceptance for `deployment.environment`. +- The 1:1 mapping between `AttrValue` variants and OTLP's `string_value` / `bytes_value` / + `int_value` / `bool_value` / `double_value` / `array_value` / `kvlist_value` (with unset + oneof representing `AttrValue::Null`). +- Reservation of the `datadog.` attribute-key prefix at the OTLP wire boundary, at both + resource and span scope, for synthesis on OTLP egress and lift on OTLP ingress of the + Datadog-native contexts, for best-effort cross-format relay. This sub-RFC reserves the + prefix while the Datadog sub-RFC specifies the context values those keys carry. +- The OTLP mapping targets fields at the OpenTelemetry + [`Stable`](https://opentelemetry.io/docs/specs/otel/versioning-and-stability/) + stability tier or higher. + +### Out of scope + +- The Datadog wire mapping (see the Datadog mapping sub-RFC). +- Zero-loss cross-format round-trip (`OTLP -> Datadog -> OTLP`); see the parent RFC's Out of + scope. +- OTLP fields at `Development` or `Alpha` stability tier; see Future Improvements for the + adoption path. + +### Zero-loss round-trip exclusions + +The effective-equivalence guarantee for `OTLP -> Vector -> OTLP` does not cover the following +input shapes. Each is justified by a paragraph in the Implementation or Rationale section +below. + +- **Promoted deprecated `deployment.environment` key** is rewritten to + `deployment.environment.name` on OTLP egress. If both keys are present on ingress, the + deprecated value is dropped (the stable key wins). See the deprecated-environment + paragraph under Implementation. +- **Reserved cross-format OTLP attributes** under the `datadog.` prefix are lifted into + the typed `TraceEvent.datadog` / `Span.datadog` slots on OTLP ingress and stripped, + then synthesized from those slots on OTLP egress, removing or replacing any existing + common attribute at the same key. +- **OTLP fields at `Development` or `Alpha` stability tier** are dropped on OTLP ingress. +- **`Span.end_time_unix_nano < start_time_unix_nano`** is clamped to zero duration on + ingress. The egress reconstruction emits + `end_time_unix_nano = start_time_unix_nano`, byte-different from the original input. +- **`Status.message` paired with `UNSET` or `OK`** is dropped on ingress; + OpenTelemetry permits a description only for `ERROR`. +- **Duplicate keys in an OTLP attribute list or nested `KeyValueList`** use wire-order + last-wins normalization. Each overwritten value saturating-increments the nearest + enclosing item's `dropped_attributes_count`. +- **Empty `ScopeSpans`** produce no event because the wire grouping supplies no trace ID + for required `TraceEvent.trace_id`. Resource and scope state carried only by empty + groupings is therefore not relayed. +- **Resource and scope `dropped_attributes_count` after partition** are cloned onto + every trace-ID event. Egress re-coalescence of those partitions restores one count; + counts that do not re-coalesce, or that merge originally separate equal groupings, + may differ from the input. + +The OTLP-side consequences of the parent RFC's zero-ID rejection and wire-domain +normalization also apply. The derived end-timestamp case is documented under "Span +timing" below. + +## Pain + +- The `opentelemetry` trace source today emits an untyped `TraceEvent(LogEvent)` whose key + layout is the source's choice. Cross-format relay to the `datadog_traces` sink requires + bespoke per-key translation; relay back to OTLP (if Vector grows an OTLP sink) would have + to re-discover the original wire shape. Both directions duplicate work that a typed model + removes by construction. +- The OTLP `AnyValue` oneof discriminator (`string_value` versus `bytes_value` versus + `int_value`, etc.) is lost when attribute values are stored as raw bytes in today's + `LogEvent`-backed trace events. Egress must guess the variant; for non-UTF-8 byte + payloads the guess is always wrong. The parent RFC's `AttrValue` storage preserves the + discriminator structurally. + +## Proposal + +### User Experience + +The OTLP wire mapping is a projection onto the typed `TraceEvent` defined in the parent +RFC. The only OTLP-specific surface the user sees is the specification below of which +wire fields populate which typed slots, used by operators diagnosing relay-path +discrepancies and by component authors writing OTLP encoders / +decoders. + +### Implementation + +#### Ingress and egress mapping + +Each OTLP `ScopeSpans` is partitioned by the wire `Span.trace_id` in first-seen ID +order. Span order within each ID is preserved. + +Every resulting event receives the containing `ResourceSpans.resource` as +`TraceEvent.resource` and the `ScopeSpans.scope` as `TraceEvent.scope`. +`TraceEvent.datadog` and each `Span.datadog` start at their defaults and are populated +only by valid reserved `datadog.*` bridge attributes as specified below. +An empty `ScopeSpans` produces no event because it supplies no trace ID. A non-empty +`ScopeSpans` whose spans are all rejected likewise produces no event under the parent +RFC's all-spans-rejected rule. A typed event made empty by a transform remains +representable because it retains `TraceEvent.trace_id`. + +The OTLP legacy shim applies the same partitioning to every `ScopeSpans` it recovers. A +pre-flip per-span OTLP event converts one-to-one. Fields the current per-span layout +does not carry take typed defaults and are not recovered. A legacy event that still +carries `ScopeSpans` groupings may fan out by distinct trace IDs and can populate +those slots. Metadata, finalizers, and acknowledgements follow the parent RFC's +conversion contract. + +The `OTLP -> Vector -> OTLP` effective-equivalence guarantee is a mapping from decoded +`ResourceSpans` / `ScopeSpans` through the typed model and back. It is not a claim that +the default pre-flip source layout is lossless. Pipeline relay of the fields that +layout drops waits until the source emits typed events. + +| OTLP | Internal | +| ------------------------------------------------------------------ | --------------------------------------------- | +| `ResourceSpans.resource.attributes["service.name"]` | `Resource.service` | +| `ResourceSpans.resource.attributes["deployment.environment.name"]` | `Resource.environment` (see below) | +| `ResourceSpans.resource.attributes["deployment.environment"]` | `Resource.environment` (legacy fallback) | +| `ResourceSpans.resource.attributes["host.name"]` | `Resource.host` | +| `ResourceSpans.resource.attributes` (others, see promotion rule) | `Resource.attributes` | +| `ResourceSpans.resource.dropped_attributes_count` | `Resource.dropped_attributes_count` | +| `ResourceSpans.schema_url` | `Resource.schema_url` | +| `ScopeSpans.scope.{name, version, attributes}` | `Scope.{name, version, attributes}` | +| `ScopeSpans.scope.dropped_attributes_count` | `Scope.dropped_attributes_count` | +| `ScopeSpans.schema_url` | `Scope.schema_url` | +| `Span.trace_id` | `TraceEvent.trace_id` | +| `Span.span_id`, `Span.parent_span_id` | same (zero-ID handling: see below) | +| `Span.trace_state` | `Span.trace_state` (verbatim) | +| `Span.flags`, `Link.flags` (see flags layout) | `Span.flags`, `SpanLink.flags` (full u32) | +| `Link.trace_id` (see zero-ID handling) | `SpanLink.trace_id` (link target) | +| `Span.name`, `Span.kind` | `Span.name`, `Span.kind` | +| `Span.start_time_unix_nano`, `end_time_unix_nano` (see timing) | `Span.start_time`, `Span.duration` (ns-exact) | +| `Span.attributes` | `Span.attributes` | +| `Span.events`, `Span.links` | `Span.events`, `Span.links` | +| `Span.status.{code,message}` (see status) | `Span.status.{code,message}` | +| `Span.dropped_*_count` | `Span.dropped_*_count` | + +Attributes under the reserved `datadog.` prefix are exceptions to the generic attribute +rows: resource-level keys lift into `TraceEvent.datadog`, while span-level keys lift +into `TraceEvent.datadog.chunk` or `Span.datadog`. The bridge section below defines +that reservation. + +`SpanLink.trace_id` identifies the link target and may differ from the enclosing +`TraceEvent.trace_id`. + +On OTLP egress, `TraceEvent`s sharing a `Resource` (including `Resource.schema_url`) and +the same resource-level Datadog bridge projection are gathered into one +`ResourceSpans`. Within it, non-empty events sharing the same `Scope` and event-level +`TraceEvent.datadog` bridge projection are re-coalesced into one `ScopeSpans`; spans retain event +order and their order within each event. Re-coalescence of partitions that share an +encoder batch restores cloned resource and scope dropped counts (see Scope). The +emitted wire grouping may again contain several trace IDs; the single-ID invariant +applies to Vector's internal `TraceEvent`, not to OTLP. + +Every emitted OTLP `Span.trace_id` is copied from the enclosing +`TraceEvent.trace_id`; internal spans have no independent trace ID. + +An empty event is not coalesced and becomes one `ScopeSpans { spans: [] }`; OTLP has no +field in that shape to carry its `TraceEvent.trace_id`. Such events arise only after +typed construction or transformation, outside the pure-relay guarantee. Two events +with identical common `Resource` content but different agent envelopes or tracer tags +therefore produce separate `ResourceSpans` messages. Datadog-native context has no +common attribute-map representation; the OTLP sink synthesizes it under reserved +resource and span keys (see "Reserved OTLP bridge keys" below) for best-effort +cross-format relay. Ordinary `_dd.*` keys have no special meaning and flow through the +generic `AttrValue` -> `AnyValue` mapping. + +#### Zero-ID detection + +OTLP wire IDs are raw byte arrays interpreted in big-endian order. Because they are +length-delimited rather than fixed-width, this mapping adds a structural requirement the +parent RFC's rule does not imply: a trace ID must be exactly 16 bytes and a span ID +exactly 8 bytes, and a wrong-length ID is malformed in the same way an all-zero ID is. +Rejection granularity and reporting then follow the parent RFC's identifier rule. + +`Span.parent_span_id` has one explicit normalization because the model represents root +spans with `None`: an empty sequence or exactly eight zero bytes maps to `None`, and +exactly eight non-zero bytes maps to `Some(SpanId)`. Any other non-empty length is +malformed and rejects the enclosing span with the same invalid-ID telemetry. On egress, +IDs use the same fixed-width big-endian encoding and a `None` parent emits an empty +`parent_span_id`. + +#### Semantic-convention promotion and the typed-slot precedence + +Three resource attribute keys promote to typed `Resource` slots on ingress: `service.name`, +`deployment.environment.name` (with `deployment.environment` accepted as a legacy +fallback -- see below), and `host.name`. + +Promotion to a typed `Resource` field is conditional on the attribute value being a +`string_value`. When the value is a non-empty string (the normal case), promotion is +move-not-copy: the key is removed from `Resource.attributes` and the typed slot is the +sole post-ingress owner of the value. This matches the move-not-copy pattern used for +`_dd.p.tid` consumption on Datadog ingest and for the reserved cross-format keys. An +empty `string_value` or a non-string variant is not promoted: the typed slot remains +`None` and the attribute remains under its original key, preserving the producer's +value and its presence on OTLP egress. + +A transform that changes the service, environment, or host writes the typed slots +(`Resource.service`, `Resource.environment`, `Resource.host`) rather than +the corresponding attribute-map keys. Because promotion strips the source attribute on +ingress, the duplicate-key case arises only when (i) a transform writes to both the typed +slot and the matching attribute key, or (ii) the source attribute was empty or non-string +and the promotion rule above left it in place. + +On OTLP egress specifically, the typed slot wins for the three pairs above: the canonical key is +emitted once with the typed value and any duplicate at the same key in `Resource.attributes` is +discarded, saturating-incrementing the emitted +`Resource.dropped_attributes_count`. This is required for spec conformance: OTLP +`Resource.attributes` mandates that "attribute keys MUST be unique." If the typed slot +is `None` and the attribute key is present, the attribute value is emitted unchanged (the +non-promotion rule above applies). The other typed slot/attribute-map pairs from the +parent RFC do not apply on OTLP egress: `TraceEvent.trace_id` supplies the single +16-byte wire `Span.trace_id` field (with no `_dd.p.tid` duplication), `Span.status` +egresses through OTLP's `Status.message` field with any `error.message` +attribute left in place as a regular attribute, and the chunk-state pair is the cross-format +synthesis covered under "Reserved OTLP bridge keys for Datadog-native state" below. + +#### Deprecated `deployment.environment` handling + +The OTLP source accepts both `deployment.environment.name` and the deprecated +`deployment.environment` as sources for `Resource.environment`. OpenTelemetry stabilized +the attribute as `deployment.environment.name` in semantic conventions +[v1.27.0](https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.27.0) +([PR #3584](https://github.com/open-telemetry/semantic-conventions/pull/3584)), with +`deployment.environment` listed as "Replaced by `deployment.environment.name`." + +The stable key wins when both are present; dropping the deprecated value +saturating-increments `Resource.dropped_attributes_count`. The selected +key is then subject to the promotion rule above: a non-empty string promotes, while an +empty or non-string value remains under its original key. On OTLP egress, a promoted +`Resource.environment` uses +`deployment.environment.name`; an unpromoted attribute retains its original key and +value. + +The Rationale for accepting both keys is in the Rationale section. + +#### Span timing + +OTLP carries timing as two independent `fixed64` nanosecond values, `start_time_unix_nano` +and `end_time_unix_nano`. On ingress, `Span.duration` is computed as +`end_time_unix_nano − start_time_unix_nano`; on egress, `end_time_unix_nano` is +reconstructed as `start_time_unix_nano + duration.as_nanos()`. Both quantities are integer +nanoseconds in memory and on the wire; the round trip is bit-exact for any span where +`end_time_unix_nano >= start_time_unix_nano`. + +A span with reversed timestamps (`end_time_unix_nano < start_time_unix_nano`) is clamped to zero +duration on ingress; this is one of the OTLP-side zero-loss exclusions listed above. + +Egress clamping of `Span.start_time`, `SpanEvent.time`, and the reconstructed +`Span.end_time_unix_nano` follows the parent RFC's numeric boundary policy against the +OTLP timestamp domain. + +#### `Span.flags` / `Link.flags` layout + +OTLP defines `Span.flags` and `Link.flags` as `fixed32`, with bits 0-7 the W3C trace-flags +byte, bits 8-9 the parent-/link-target-remote tristate (`CONTEXT_HAS_IS_REMOTE`, +`CONTEXT_IS_REMOTE`), and bits 10-31 reserved. The full word is stored verbatim in the +parent RFC's `TraceFlags(u32)`, so all defined bits and any future spec additions +round-trip unchanged. + +#### `Span.status.code` and `Span.status.message` + +`Status.message` round-trips when `code = ERROR` (carried by `SpanStatus::Error(String)`) or when +`code` is an unrecognized future value (carried by `SpanStatus::Other(i32, String)`). For `code = +UNSET` or `OK` the message is dropped on ingest because the OpenTelemetry [Set +Status](https://opentelemetry.io/docs/specs/otel/trace/api/#set-status) rule restricts `Description` +to the `Error` code. A wire `Status.message` paired with `code = UNSET` or `OK` is non-conformant +and is dropped on ingest. See the Rationale subsection below for the +closed-enum-with-escape-hatch choice that makes future status codes round-trip unchanged. + +#### Attribute encoding: `AttrValue` <-> `AnyValue` + +The parent RFC's `AttrValue` mirrors OTLP `AnyValue` directly, so the mapping is 1:1 +across the named variants. `AttrValue::Null` corresponds to an unset `AnyValue` oneof +(equivalently, for proto3, a `KeyValue` whose `value` field is absent), and the +mapping applies recursively into `Array` and `Map`. `Value::Timestamp` and +`Value::Regex` have no `AttrValue` representation, so OTLP +egress never sees those variants in storage. + +OTLP attribute fields and nested `KeyValueList` values are repeated key-value sequences, +while the typed model requires unique map keys. Ingest processes each sequence in wire +order before promotion or reserved-key lifting; when a key repeats, the last value wins. +Each overwritten earlier value saturating-increments the nearest enclosing resource, +scope, span, event, or link's `dropped_attributes_count`. OTLP egress is +unique-keyed by construction. + +#### Default-valued / absent equivalence + +OTLP's spec defines its "field absent" and "field default-valued" forms as semantically +equivalent for `ResourceSpans.resource`, `ScopeSpans.scope`, `Span.status`, and +`Span.kind`. The typed model relies on that equivalence rather than tracking wire +presence: it carries `Resource` and `Scope` as values rather than `Option`s, represents +an unset status as `SpanStatus::Unset`, and egress emits each field unconditionally at +its default. A round trip therefore preserves the spec-defined semantics even where the +wire bytes differ, and no exclusion is needed. + +`Scope.{name, version}` and both `schema_url` fields +are non-optional proto3 strings, so decoding cannot distinguish absent from empty and they +map empty to `None`. Unknown `Span.status.code` or `Span.kind` values are carried by +the parent RFC's escape-hatch variants and egress as the same integer, so values +introduced by future OpenTelemetry versions round-trip unchanged. + +#### Reserved OTLP bridge keys for Datadog-native state + +The generic `AttrValue` -> `AnyValue` mapping does not carry the separate Datadog +contexts through OTLP. This mapping therefore reserves the `datadog.` attribute-key +prefix at the OTLP wire boundary: OTLP egress synthesizes the typed Datadog contexts +under reserved keys, and OTLP ingress lifts those keys back into the typed contexts +and strips them from the common attribute maps. The agent envelope and tracer tags are +event-scoped and use `Resource.attributes`; chunk state and Datadog-native span fields +use `Span.attributes`. + +The contract is: + +- The typed namespace is the single source of truth on egress. A common attribute at a + reserved key is removed and replaced when the typed value requires emission, + saturating-incrementing the corresponding emitted dropped count. Ingress lifting and + stripping ensures OTLP egress through the same Vector emits each value once. +- A reserved key or nested member that is malformed -- wrong `AnyValue` variant, + unrecognized member, or out-of-domain value -- is stripped without populating the + typed slot and saturating-increments the enclosing item's + `dropped_attributes_count`. Valid siblings continue to lift. +- Ordinary `_dd.*` resource and span attributes are not reserved, carry no + Datadog-native meaning, and never acquire Datadog egress authority from their + spelling. +- Recovery of Datadog state through these keys is best-effort and is explicitly outside + the OTLP round-trip guarantee. + +One consequence not derivable from the rules above: +`datadog.chunk.priority` is emitted whenever the typed priority is `Some`, including +`AutoReject` (wire `0`). Omitting `0` as a proto3 default would make an explicit reject +indistinguishable from a missing priority after an OTLP hop, and Datadog egress of a +missing priority is `AutoKeep`, so the round trip would invert the sampling decision. + +The reserved key names, their per-member `AnyValue` types, and the presence rules for +optional and default-valued members are implementation choices that satisfy this +contract. Conflicting valid `datadog.chunk.*` values across spans in one trace-ID +partition of a `ScopeSpans` use wire-order first; the conflict is reported. + +## Rationale + +- The OTLP source accepts both `deployment.environment.name` and the deprecated + `deployment.environment`. Accepting only the new key would silently drop the value for + producers still on pre-stabilization conventions; accepting only the old key would + silently drop it for producers on current conventions. Both matter because + `Resource.environment` populates Datadog's `TracerPayload.env` on cross-format egress + (see the Datadog sub-RFC): a route that fails to recognize the producer's chosen key + emits an empty `TracerPayload.env` and loses environment attribution at the Datadog + backend. The collision rule (current key wins) and egress emission (current key only) + are documented under Implementation. A bit-exact relay alternative would require either + adding provenance state to `Resource` or moving the typed slot to a derived view over + `Resource.attributes`; both pay substantive cost for one transitional attribute. +- The OTLP `AnyValue.string_value` / `AnyValue.bytes_value` discriminator is preserved + structurally because the parent RFC's `AttrValue` carries `String` and `Bytes` as + distinct variants. OTLP egress is a 1:1 variant routing with no payload inspection, + so the round trip is bit-exact for pure-relay pipelines. +- The `datadog.*` namespace at the OTLP wire boundary carries Datadog-native state that + has no common attribute-map representation in the typed model. The explicit prefix + declares authoring intent and limits collisions with user-set OTLP attributes to a + small reserved set. Ordinary `_dd.*` resource and span attributes remain common + attributes and never acquire Datadog egress authority merely from their spelling. +- Always-present logical Datadog contexts let typed consumers use stable fields + without first testing the source format. Default contexts synthesize no OTLP bridge + attributes, so purely OTLP-native traffic pays no wire-size cost. + +## Drawbacks + +- Partitioning a `ScopeSpans` duplicates its `Resource` and `Scope` in memory and in + Vector's internal wire format once per distinct `trace_id`. OTLP egress re-coalesces + compatible non-empty partitions, but the duplication remains through the topology and + any disk buffer. +- Empty wire `ScopeSpans` values, including their resource and scope state, no longer + produce an event because no required top-level trace ID can be derived. +- Best-effort recovery of Datadog state from reserved OTLP resource and span attributes + is not guaranteed: a transform that drops or rewrites one of these attributes on + OTLP-stage traffic loses the corresponding Datadog state. Operators should not use + the reserved keys for unrelated custom attributes. + +## Prior Art + +- [OTLP traces protocol](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto) + -- the primary shape this RFC adopts. The container `TraceEvent` is structurally one + trace-ID partition of a `ScopeSpans` plus its `Resource`, with explicit Datadog + contexts projected through reserved OTLP bridge attributes when present. +- The OpenTelemetry [Collector OTLP receiver](https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver) + is the reference implementation of the OTLP ingress semantics; Vector's OTLP source + follows the same wire decoding. + +## Alternatives + +Wire-format-specific alternatives are in the parent RFC (`TraceFlags` width, `SpanStatus` +escape hatch). This sub-RFC inherits those choices. + +## Outstanding Questions + +- N/A. + +## Plan Of Attack + +The parent RFC's stages 2 and 4 define the common sink-local validation and subsequent +coexistence integration. This sub-RFC supplies the following OTLP-specific work: + +1. Implement the OTLP converter and unique detection of historical pre-hint layouts. +2. Implement `TraceEvent -> OTLP` encoding satisfying the mapping and bridge-key + contracts above. +3. During the parent RFC's sink-local stage, migrate the OTLP sink and establish the + `OTLP -> Vector -> OTLP` effective-equivalence guarantee, validating every declared + exclusion. +4. During coexistence integration, register the converter as the OTLP shim and adapt + every OTLP sink intake path to accept both variants. +5. After the parent RFC's compile-time consumer gate, migrate the source. Typed input + must pass end-to-end through OTLP export before the source emits typed events; typed + source events retain the migration hint for the window specified by the parent RFC. +6. Publish the OTLP field mapping and bridge-key conventions in the user migration + guide. + +## Future Improvements + +- Adopt typed support for OTLP fields as they reach `Stable` stability. The current scope + excludes `Development` / `Alpha` -tier additions; when upstream stabilizes any of these, + evaluate adding the corresponding typed slot and round-trip support, including a + cross-format storage convention for fields with no Datadog wire analog. +- VRL helper for decoding raw OTLP `Span` protobuf into the typed surface + (`decode_otlp_span`), parallel to the Datadog sub-RFC's `decode_datadog_span` helper. diff --git a/rfcs/_YYYY-MM-DD-issue#-title.md b/rfcs/_YYYY-MM-DD-issue#-title.md index e95d704a9b380..427b1254b700c 100644 --- a/rfcs/_YYYY-MM-DD-issue#-title.md +++ b/rfcs/_YYYY-MM-DD-issue#-title.md @@ -37,6 +37,8 @@ One paragraph description of the change. - Explain your change as if you were presenting it to the Vector team. - When possible, demonstrate with pseudo code not text. - Be specific. Be opinionated. Avoid ambiguity. +- Keep explanatory text about benefits of choices in "Rationale". +- Keep discussions of other options in "Alternatives". ## Alternatives