Skip to content

[Geneva] Batch OTLP-protobuf metric-point UDS sends to reduce per-export syscalls #4457

Description

@arman-msft

Component(s)

OpenTelemetry.Exporter.Geneva (metric exporter)

Is your feature request related to a problem? Please describe.

The Geneva metric exporter performs one Unix-domain-socket Socket.Send syscall per individual metric point during each export interval, in OtlpProtobufSerializer.SendMetricPoint.

At realistic cardinality (e.g. CardinalityLimit=4000 across multiple histograms ⇒ up to ~20K points/scrape), this dominates CPU on the exporter thread. Profile evidence from a production high-throughput .NET service shows Interop+Sys.Send at ~13% exclusive CPU, with the Geneva metric exporter as the dominant contributor.

The same pattern was identified in the Rust contrib repo for the Geneva log exporter: open-telemetry/opentelemetry-rust-contrib#306. The .NET Geneva log exporter already does buffered batching; only the metric exporter has the per-event syscall pattern.

Describe the solution you'd like

Batch metric-point events into bounded UDS sends.

Wire-format guarantee: when prefixBufferWithUInt32LittleEndianLength is set (the only mode used by MetricUnixDomainSocketDataTransport), each serialized event is already framed with a uint32-LE length prefix. The Geneva MDSD receiver already demultiplexes multiple events from a single read by that prefix, so concatenating events into one Send requires no wire-format change and no receiver change.

Proposed API additions to IMetricDataTransport (internal):

bool TryAppendOtlpProtobufEvent(byte[] buffer, int size);
void FlushOtlpProtobufEvents();
  • MetricUnixDomainSocketDataTransport accumulates events into an internal buffer (sized to match the existing per-event allocation, GenevaMetricExporter.BufferSize = 65360 bytes) and flushes via the existing UDS path.
  • MetricUnixUserEventsDataTransport and MetricWindowsEventTracingDataTransport preserve per-event semantics — each ETW / user_events write is a logical event for the consumer, so they forward TryAppend directly to SendOtlpProtobufEvent and treat Flush as a no-op.
  • OtlpProtobufSerializer.SendMetricPoint becomes TryAppend → Flush+retry → fallback to standalone Send (for any single event that exceeds the accumulation buffer; preserves current behavior).
  • SerializeAndSendMetrics calls FlushOtlpProtobufEvents() once at the end.
  • TlvMetricExporter uses a different framing (BinaryHeader + MetricEventType.TLV per event) and is intentionally out of scope.

This is intended to be transparent — no opt-in feature flag — since the wire format is unchanged.

Describe alternatives you've considered

  • Behind a feature flag — rejected: increases surface area and adds a flag users would never want to leave off.
  • Batching at the receiver side — not possible; the bottleneck is in the producer's syscall path.
  • Switching to SOCK_DGRAM — the existing transport is SOCK_STREAM, and dgram has its own per-message kernel limits; not necessary given the existing length-prefix framing.

Expected impact

  • Syscall reduction: 100×–1000× depending on metric-point count per scrape.
  • CPU reduction: in the profile that motivated this fix, Interop+Sys.Send was 12.82% exclusive with the metric exporter as the majority contributor. Recovering ~10% exclusive CPU on a high-throughput worker is the realistic target.
  • No wire change, no receiver (MDSD) change, no SDK change.

Additional context

Implementation ready as a PR — will link below. Tests include byte-for-byte equivalence between batched bytes and concatenated per-event sends, fallback for oversize events, and flush-failure semantics.

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp:exporter.genevaThings related to OpenTelemetry.Exporter.Geneva

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions