How to use OpenTelemetry sink ?

[zyy17]

Notice that Vector already has OpenTelemetry Sink and try the following config:

sources:
  otel_traces:
    type: "opentelemetry"
    http:
      address: "0.0.0.0:4318"
    grpc:
      address: "0.0.0.0:4317"

sinks:
  store_traces:
    type: opentelemetry
    inputs:
      - otel_traces.traces
    # My OpenTelemetry traces the backend.
    # It will accept the protobuf encode traces data through HTTP.
    protocol:
      type: "http"
      uri: "http://localhost:4000/v1/otlp/v1/traces"
      method: post
      encoding:
        codec: "native"

Unfortunately, my trace backend can't parse the data. It seems the data have been transformed to another format in Vector.

My vector version is:

./vector --version       
vector 0.44.0 (aarch64-apple-darwin 3cdc7c3 2025-01-13 21:26:04.735691656)

Also, I can't find any documents that describe the example of opentelemetry. How can I use OpenTelemetry sink?

[pront]

Hey @zyy17, if I understand correctly you are doing OTEL traces -> OTEL traces via Vector. Upon receiving the log, the OTEL source will decode them into Vector events. Native codecs aren't supported yet, so those Vector events need to be remapped back into what the OTEL collector expects.

See example here:
https://vector.dev/docs/reference/configuration/sinks/opentelemetry/#how-it-works

Edit:
For DYI solution:

• Reduce/Remap to publish batches: [sink OTLP] cannot ingest log batches  #22188 (comment) (credits to @someStrangerFromTheAbyss)
• VRL mappings: Provide OOTB mapping for incoming Opentelemetry logs to outgoing Opentelemetry logs #22696 (comment)

[rauanmayemir]

@pront I know that, but do I need to specify protocol.encoding.protobuf.desc_file? I'm guessing I do.

[pront]

Yes, you need to provide that.

[rauanmayemir]

That's what I thought, thanks! Was really looking forward to it, will wait until native protobuf is supported.
(sending json payloads was already possible with a http sink)

[pront]

Can you help me understand your use case better? It might be a while since that is supported. And when it is, it should be trivial to migrate.

IMO main problem we have now is batching and lack of auto-grouping, which can be worked around with the reduce transform.

Ref: #22188

[rauanmayemir]

My use case is that certain tools allow ingesting logs in opentelemetry format, but only support binary protobuf payloads.

[pront]

My use case is that certain tools allow ingesting logs in opentelemetry format, but only support binary protobuf payloads

Moving to a new thread @rauanmayemir

This is possible today by configuring the OTEL sink like so:

sinks:
  my_sink:
    inputs: ["remap_syslog"]
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:4318/v1/logs
      method: post
      codec: protobuf
      protobuf:
        desc_file: path/to/proto.desc
        message_type: Type.Name
      framing:
            method: bytes
      batch:
            max_events: 1
      headers:
        content-type: application/protobuf

Not sure about the framing section but here is what is available:
https://vector.dev/docs/reference/configuration/sinks/opentelemetry/#protocol.framing.method

(here's certainly room for UX improvements here but I am just demonstrating that it is possible today.

References:

• https://opentelemetry.io/docs/specs/otlp/#otlphttp-request
• https://github.com/open-telemetry/opentelemetry-proto/tree/v1.5.0/examples

[kmatata]

facing a similar issue, I think...

tried looking into the network traffic; sudo tcpdump -i lo port 5318 -A

found vector sending an array instead of json, even after setting the appropriate fields to use json:

{"code":3,"message":"ReadObjectCB: expect { or n, but found [, error found in #1 byte of ...|[{\"composit|..., bigger context ...|[{\"composite_sort_key\":\"20250217212324_0002\",\"level|..."}
03:43:14.916651 IP localhost.39072 > localhost.5318: Flags [.], ack 325, win 510, options [nop,nop,TS val 2844527821 ecr 2844527821], length 0
E..4..@.@.%.................c........(.....

---

suggestion by @pront was very useful; https://vector.dev/docs/reference/configuration/sinks/opentelemetry/#how-it-works

also helped to dealing with the array issue by handling it at the encoding level:
("at least for my case":)
vector sends data as,:

[{
  "resourceLogs": [{...}]
}]

and OLTP expects:

{
  "resourceLogs": [{...}]
}

hence adding this to transforms worked out well

# First create a new transform
transforms:
  format_for_otlp:
    type: "remap"
    inputs: ["prev_parsed_log"]
    source: |
    # Remove wrapping
    ...

im not sure why using the endpoint http:localhost/port/v1/traces doesnt work only /v1/logs....curious

Hope this helps in some small way