enhancement(datadog metrics sink): implement V3 by StephenWakely · Pull Request #26285 · vectordotdev/vector

StephenWakely · 2026-09-02T16:47:14Z

Summary

This changes the Datadog metrics sink to send metrics via the V3 protocol.

V3 is Datadog's columnar protobuf format. This format uses dictionary-based string
deduplication and delta encoding, making it more efficient than v2 for workloads with many
metrics that share common names or tags.

We lean on the V3 encoder crate currently in use by Saluki.

References

Vector configuration

sinks:
 datadog_metrics:
    inputs: ["host_metrics", "statsd"]
    type: "datadog_metrics"
    default_api_key: "${DD_API_KEY}"
    site: "datad0g.com"
    #series_api_version: v2
    batch:
      timeout_secs: 5

How did you test this PR?

#26086 was dogfooded internally. That branch created V2 and V3 transactions simultaneously. A backend service compared both payloads to check for differences.

For this PR, I created an account in datadog and sent dogstatsd and host metrics via v3.

Is this a breaking change?

Yes
[] No

Potentially. We change the default to send V3 metrics. The datadog backend handles V3, but this could break systems that depend on metrics being V2. To restore, configure:

   series_api_version: v2

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

Contributor Guidelines

Please read our Vector contributor resources.
Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
Some CI checks run only after we manually approve them. To catch issues early, add a pre-push hook (template) or run the following locally before pushing:
- make fmt
- make check-clippy (auto-fix with make clippy-fix)
- make test
After a review is requested, please avoid force pushes to help us review incrementally.
- Feel free to push as many commits as you want. They will be squashed into one before merging.
- For example, you can run git merge origin master and git push.
If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
run make build-licenses to regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.

…s protocol Add a `v3` option for the `datadog_metrics` sink's `series_api_version`, encoding series metrics with Datadog's columnar protobuf format at `/api/intake/metrics/v3/series`. The format uses dictionary-based string deduplication and delta encoding, making it more efficient than `v2` for workloads with many metrics sharing common names or tags. Includes: - new `encoder_v3` implementation and request builder support - resource handling for host and device fields, with correct precedence and multi-value support - timestamp resolution performed once per flush - sketches remain on the v2 protocol; `sketches_api_version: v3` is rejected - rate limiting of request-failed error logs - generated docs and changelog fragments

…rics-v3-main # Conflicts: # Cargo.lock # Cargo.toml # src/sinks/datadog/metrics/config.rs # src/sinks/datadog/metrics/encoder.rs

…ies default The generated advanced example embeds each option's default, so changing `series_api_version`'s default to `v3` left it out of sync and CI's `vdev check component-examples` failed on it.

chatgpt-codex-connector · 2026-09-21T16:05:29Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-24T11:26:35.807438Z	`a65dcb1`	New commits
🔒 Security Review	✅ Completed 2026-09-23T13:49:24.444641Z	`eb42f10`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…opping them The sink asked the encoder how many pieces to split an oversized payload into, a count derived from a byte-size ratio, and then partitioned the metrics by count. With unevenly sized metrics one piece could still be over the limit, and `encode_now_or_never` dropped that whole piece even though smaller subdivisions of it would have fit. This is the ordinary overflow path for V3 rather than an edge case: V3 accumulates a whole batch and only learns the payload size at `finish()` time, so unlike V1/V2 it cannot stop before the limit. Replace the single count-based partition with the work queue the Agent's V3 serializer uses (`encode_v3_payload_requests` in saluki): encode a piece, and if the finished payload is still too large, push its two halves back onto the front of the queue and retry. Halving is driven by what the encoder actually produced rather than by an estimate, so it terminates, and the only thing ever dropped is a single metric that is too large to send on its own.

pront

This PR looks great, thanks! Left a few comments but nothing major.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: eb42f105a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…integration tests `smoke` and `all_series_metric_types` build their sink from a config that doesn't set `series_api_version`, and match only the v1 and v2 intake paths. Defaulting to v3 sent every request to their `_ =>` arm, so both panicked with "Unexpected request type received!" on any run. Those two decode `MetricPayload`, which is the v1/v2 wire shape, so pin them to v2 explicitly: that is the protocol they intend to exercise. Cover the new default with a separate test asserting the request reaches `/api/intake/metrics/v3/series` as a non-empty zstd protobuf body. V3's columnar payload has no decoder in this crate, so contents stay verified by the e2e suite, where fakeintake decodes them.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 32a37da7f9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…size limits The payload size limits are inclusive: the intake accepts a payload whose encoded size is exactly the maximum, and the Agent agrees -- `V3PayloadLimits::request_fits` in saluki compares both dimensions with `<=`. Deriving the check from `len / limit + 1` classified equality as an overflow, because integer division yields 1 at exactly the limit and the `+ 1` pushed it past the "no splits needed" case. For a batch of a single metric that is destructive rather than merely wasteful: `split_and_encode` has nothing left to halve, so it dropped a payload the intake would have accepted. Compare the two sizes against their limits directly, and compute the (now only logged) split hint with `div_ceil` so an overflowing payload still reports a sensible number.

github-actions Bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. labels Sep 2, 2026

pront mentioned this pull request Sep 2, 2026

feat(datadog_agent): support v3 series metrics intake #26264

Open

2 of 4 tasks

StephenWakely added 4 commits September 16, 2026 16:28

Merge remote-tracking branch 'origin/master' into stephen/datadog-met…

3799de4

…rics-v3-main # Conflicts: # Cargo.lock # Cargo.toml # src/sinks/datadog/metrics/config.rs # src/sinks/datadog/metrics/encoder.rs

Remove unneeded changes

fdd9998

Add e2e tests for v3

f5b396a

StephenWakely force-pushed the stephen/datadog-metrics-v3 branch from 3e0995c to f5b396a Compare September 18, 2026 13:02

github-actions Bot added the meta: breaking change Anything that breaks backward compatibility. label Sep 18, 2026

chore(datadog_metrics sink): regenerate example config for the v3 ser…

307b776

…ies default The generated advanced example embeds each option's default, so changing `series_api_version`'s default to `v3` left it out of sync and CI's `vdev check component-examples` failed on it.

StephenWakely marked this pull request as ready for review September 21, 2026 16:05

StephenWakely requested review from a team as code owners September 21, 2026 16:05

pront reviewed Sep 22, 2026

View reviewed changes

Comment thread changelog.d/datadog_metrics_series_v3_default.breaking.md

pront reviewed Sep 22, 2026

View reviewed changes

Comment thread tests/e2e/datadog-metrics/config/compose.yaml Outdated

Comment thread tests/e2e/datadog-metrics/config/test.yaml Outdated

pront reviewed Sep 22, 2026

View reviewed changes

Remove extraneous comments

c1dd54f

StephenWakely requested a review from pront September 23, 2026 13:05

docs(datadog_metrics): clarify V3 receiver upgrade order

eb42f10

pront approved these changes Sep 23, 2026

View reviewed changes

pront enabled auto-merge September 23, 2026 13:47

github-actions Bot removed the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Sep 23, 2026

chatgpt-codex-connector Bot reviewed Sep 23, 2026

View reviewed changes

Comment thread src/sinks/datadog/metrics/config.rs

Comment thread src/sinks/datadog/metrics/encoder_v3.rs

pront disabled auto-merge September 23, 2026 14:05

git-thuerk-done approved these changes Sep 23, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 24, 2026

View reviewed changes

Comment thread src/sinks/datadog/metrics/encoder_v3.rs Outdated

StephenWakely added 2 commits September 24, 2026 12:22

Remove extraneous comments

b4c8a9c

pront added this pull request to the merge queue Sep 24, 2026

Merged via the queue into master with commit 830b075 Sep 24, 2026
61 checks passed

pront deleted the stephen/datadog-metrics-v3 branch September 24, 2026 14:29

github-actions Bot locked and limited conversation to collaborators Sep 24, 2026

Navigation Menu

Repository navigation

enhancement(datadog metrics sink): implement V3 - #26285

enhancement(datadog metrics sink): implement V3#26285
pront merged 11 commits into
masterfrom
stephen/datadog-metrics-v3

StephenWakely commented Sep 2, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pront left a comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

StephenWakely commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

References

Vector configuration

How did you test this PR?

Is this a breaking change?

Does this PR include user facing changes?

Contributor Guidelines

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pront left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

StephenWakely commented Sep 2, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading