The metric "opentelemetry_allocator_targets" is not accurate

[chenlujjj]

Component(s)

target allocator

What happened?

Description

The metric "opentelemetry_allocator_targets" is not accurate

Steps to Reproduce

Deploy the target allocator within a k8s cluster in which there are otel-collectors and service monitors.

Expected Result

The metric "opentelemetry_allocator_targets" should reflect the accurate number of targets

Actual Result

Seems the metrics value is not equal to number of targets.

For example, in this cluster there is only one target, grafana-operator, but the metric value is 700+

As a contrast, according to metric opentelemetry_allocator_targets_per_collector, the total targets number is 1

Kubernetes Version

1.30.0

Operator version

v0.103.0

Collector version

v0.98.0

Environment information

No response

Log output

No response

Additional context

No response

[swiatekm]

This metric measures the number of targets discovered by the service discovery mechanism, before relabelling. Would that account for the discrepancy?

[chenlujjj]

Let me double check for that

[nicolastakashi]

Another thing I'd like to highlight is the fact you're using Prometheus Operator and since Prometheus Operator is using relabelling to drop targets means if you have two service monitors on a given namespace and this namespace you have 300 targets the discovered targets metrics value will be 600, because every scrape config generated by the operator is discovering the 300 targets and relabelling to keep only the targets selected by the service monitor.