Skip to content

Conversation

@pradhyum6144
Copy link
Contributor

Summary

Fixes #34263

Counter metrics are now correctly typed as Sum even when metadata is unavailable (e.g., when using Target Allocator).

Problem

When using Target Allocator, counter metrics were being incorrectly typed as Gauge instead of Sum when the MetricMetadataStore doesn't contain TYPE information from # TYPE declarations. This caused grouped counter metrics like cnpg_pg_stat_database_blks_read to appear "silently dropped" since they were typed incorrectly.

Solution

Implemented fallback heuristics to detect counter metrics from metric names using Prometheus naming conventions when metadata is unavailable:

Explicit counter suffixes:

  • _total (most reliable)
  • _created
  • _bytes_total
  • _packets_total

Common counter patterns:

  • Operations: _read, _written, _sent, _received, _processed, _completed
  • Events: _requests, _errors, _failures, _hits, _misses, _dropped, _retries

False positive protection:

  • Excludes metrics ending with _ratio or _percent
  • Excludes ambiguous _count suffix (could be histogram or poorly named metric)

Changes

  • metadata.go: Added isLikelyCounter() heuristic function
  • metadata_test.go: Added comprehensive tests for counter detection (20+ patterns)
  • metricfamily.go: Added diagnostic logging when Unknown types default to Gauge
  • grouped_counter_test.go: Test reproducing and verifying the fix for Grouped metric type "counter" silently dropped #34263

…out metadata (open-telemetry#34263)

Use heuristics to detect counters from metric names when metadata unavailable.
@pradhyum6144 pradhyum6144 requested review from a team, ArthurSens and dashpole as code owners February 10, 2026 08:36
@github-actions github-actions bot added the receiver/prometheus Prometheus receiver label Feb 10, 2026
@dashpole
Copy link
Contributor

Why is type information not available when using the target allocator? I don't generally like the idea of using heuristics based on naming conventions.

@dashpole
Copy link
Contributor

Were you able to root-cause the issue?

@pradhyum6144
Copy link
Contributor Author

Why is type information not available when using the target allocator? I don't generally like the idea of using heuristics based on naming conventions.

@dashpole Thank you for the feedback sir!

The Prometheus receiver has PassMetadataInContext: true configured correctly . The scraped data contains # TYPE declarations (confirmed by issue reporter). Something in the Target Allocator flow prevents the MetricMetadataStore from being populated

@pradhyum6144
Copy link
Contributor Author

Were you able to root-cause the issue?

I believe the metadata is lost somewhere in how Target Allocator distributes targets, but I couldn't pinpoint the exact location. The Prometheus scraper should be extracting # TYPE metadata, but it's not reaching the receiver.

@pradhyum6144
Copy link
Contributor Author

@dashpole sir Would you prefer I
. Investigate deeper to find where metadata is lost in Target Allocator integration?

. Close this PR and open an issue for proper root cause analysis?

I agree heuristics aren't ideal. Can you point me to the right place to debug why Target Allocator doesn't preserve metadata?

@dashpole
Copy link
Contributor

The target allocator just tells the prometheus receiver which endpoints to scrape. It shouldn't impact type metadata. We should be able to reproduce it without the target allocator. I would recommend trying to take the scrape output from the issue and put it in a unit test (we have many examples in the receiver). Hopefully that reproduces the issue, and allows you to make a simple unit test reproduction, and you can debug from there.

…al scrape output

This test uses the exact Prometheus scrape output from issue open-telemetry#34263 to
verify that the receiver correctly handles grouped counter metrics.

The test confirms:
- Counter metrics with # TYPE declarations work correctly
- The heuristic fallback handles missing metadata gracefully
- No regression in gauge metric handling
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

receiver/prometheus Prometheus receiver

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Grouped metric type "counter" silently dropped

3 participants