Delta temporality is causing issues in Grafana Mimir

[musq]

• Is there a way to switch to Cumulative temporality in the SDK itself?
• I have even tried the experimental otelcol.processor.deltatocumulative component in Grafana Alloy, but it doesn't actually do any cumulation, and just renames the temporality from Delta to Cumulative 🤦

---

When I try to send custom metrics via my pipeline (Laravel App -> Grafana Alloy -> Grafana Mimir), Grafana Alloy throws this error:

ts=2025-09-19T14:10:35.164886803Z level=error msg="Exporting failed. Dropping data." component_path=/ component_id=otelcol.exporter.otlphttp.mimir error="not retryable error: Permanent error: rpc error: code = InvalidArgument desc = error exporting items, request to http://mimir:9009/otlp/v1/metrics responded with HTTP Status Code 400, Message=otlp parse error: invalid temporality and type combination for metric \"server_api_latency\"; invalid temporality and type combination for metric \"server_api_errors_count\"; invalid temporality and type combination for metric \"my-meter\", Details=[]" dropped_items=3

My OTLP Payload:

{
    "resource": {
        "attributes": {
            "service.name": "xyz",
            "host.name": "Ashishs-MacBook-Air.local",
            "host.arch": "arm64",
            "host.id": "C71EB3AF-3A47-5ECA-A62A-7A48F60B21F3",
            "os.type": "darwin",
            "os.description": "24.6.0",
            "os.name": "Darwin",
            "os.version": "Darwin Kernel Version 24.6.0: Mon Jul 14 11:29:54 PDT 2025; root:xnu-11417.140.69~1\/RELEASE_ARM64_T8122",
            "process.runtime.name": "cli-server",
            "process.runtime.version": "8.2.29",
            "process.pid": 88678,
            "process.executable.path": "\/Users\/ashish\/Library\/Application Support\/Herd\/bin\/php82",
            "process.owner": "ashish",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.language": "php",
            "telemetry.sdk.version": "1.7.1",
            "service.instance.id": "27322762-7652-435f-9358-dde27cb0090c"
        },
        "dropped_attributes_count": 0
    },
    "scope": {
        "name": "laravel-opentelemetry",
        "version": "1.11.0",
        "attributes": [],
        "dropped_attributes_count": 0,
        "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.32.0",
        "metrics": [
            {
                "name": "server_api_latency",
                "description": "API latency tracking meter",
                "unit": "ms",
                "data": {
                    "dataPoints": [
                        {
                            "count": 1,
                            "sum": 60.710291,
                            "min": 60.710291,
                            "max": 60.710291,
                            "bucketCounts": [
                                0,
                                0,
                                0,
                                0,
                                0,
                                1,
                                0,
                                0,
                                0,
                                0,
                                0
                            ],
                            "explicitBounds": [
                                0,
                                5,
                                10,
                                25,
                                50,
                                75,
                                100,
                                250,
                                500,
                                1000
                            ],
                            "attributes": {
                                "route": "\/api\/me",
                                "status_code": 200
                            },
                            "startTimestamp": 1758291366977506043,
                            "timestamp": 1758291367038475918,
                            "exemplars": [
                                {
                                    "value": 60.710291,
                                    "timestamp": 1758291367038033251,
                                    "attributes": [],
                                    "traceId": "a92b2d1e4344578965a5d58ce3a7c9ca",
                                    "spanId": "5fe66f8c53fedd98"
                                }
                            ]
                        }
                    ],
                    "temporality": "Delta"
                }
            },
            {
                "name": "server_api_errors_count",
                "description": "API errors counting meter",
                "unit": "times",
                "data": {
                    "dataPoints": [
                        {
                            "value": 1,
                            "attributes": {
                                "route": "\/api\/me",
                                "status_code": 200
                            },
                            "startTimestamp": 1758291366977506043,
                            "timestamp": 1758291367038475918,
                            "exemplars": [
                                {
                                    "value": 1,
                                    "timestamp": 1758291367038063043,
                                    "attributes": [],
                                    "traceId": "a92b2d1e4344578965a5d58ce3a7c9ca",
                                    "spanId": "5fe66f8c53fedd98"
                                }
                            ]
                        }
                    ],
                    "temporality": "Delta",
                    "monotonic": true
                }
            },
            {
                "name": "my-meter",
                "description": "my custom meter",
                "unit": "times",
                "data": {
                    "dataPoints": [
                        {
                            "value": 1,
                            "attributes": [],
                            "startTimestamp": 1758291366977506043,
                            "timestamp": 1758291367038475918,
                            "exemplars": [
                                {
                                    "value": 1,
                                    "timestamp": 1758291367038121376,
                                    "attributes": [],
                                    "traceId": "a92b2d1e4344578965a5d58ce3a7c9ca",
                                    "spanId": "5fe66f8c53fedd98"
                                }
                            ]
                        }
                    ],
                    "temporality": "Delta",
                    "monotonic": true
                }
            }
        ]
    }
}
{
    "resource": {
        "attributes": {
            "service.name": "xyz",
            "host.name": "Ashishs-MacBook-Air.local",
            "host.arch": "arm64",
            "host.id": "C71EB3AF-3A47-5ECA-A62A-7A48F60B21F3",
            "os.type": "darwin",
            "os.description": "24.6.0",
            "os.name": "Darwin",
            "os.version": "Darwin Kernel Version 24.6.0: Mon Jul 14 11:29:54 PDT 2025; root:xnu-11417.140.69~1\/RELEASE_ARM64_T8122",
            "process.runtime.name": "cli-server",
            "process.runtime.version": "8.2.29",
            "process.pid": 88678,
            "process.executable.path": "\/Users\/ashish\/Library\/Application Support\/Herd\/bin\/php82",
            "process.owner": "ashish",
            "telemetry.sdk.name": "opentelemetry",
            "telemetry.sdk.language": "php",
            "telemetry.sdk.version": "1.7.1",
            "service.instance.id": "27322762-7652-435f-9358-dde27cb0090c"
        },
        "dropped_attributes_count": 0
    },
    "scope": {
        "name": "laravel-opentelemetry",
        "version": "1.11.0",
        "attributes": [],
        "dropped_attributes_count": 0,
        "schema_url": "https:\/\/opentelemetry.io\/schemas\/1.32.0",
        "metrics": [
            {
                "name": "server_api_latency",
                "description": "API latency tracking meter",
                "unit": "ms",
                "data": {
                    "dataPoints": [],
                    "temporality": "Delta"
                }
            },
            {
                "name": "server_api_errors_count",
                "description": "API errors counting meter",
                "unit": "times",
                "data": {
                    "dataPoints": [],
                    "temporality": "Delta",
                    "monotonic": true
                }
            },
            {
                "name": "my-meter",
                "description": "my custom meter",
                "unit": "times",
                "data": {
                    "dataPoints": [],
                    "temporality": "Delta",
                    "monotonic": true
                }
            }
        ]
    }
}

[cappuc]

Hi @musq,
support for metrics has been recently added and I'm not too familiar with it.
What meter functions are you using? Can you share a simple snippet to reproduce the issue?

[musq]

I figured this out and was able to fix it.

PROBLEM

The root of the problem is that opentelemetry-php SDK does not respect OTEL_RESOURCE_ATTRIBUTES=service.instance.id=xxx environment variable and always generates a random service.instance.id on each Laravel request.

This behavior results in each metric being a brand new timeseries. So, the otelcol.processor.deltatocumulative component of Grafana Alloy isn't able to run its logic of cumulation because each metrics is brand new, and has no history.

SOLUTION

Since I wasn't able to override the hardcoded logic to generate random service.instance.id via code, I used a Grafana Alloy component to override the resource attribute as follows.

otelcol.processor.resourcedetection "default" {
    // The env detector reads resource information from the
    // OTEL_RESOURCE_ATTRIBUTES environment variable.
    // This variable must be in the format <key1>=<value1>,<key2>=<value2>,...
    detectors = ["env"]

    output {
        metrics = [otelcol.processor.deltatocumulative.default.input]
    }
}

// Convert delta temporality metrics to cumulative temporality
// https://grafana.com/blog/2023/09/26/opentelemetry-metrics-a-guide-to-delta-vs.-cumulative-temporality-trade-offs/
otelcol.processor.deltatocumulative "default" {
    output {
        metrics = [otelcol.exporter.otlphttp.mimir.input]
    }
}

This has worked for me, and the otelcol.processor.deltatocumulative component is now able to cumulate the metrics accurately.

NOTE

This approach has only worked for me because I'm only using one Laravel app instance to send metrics. However if you have multiple instances of your Laravel app, this approach will not work. That's because the metrics from all your app instances would have the same labels, and they would confuse the otelcol.processor.deltatocumulative processor and it'd report less metrics count than actual.

The real solution is for opentelemetry-php to fix it's implementation so that it respects OTEL_RESOURCE_ATTRIBUTES environment variable. That way you can deploy your multiple Laravel app instances with different instance ids, e.g.

• OTEL_RESOURCE_ATTRIBUTES=service.instance.id=123
• OTEL_RESOURCE_ATTRIBUTES=service.instance.id=456

Or, if you can find a way to monkey-patch this class during Laravel boot, and make it respect OTEL_RESOURCE_ATTRIBUTES env var, that can also work.

THANKS

Thanks @cappuc for being active on the issues in this repo. Your prompt response gave me the courage to dig deeper 🙇

[musq]

For anyone wondering about my Grafana LGTM stack setup, you can find it here: https://github.com/musq/comcon/tree/dev

[cappuc]

Hi @musq,

I tagged 1.12 with:

• exporters.otlp.metrics_temporality config which allows to force the output temporality in the otlp exporter
• service_instance_id config which allows to manually set the service.instance.id attribute

[musq]

Thanks @cappuc! I have been able to verify than 1.12 has solved my issue via the new env variable OTEL_SERVICE_INSTANCE_ID.

[musq]

Update from opentelemetry-php

opentelemetry-php v1.9.0 package has now replaced the hardcoded service_instance_id implementation with an optional one.

PR: open-telemetry/opentelemetry-php#1644

Now, we can specify service_instance_id directly via OTEL_RESOURCE_ATTRIBUTES=service.instance.id=1 without relying on the custom env var OTEL_SERVICE_INSTANCE_ID in laravel-opentelemetry.

To use this, just update your opentelemetry-php version to 1.9.0.