-
Notifications
You must be signed in to change notification settings - Fork 174
feat: Add OTel Collector internal metrics doc & dashboard page #3545
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
ca20cb9
feat: Add OTel Collector internal metrics doc & dashboard page
crazyuploader 5e031be
chore: Address review comments
crazyuploader 840f1bd
chore: Update OpenTelemetry Collector metrics dashboard page
crazyuploader 9e2510b
chore: Update the docs for CMS
crazyuploader eab8be5
chore: Update sideview nav
crazyuploader File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+73.2 KB
.../img/docs/dashboards/dashboard-templates/opentelemetry-collector-dashboard.webp
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
90 changes: 90 additions & 0 deletions
90
data/docs/dashboards/dashboard-templates/opentelemetry-collector-dashboard.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| --- | ||
| date: 2026-06-23 | ||
| title: OpenTelemetry Collector Pipeline Health Dashboard Template | ||
| description: Monitor OTel Collector receiver throughput, processor drops, exporter delivery, queue depth, and process resource usage. | ||
| doc_type: explanation | ||
| --- | ||
|
|
||
| Use this dashboard to monitor your OpenTelemetry Collector instances across receiver throughput, processor drop rates, exporter delivery, queue depth, and process resource usage. | ||
|
|
||
| <Admonition type="note"> | ||
| Before importing this dashboard, send OTel Collector internal metrics to SigNoz. Follow the [Send OTel Collector Metrics](https://signoz.io/docs/metrics-management/opentelemetry-collector-metrics/) guide to configure the built-in OTLP telemetry reader. | ||
| </Admonition> | ||
|
|
||
| <Figure | ||
| src="/img/docs/dashboards/dashboard-templates/opentelemetry-collector-dashboard.webp" | ||
| alt="OpenTelemetry Collector Pipeline Health Dashboard" | ||
| caption="OpenTelemetry Collector Pipeline Health Dashboard" | ||
| /> | ||
|
|
||
| <div className="flex justify-center"> | ||
| <DashboardActions | ||
| dashboardJsonUrl="https://raw.githubusercontent.com/SigNoz/dashboards/refs/heads/main/opentelemetry-collector/opentelemetry-collector-dashboard.json" | ||
| dashboardName="OpenTelemetry Collector" | ||
| /> | ||
| </div> | ||
|
|
||
| ## What This Dashboard Monitors | ||
|
|
||
| - **Overview**: Spans, metric points, and log records received and sent per second at a glance | ||
| - **Receivers**: Accepted and refused/failed signals per second, broken down by receiver | ||
| - **Processors**: Items entering and leaving each processor, batch send size percentiles, and timeout-triggered flushes | ||
| - **Exporters**: Sent and failed signals by exporter, queue size vs. capacity, queue utilization percentage, and in-flight requests | ||
| - **Process Resources**: Heap memory, RSS memory, CPU usage, and memory allocation rate per Collector instance | ||
|
|
||
| ## Metrics Included | ||
|
|
||
| ### Overview | ||
|
|
||
| - **Spans Received /s**: Spans entering the pipeline per second across all receivers | ||
| - **Metric Points Received /s**: Metric data points entering the pipeline per second | ||
| - **Log Records Received /s**: Log records entering the pipeline per second | ||
| - **Spans Sent /s**: Spans delivered to the backend per second across all exporters | ||
| - **Metric Points Sent /s**: Metric data points delivered to the backend per second | ||
| - **Log Records Sent /s**: Log records delivered to the backend per second | ||
|
|
||
| ### Receivers | ||
|
|
||
| - **Accepted Spans /s by Receiver**: Spans entering the pipeline per second, by receiver | ||
| - **Refused & Failed Spans /s by Receiver**: Refused spans from pipeline back-pressure and failed spans from receiver errors, grouped by receiver. Investigate any non-zero value. | ||
| - **Accepted Metric Points /s by Receiver**: Metric points entering the pipeline per second, by receiver | ||
| - **Refused & Failed Metric Points /s by Receiver**: Refused and failed metric points by receiver | ||
| - **Accepted Log Records /s by Receiver**: Log records entering the pipeline per second, by receiver | ||
| - **Refused & Failed Log Records /s by Receiver**: Refused and failed log records by receiver | ||
|
|
||
| ### Processors | ||
|
|
||
| - **Items Incoming /s by Processor**: Signals entering each processor per second | ||
| - **Items Outgoing /s by Processor**: Signals leaving each processor per second. A rate below incoming means the processor is dropping or filtering data. | ||
| - **Batch Send Size (p50/p95/p99)**: Items per batch at three percentiles. A large gap between p50 and p99 points to bursty traffic. | ||
| - **Batch Timeout Trigger Sends /s by Processor**: Batches flushed by timeout per second by processor. A high rate with small batch sizes means the configured batch size is too large for current traffic. | ||
|
|
||
| ### Exporters | ||
|
|
||
| - **Spans Sent /s by Exporter**: Spans delivered to the backend per second, by exporter | ||
| - **Span Send Failures /s by Exporter**: Spans the exporter failed to deliver per second. Investigate any non-zero value. | ||
| - **Metric Points Sent /s by Exporter**: Metric points delivered per second, by exporter | ||
| - **Metric Point Send Failures /s by Exporter**: Metric points the exporter failed to deliver per second | ||
| - **Log Records Sent /s by Exporter**: Log records delivered per second, by exporter | ||
| - **Log Record Send Failures /s by Exporter**: Log records the exporter failed to deliver per second | ||
| - **Exporter Queue Size vs Capacity**: Queue depth vs. capacity per exporter. Size approaching capacity means the exporter cannot keep up with incoming data. | ||
| - **Exporter Queue Utilization %**: Queue fill percentage per exporter. Above 80% the exporter risks dropping data under sustained load. | ||
| - **Exporter In-Flight Requests by Exporter**: Active export requests including retries. High values alongside a slow-draining queue point to backend latency or connectivity issues. | ||
|
|
||
| ### Process Resources | ||
|
|
||
| - **Heap Memory Allocated**: Heap bytes held by live objects per Collector instance. Sustained growth between GC cycles points to a memory leak. | ||
| - **Process RSS Memory**: Physical memory per Collector instance, including Go runtime overhead | ||
| - **CPU Usage (user + system)**: CPU seconds consumed per second per instance. Values near the available core count indicate CPU saturation. | ||
| - **Memory Allocation Rate**: Heap allocation throughput in bytes per second per instance. High rates increase GC pressure and CPU overhead. | ||
|
|
||
| ## Dashboard Variables | ||
|
|
||
| - **service_name**: Filter by Collector service name | ||
|
|
||
| ## Related Dashboards | ||
|
|
||
| - [Host Metrics (VM)](https://signoz.io/docs/dashboards/dashboard-templates/hostmetrics-vm/) | ||
| - [Kubernetes Node Metrics](https://signoz.io/docs/dashboards/dashboard-templates/kubernetes-node-metrics-overall/) | ||
| - [FluxCD](https://signoz.io/docs/dashboards/dashboard-templates/fluxcd-dashboard/) | ||
| - [KEDA](https://signoz.io/docs/dashboards/dashboard-templates/keda/) | ||
175 changes: 175 additions & 0 deletions
175
data/docs/metrics-management/opentelemetry-collector-metrics.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| --- | ||
| date: 2026-06-23 | ||
| title: Monitor OpenTelemetry Collector Pipeline Health with SigNoz | ||
| description: Push OpenTelemetry Collector internal metrics to SigNoz using the built-in OTLP telemetry reader to monitor pipeline health. | ||
| doc_type: howto | ||
| --- | ||
|
|
||
| The OpenTelemetry Collector emits its own pipeline health metrics under the `otelcol_*` prefix. Configure the built-in OTLP telemetry reader to push those metrics to SigNoz. | ||
|
|
||
| <Admonition type="info" title="Using self-hosted SigNoz?" defaultCollapsed="true"> | ||
| Steps are the same. Update the exporter endpoint and remove the ingestion key header or `signozApiKey` value as shown in [Cloud to Self-Hosted](https://signoz.io/docs/ingestion/cloud-vs-self-hosted/#cloud-to-self-hosted). | ||
| </Admonition> | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - OpenTelemetry Collector v0.92 or later | ||
| - An instance of SigNoz ([Cloud](https://signoz.io/teams/) or [Self-Hosted](https://signoz.io/docs/install/self-host/)) | ||
|
|
||
| ## How it works | ||
|
|
||
| The Collector has a built-in telemetry pipeline that reports internal counters and gauges, including received spans, processor drops, exporter queue depth, and process memory. Add an OTLP periodic reader under **service.telemetry.metrics** to send those metrics to SigNoz at a configured interval. | ||
|
|
||
| `level: normal` (the default) covers receiver throughput, exporter queues, batch processor stats, and process resources. `level: detailed` adds per-method HTTP/RPC dimensions on exporter calls. | ||
|
|
||
| ## Steps | ||
|
|
||
| ### Step 1: Add the telemetry reader | ||
|
|
||
| Add a `service.telemetry` block to your Collector config with a periodic OTLP exporter pointing at SigNoz: | ||
|
|
||
| ```yaml:config.yaml | ||
| service: | ||
| telemetry: | ||
| resource: | ||
| service.name: "<service-name>" | ||
| metrics: | ||
| level: normal | ||
| readers: | ||
| - periodic: | ||
| interval: 60000 | ||
| exporter: | ||
| otlp: | ||
| protocol: "http/protobuf" | ||
| endpoint: "https://ingest.<region>.signoz.cloud:443" | ||
| headers: | ||
| signoz-ingestion-key: "<your-ingestion-key>" | ||
| ``` | ||
|
|
||
| Verify these values: | ||
|
|
||
| - `<region>`: Your [SigNoz Cloud region](https://signoz.io/docs/ingestion/signoz-cloud/overview/#endpoint) | ||
| - `<your-ingestion-key>`: Your SigNoz [ingestion key](https://signoz.io/docs/ingestion/signoz-cloud/keys/) | ||
| - `<service-name>`: A unique name for this Collector instance. If you run multiple instances, use different names to distinguish them in the dashboard. | ||
|
|
||
| <Admonition type="note"> | ||
| This config block is independent of your pipeline. You do not need to add a receiver, processor, or pipeline. | ||
| </Admonition> | ||
|
|
||
| ### Step 2: Apply and restart | ||
|
|
||
| <Tabs entityName="environment"> | ||
| <TabItem value="vm" label="VM" default> | ||
|
|
||
| Validate the config and restart the Collector service: | ||
|
|
||
| ```bash | ||
| sudo /usr/bin/otelcol-contrib validate --config /etc/otelcol-contrib/config.yaml | ||
| sudo systemctl restart otelcol-contrib | ||
| ``` | ||
|
|
||
| Tail the logs to confirm the telemetry reader started with no errors: | ||
|
|
||
| ```bash | ||
| sudo journalctl -u otelcol-contrib -f | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| <TabItem value="docker" label="Docker"> | ||
|
|
||
| Restart the Collector container: | ||
|
|
||
| ```bash | ||
| docker compose up -d | ||
| docker logs -f signoz-collection-agent | ||
| ``` | ||
|
|
||
| For `docker run`, stop and rerun the container with the updated config mount. | ||
|
|
||
| </TabItem> | ||
| <TabItem value="k8s" label="Kubernetes"> | ||
|
|
||
| Enable self-telemetry metrics in your `override-values.yaml` for the [SigNoz k8s-infra chart](https://signoz.io/docs/opentelemetry-collection-agents/k8s/k8s-infra/install-k8s-infra/): | ||
|
|
||
| ```yaml:override-values.yaml | ||
| presets: | ||
| selfTelemetry: | ||
| endpoint: https://ingest.<region>.signoz.cloud:443 | ||
| insecure: false | ||
| insecureSkipVerify: false | ||
| signozApiKey: <your-ingestion-key> | ||
| metrics: | ||
| enabled: true | ||
| ``` | ||
|
|
||
| Use this preset for both the DaemonSet collector (`otelAgent`) and the deployment collector (`otelDeployment`). | ||
|
|
||
| If you use the OpenTelemetry Helm chart instead of k8s-infra, add the `service.telemetry` block from Step 1 under `config.service.telemetry` in your chart values. | ||
|
|
||
| Deploy the chart update: | ||
|
|
||
| ```bash | ||
| helm upgrade --install <release-name> signoz/k8s-infra \ | ||
| -n signoz \ | ||
| -f override-values.yaml | ||
| ``` | ||
|
|
||
| Check the Collector pod logs: | ||
|
|
||
| ```bash | ||
| kubectl get pods -n signoz | ||
| kubectl logs -n signoz <daemonset-pod-name> -f | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| ## Validate | ||
|
|
||
| Open [Metrics Explorer](https://signoz.io/docs/metrics-management/metrics-explorer/) in SigNoz and search for `otelcol_`. You should see metrics such as `otelcol_exporter_send_failed_spans`. | ||
|
|
||
| <Figure | ||
| src="/img/docs/metrics/otel-collector/sample-metrics.webp" | ||
| alt="OTel Collector internal metrics in SigNoz Metrics Explorer" | ||
| caption="OTel Collector internal metrics in SigNoz Metrics Explorer" | ||
| /> | ||
|
|
||
| <details> | ||
| <ToggleHeading> | ||
|
|
||
| ## Troubleshooting | ||
| </ToggleHeading> | ||
|
|
||
| ### No `otelcol_` metrics in SigNoz | ||
|
|
||
| Check the Collector logs for export errors: | ||
|
|
||
| ```bash | ||
| sudo journalctl -u otelcol-contrib -f | ||
| ``` | ||
|
|
||
| For Docker, use `docker logs -f signoz-collection-agent`. For Kubernetes, use the Collector pod logs. | ||
|
|
||
| If the logs show authentication errors, confirm the ingestion key and endpoint region are correct. If they show connection errors, confirm the Collector can reach `ingest.<region>.signoz.cloud:443`. | ||
|
|
||
| Wait at least one full `interval` (60 seconds with the config above) after restarting before checking. | ||
|
|
||
| ### Some panels are empty | ||
|
|
||
| Check `service.telemetry.metrics.level`. `basic` omits batch processor metrics. `none` disables telemetry. Use `normal` or `detailed`. | ||
|
|
||
| ### Duplicate metric points | ||
|
|
||
| You can run multiple Collector instances with the same `service.name`. The dashboard groups them by `service.instance.id`, which the Collector assigns per process. All instances appear under the shared service name and can be filtered by instance. | ||
|
|
||
| </details> | ||
|
|
||
| ## Next Steps | ||
|
|
||
| - Import the prebuilt [OpenTelemetry Collector dashboard](https://signoz.io/docs/dashboards/dashboard-templates/opentelemetry-collector-dashboard/). | ||
| - Set up [alerts](https://signoz.io/docs/userguide/alerts-management/). | ||
| - Learn more about the [OpenTelemetry Collector configuration](https://signoz.io/docs/opentelemetry-collection-agents/opentelemetry-collector/configuration/). | ||
|
|
||
| ## Get Help | ||
|
|
||
| <GetHelp /> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.