Skip to content

[enterprise-4.14] OBSDOCS-1676: Tempo-OTel Multitenancy documentation improvements #92484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
179 changes: 0 additions & 179 deletions modules/distr-tracing-tempo-config-multitenancy.adoc

This file was deleted.

82 changes: 46 additions & 36 deletions modules/distr-tracing-tempo-config-spanmetrics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,27 +2,25 @@
//
// * observability/distr_tracing/distr_tracing_tempo/distr-tracing-tempo-configuring.adoc

:_mod-docs-content-type: REFERENCE
:_mod-docs-content-type: PROCEDURE
[id="distr-tracing-tempo-config-spanmetrics_{context}"]
= Configuration of the monitor tab in Jaeger UI
= Configuring the Monitor tab in Jaeger UI

Trace data contains rich information, and the data is normalized across instrumented languages and frameworks.
Therefore, request rate, error, and duration (RED) metrics can be extracted from traces.
The metrics can be visualized in Jaeger console in the *Monitor* tab.
You can have the request rate, error, and duration (RED) metrics extracted from traces and visualized through the Jaeger Console in the *Monitor* tab of the {product-title} web console. The metrics are derived from spans in the OpenTelemetry Collector that are scraped from the Collector by Prometheus, which you can deploy in your user-workload monitoring stack. The Jaeger UI queries these metrics from the Prometheus endpoint and visualizes them.

The metrics are derived from spans in the OpenTelemetry Collector that are scraped from the Collector by the Prometheus deployed in the user-workload monitoring stack.
The Jaeger UI queries these metrics from the Prometheus endpoint and visualizes them.
.Prerequisites

[id="distr-tracing-tempo-config-spanmetrics_opentelemetry-collector-configuration_{context}"]
== OpenTelemetry Collector configuration
* You have configured the permissions and tenants for the {TempoShortName}. For more information, see "Configuring the permissions and tenants".

The OpenTelemetry Collector requires configuration of the `spanmetrics` connector that derives metrics from traces and exports the metrics in the Prometheus format.
.Procedure

.OpenTelemetry Collector custom resource for span RED
. In the `OpenTelemetryCollector` custom resource of the OpenTelemetry Collector, enable the Spanmetrics Connector (`spanmetrics`), which derives metrics from traces and exports the metrics in the Prometheus format.
+
.Example `OpenTelemetryCollector` custom resource for span RED
[source,yaml]
----
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
apiVersion: opentelemetry.io/v1alpha1
metadata:
name: otel
spec:
Expand All @@ -46,35 +44,46 @@ spec:
endpoint: 0.0.0.0:8889
add_metric_suffixes: false
resource_to_telemetry_conversion:
enabled: true # by default resource attributes are dropped
enabled: true # <5>

otlp:
endpoint: "tempo-simplest-distributor:4317"
auth:
authenticator: bearertokenauth
endpoint: tempo-redmetrics-gateway.mynamespace.svc.cluster.local:8090
headers:
X-Scope-OrgID: dev
tls:
insecure: true
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
insecure: false

extensions:
bearertokenauth:
filename: /var/run/secrets/kubernetes.io/serviceaccount/token

service:
extensions:
- bearertokenauth
pipelines:
traces:
receivers: [otlp]
exporters: [otlp, spanmetrics] # <5>
exporters: [otlp, spanmetrics] # <6>
metrics:
receivers: [spanmetrics] # <6>
receivers: [spanmetrics] # <7>
exporters: [prometheus]

# ...
----
<1> Creates the `ServiceMonitor` custom resource to enable scraping of the Prometheus exporter.
<2> The Spanmetrics connector receives traces and exports metrics.
<3> The OTLP receiver to receive spans in the OpenTelemetry protocol.
<4> The Prometheus exporter is used to export metrics in the Prometheus format.
<5> The Spanmetrics connector is configured as exporter in traces pipeline.
<6> The Spanmetrics connector is configured as receiver in metrics pipeline.

[id="distr-tracing-tempo-config-spanmetrics_tempo-configuration_{context}"]
== Tempo configuration

The `TempoStack` custom resource must specify the following: the *Monitor* tab is enabled, and the Prometheus endpoint is set to the Thanos querier service to query the data from the user-defined monitoring stack.
<5> The resource attributes are dropped by default.
<6> The Spanmetrics connector is configured as exporter in traces pipeline.
<7> The Spanmetrics connector is configured as receiver in metrics pipeline.

.TempoStack custom resource with the enabled Monitor tab
. In the `TempoStack` custom resource, enable the *Monitor* tab and set the Prometheus endpoint to the Thanos querier service to query the data from your user-defined monitoring stack.
+
.Example `TempoStack` custom resource with the enabled Monitor tab
[source,yaml]
----
apiVersion: tempo.grafana.com/v1alpha1
Expand All @@ -87,28 +96,29 @@ spec:
name: minio-test
type: s3
storageSize: 1Gi
tenants:
mode: openshift
authentication:
- tenantName: dev
tenantId: "1610b0c3-c509-4592-a256-a1871353dbfa"
template:
gateway:
enabled: false
enabled: true
queryFrontend:
jaegerQuery:
enabled: true
monitorTab:
enabled: true # <1>
prometheusEndpoint: https://thanos-querier.openshift-monitoring.svc.cluster.local:9092 # <2>
redMetricsNamespace: "" <3>
ingress:
type: route

# ...
----
<1> Enables the monitoring tab in the Jaeger console.
<2> The service name for Thanos Querier from user-workload monitoring.
<3> Optional: The metrics namespace on which the Jaeger query retrieves the Prometheus metrics. Include this line only if you are using an OpenTelemetry Collector version earlier than 0.109.0. If you are using an OpenTelemetry Collector version 0.109.0 or later, omit this line.

[id="distr-tracing-tempo-config-spanmetrics_span-red-metrics-and-alerting-rules_{context}"]
== Span RED metrics and alerting rules

The metrics generated by the `spanmetrics` connector are usable with alerting rules. For example, for alerts about a slow service or to define service level objectives (SLOs), the connector creates a `duration_bucket` histogram and the `calls` counter metric. These metrics have labels that identify the service, API name, operation type, and other attributes.

. Optional: Use the span RED metrics generated by the `spanmetrics` connector with alerting rules. For example, for alerts about a slow service or to define service level objectives (SLOs), the connector creates a `duration_bucket` histogram and the `calls` counter metric. These metrics have labels that identify the service, API name, operation type, and other attributes.
+
.Labels of the metrics created in the `spanmetrics` connector
[options="header"]
[cols="a, a, a"]
Expand All @@ -135,8 +145,8 @@ The metrics generated by the `spanmetrics` connector are usable with alerting ru
* `SPAN_KIND_INTERNAL`

|===

.Example `PrometheusRule` CR that defines an alerting rule for SLO when not serving 95% of requests within 2000ms on the front-end service
+
.Example `PrometheusRule` custom resource that defines an alerting rule for SLO when not serving 95% of requests within 2000ms on the front-end service
[source,yaml]
----
apiVersion: monitoring.coreos.com/v1
Expand Down
Loading