Skip to content

Commit 5fb83d3

Browse files
fdevansfdevans
committed
Metrics Updates
1 parent ea1de81 commit 5fb83d3

4 files changed

Lines changed: 150 additions & 0 deletions

File tree

‎docs/.vuepress/sidebar-menus/administration.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ export default [{
7171
'/administration/runner/runner-management/node-dispatch.md',
7272
'/administration/runner/runner-management/managing-replicas.md',
7373
'/administration/runner/runner-management/monitoring-runners.md',
74+
{link:'/administration/runner/runner-management/runner-metrics.md',text:"Runner Metrics Reference"},
7475
'/administration/runner/runner-management/runner-logging.md',
7576
'/administration/runner/runner-management/upgrading-runners.md',
7677
{link:'/administration/runner/runner-management/runner-high-availability.md',text:"Runner High Availability"}

‎docs/administration/runner/runner-config.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,10 @@ This error means the server did not receive any status report from the Runner fo
6969

7070
The properties below are the main levers available to increase capacity and avoid this class of error. All of them are JVM system properties — pass them with `-D` on the Runner command line (or via `command:` in Docker Compose).
7171

72+
:::tip
73+
To confirm which bottleneck is causing the timeout before tuning, consult the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md). It documents the operation-queue and report-delivery metrics that show saturation building before the server times out.
74+
:::
75+
7276
### Operation concurrency
7377

7478
The Runner runs each operation in a thread from a fixed-size pool. When the pool is full, additional operations wait in an unbounded queue, and no status reports are emitted for queued operations.

‎docs/administration/runner/runner-management/monitoring-runners.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,3 +79,18 @@ For example, the **Datadog** agent can be configured to monitor the Runners JVM
7979
The JVM metrics will then be associated with the Runner's host in Datadog:
8080

8181
![Datadog Monitoring Runner](/assets/img/datadog-monitoring-runner.png)<br>
82+
83+
## Operation and report-delivery metrics
84+
85+
Rundeck 6.0 adds operation-queue and report-delivery metrics that help diagnose and predict the server-side `Runner did not deliver reports in the configured timeout period` error. On the Replica these are exposed through the **same JMX MBeans** described above — no extra configuration beyond enabling JMX is required.
86+
87+
The Replica does **not** expose an HTTP metrics endpoint and does not bind to a network port for monitoring. To consume the Runner metrics with Prometheus, attach the [`jmx_prometheus_javaagent`](https://github.com/prometheus/jmx_exporter) as a Java agent and point it at a JMX config:
88+
89+
```bash
90+
java -javaagent:/path/to/jmx_prometheus_javaagent.jar=9404:/path/to/jmx-config.yml \
91+
-jar runner-6281cf48-37a2-4659-93c9-907539177022.jar
92+
```
93+
94+
The agent then exposes the JMX MBeans (including JVM and Runner metrics) in Prometheus format on its own port (`9404` in the example above).
95+
96+
The Runner-side metrics are only part of the report-delivery picture; the matching server-side metrics are exposed on the Rundeck server's [`/monitoring/prometheus`](/administration/monitoring/monitoring.md) endpoint. For the full list of metrics across both processes and guidance on interpreting them, see the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md).

‎docs/administration/runner/runner-management/runner-metrics.md‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# Runner Metrics Reference
2+
3+
Rundeck 6.0 adds a focused set of metrics for diagnosing and predicting Runner report-delivery problems — in particular the server-side error:
4+
5+
```text
6+
Failed: IOFailure: Runner did not deliver reports in the configured timeout period
7+
```
8+
9+
This error fires when the server receives no status report for an in-flight operation for a consecutive 10-minute (600-second) window. It usually means the Runner is saturated rather than that the operation itself hung. The metrics below let you see the saturation building **before** the timeout occurs (the 540-second / 9-minute thresholds used in the alerts below fire about a minute ahead), and confirm where the time is being spent once it does.
10+
11+
For the configuration properties used to relieve this saturation, see [Performance tuning for high-throughput Runners](/administration/runner/runner-config.md#performance-tuning-for-high-throughput-runners).
12+
13+
## Where metrics are exposed
14+
15+
The report-delivery pipeline spans two processes, so the metrics are split across two exposure points:
16+
17+
| Source | Process | How it's exposed |
18+
|---|---|---|
19+
| Runner-side metrics (`runner.operations.*`, `runner.reporter.*`) | Runner / Replica | JMX MBeans. The Replica has no HTTP metrics endpoint; for Prometheus, attach the `jmx_prometheus_javaagent`. See [Operation and report-delivery metrics](/administration/runner/runner-management/monitoring-runners.md#operation-and-report-delivery-metrics). |
20+
| Server-side metrics (`runner.server.*`, `runner.report.*`) | Rundeck server | The server's modern monitoring endpoint `/monitoring/prometheus`. See [Using monitoring data](/administration/monitoring/monitoring.md). |
21+
22+
:::tip Metric names
23+
Metric names in this page use the dot-separated base form (`runner.operations.running`). The exact series name in your monitoring tool depends on how each metric is exposed — see [Server-side metric names in Prometheus](#server-side-metric-names-in-prometheus) below and the runner-side note about `jmx_prometheus_javaagent` mappings.
24+
:::
25+
26+
## Runner-side metrics
27+
28+
These are emitted by the Runner (Replica) process.
29+
30+
### Operation queue
31+
32+
The Runner executes each operation in a thread from a fixed-size pool. When the pool is full, additional operations wait in an internal queue and emit no status reports until they start.
33+
34+
| Metric | Type | Description |
35+
|---|---|---|
36+
| `runner.operations.running` | Gauge | Operations currently executing on a worker thread. |
37+
| `runner.operations.queued` | Gauge | Operations waiting for a worker thread (executor internal queue). |
38+
| `runner.operations.inflight` | Gauge | Total tracked operations (queued + running + pending state-handler cleanup). |
39+
| `runner.operations.pool.capacity` | Gauge | Maximum worker threads in the pool (set by `runner.operations.maxRunning`). |
40+
| `runner.operations.pool.utilization` | Gauge | `running / capacity`, ranging from `0.0` to `1.0`. |
41+
| `runner.operations.completed_total` | Gauge | Monotonic total of completed operations. |
42+
| `runner.operations.queue_wait_time` | Timer | Time each operation waited in the queue before it began executing. |
43+
44+
### Report delivery
45+
46+
The Runner batches status reports in memory and flushes them to the server on a fixed interval. If the backlog grows faster than it is sent, the server eventually times out waiting for a batch.
47+
48+
| Metric | Type | Description |
49+
|---|---|---|
50+
| `runner.reporter.max_delivery_delay_seconds` | Gauge | **Leading indicator.** Age of the oldest pending report across all active queues. As this approaches `540` a server-side timeout is imminent. |
51+
| `runner.reporter.queue.size.total` | Gauge | Total reports waiting across all operation queues (global backlog). |
52+
| `runner.reporter.batch.size` | Summary | Distribution of batch sizes actually sent. |
53+
54+
## Server-side metrics
55+
56+
These are emitted by the Rundeck server and measure the full delivery pipeline as the server experiences it. They are registered in Rundeck's internal (Dropwizard) metric registry and bridged into Micrometer, so they are available from the modern `/monitoring/prometheus` endpoint that is enabled by default in Rundeck 6.0. (They also appear on the legacy `/metrics/*` endpoints, which are disabled by default in 6.0 — see [Monitoring configuration](/administration/monitoring/configuration.md#legacy-endpoints).)
57+
58+
### Timers
59+
60+
Timers expose a distribution (mean, p50, p95, p99, count).
61+
62+
| Metric | Description |
63+
|---|---|
64+
| `runner.server.report.end_to_end_latency` | Full journey: report creation on the Runner to handler invocation on the server. Covers Runner batching, HTTP transit, and the server-side queue wait. |
65+
| `runner.server.report.runner_network_delay` | Phase A only: report creation on the Runner to receipt by the server. Covers Runner batching and HTTP transit. |
66+
| `runner.report.batch.processing_time` | Batch lifecycle in cluster mode: database poll to delivery. |
67+
68+
### Gauges
69+
70+
| Metric | Description |
71+
|---|---|
72+
| `runner.server.report.end_to_end_latency.max_seconds` | Worst current end-to-end latency across all active operations. Predicts imminent timeouts. |
73+
| `runner.operations.total_queue_size` | Total reports waiting across all server-side processor queues. |
74+
| `runner.operations.active_count` | Total operations currently tracked by the service. |
75+
76+
### Counter
77+
78+
| Metric | Description |
79+
|---|---|
80+
| `runner.server.report.timeout.count` | Hard timeouts surfaced to the user. Each increment corresponds to one "Runner did not deliver reports" failure. |
81+
82+
### Histogram
83+
84+
| Metric | Description |
85+
|---|---|
86+
| `runner.report.batch.size` | Distribution of batch sizes processed in cluster mode. |
87+
88+
### Server-side metric names in Prometheus
89+
90+
The Dropwizard-to-Micrometer bridge expands each metric type into one or more Prometheus series, and dots in the base name become underscores. Knowing this is necessary to write working queries:
91+
92+
| Source type | Prometheus series | Notes |
93+
|---|---|---|
94+
| Gauge | `<name>` | For example `runner_server_report_end_to_end_latency_max_seconds` (value in **seconds**). |
95+
| Counter | `<name>` | Exposed as a plain gauge of the current count — **no `_total` suffix**. For example `runner_server_report_timeout_count`. |
96+
| Timer | `<name>_count`, `<name>_mean`, `<name>_50thpercentile`, `<name>_95thpercentile`, `<name>_99thpercentile` | Percentile/mean values are in **milliseconds**. For example `runner_server_report_end_to_end_latency_95thpercentile`. |
97+
| Histogram | `<name>_count`, `<name>_mean`, `<name>_50thpercentile`, `<name>_95thpercentile`, `<name>_99thpercentile` | Raw value distribution (for example batch sizes). |
98+
99+
## Diagnosing report-delivery timeouts
100+
101+
Use the metrics together to answer three questions:
102+
103+
1. **Is a timeout about to happen?** Watch `runner.reporter.max_delivery_delay_seconds` (Runner) and `runner.server.report.end_to_end_latency.max_seconds` (server). Both track the worst current latency; as either approaches `540` seconds a timeout is imminent.
104+
2. **How often are timeouts actually happening?** `runner.server.report.timeout.count` increments once per user-visible failure.
105+
3. **Where is the time being spent?** Compare `runner.server.report.runner_network_delay` (Runner batching + HTTP) against `runner.server.report.end_to_end_latency` (the full path). A large gap between them points to a backed-up server-side queue; a high `runner_network_delay` with healthy server queues points to a saturated Runner or network.
106+
107+
To distinguish the two Runner-side bottlenecks:
108+
109+
- **Operation queue saturation** — `runner.operations.pool.utilization` at `1.0` with a rising `runner.operations.queue_wait_time` means operations are waiting for a worker thread and have not started reporting yet. Increase [`runner.operations.maxRunning`](/administration/runner/runner-config.md#runner-operations-maxrunning).
110+
- **Report backlog** — a rising `runner.reporter.queue.size.total` and `runner.reporter.max_delivery_delay_seconds` while operations are running means reports are produced faster than they are flushed. Tune [report delivery](/administration/runner/runner-config.md#report-delivery).
111+
112+
## Suggested alerts
113+
114+
Treat the thresholds below as starting points and adjust for your workload.
115+
116+
The **server-side** examples use the series produced by `/monitoring/prometheus` (see [Server-side metric names in Prometheus](#server-side-metric-names-in-prometheus)):
117+
118+
```text
119+
# Timeout imminent — worst current end-to-end latency (gauge, seconds)
120+
runner_server_report_end_to_end_latency_max_seconds > 540 # 9 min: timeout in < 1 min
121+
runner_server_report_end_to_end_latency_max_seconds > 480 # 8 min: warning
122+
123+
# Actual user-visible failures. The counter is bridged as a gauge, so use delta() (not increase()).
124+
delta(runner_server_report_timeout_count[5m]) > 0
125+
```
126+
127+
The **runner-side** indicators are equally useful, but their Prometheus series names depend on how your `jmx_prometheus_javaagent` configuration maps the JMX MBeans. Express these against whatever names your mapping produces:
128+
129+
- `runner.reporter.max_delivery_delay_seconds` (seconds) crossing `540` (critical) or `480` (warning) — the earliest leading indicator, measured on the Runner itself.
130+
- `runner.operations.pool.utilization` sustained at `1.0` together with a rising `runner.operations.queue_wait_time` — the operation queue is saturated and work is waiting to start.

0 commit comments

Comments
 (0)