Skip to content

Commit 737c719

Browse files
fdevansfdevans
committed
Monitoring HowTos
1 parent 5fb83d3 commit 737c719

5 files changed

Lines changed: 386 additions & 2 deletions

File tree

‎docs/.vuepress/sidebar-menus/learning.ts‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,8 @@ export default [{
146146
'/learning/howto/config-sn-nodesource.md',
147147
'/learning/howto/configure-gcp-plugins.md',
148148
'/learning/howto/sn-midserver.md',
149+
{link:'/learning/howto/monitor-server-grafana.md',text:'Monitor the Server with Prometheus and Grafana'},
150+
{link:'/learning/howto/monitor-runner-grafana.md',text:'Monitor a Runner with Prometheus and Grafana'},
149151
'/learning/howto/rundeck-exporter.md',
150152
'/learning/howto/vault-integration.md',
151153
'/learning/howto/howtojenkins.md',

‎docs/learning/howto/index.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,9 @@ Available Guides by Section
5151
[Configure ServiceNow CMDB as a Node Source](/learning/howto/config-sn-nodesource.md)
5252
[Configure GCP Integration](/learning/howto/configure-gcp-plugins.md)
5353
[Use a serviceNow MID Server](/learning/howto/sn-midserver.md)
54-
[Monitor Using Prometheus and Grafana](/learning/howto/rundeck-exporter.md)
54+
[Monitor the Server with Prometheus and Grafana](/learning/howto/monitor-server-grafana.md)
55+
[Monitor a Runner with Prometheus and Grafana](/learning/howto/monitor-runner-grafana.md)
56+
[Monitor Using Prometheus and Grafana (legacy exporter)](/learning/howto/rundeck-exporter.md)
5557
[Integrate with Hashicorp Vault](/learning/howto/vault-integration.md)
5658
[Integrate with Jenkins](/learning/howto/howtojenkins.md)
5759
[Manage Kubernetes](/learning/howto/how2kube.md)

‎docs/learning/howto/monitor-runner-grafana.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Monitor a Runner with Prometheus and Grafana
2+
3+
Rundeck Runners expose operation-queue, report-delivery, and JVM metrics that are invaluable for diagnosing the `Runner did not deliver reports in the configured timeout period` error and for capacity planning. Unlike the Rundeck server, a Runner does **not** serve an HTTP metrics endpoint — its metrics are published as JMX MBeans. To get them into Prometheus and Grafana, run the [Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) alongside the Runner.
4+
5+
This guide uses the `jmx_prometheus_javaagent`, which runs in-process and exposes the Runner's metrics on an HTTP port that Prometheus can scrape.
6+
7+
:::warning Validate metric names in your environment
8+
The JMX-to-Prometheus name mapping depends on the exporter rules you use, so the exact Prometheus series names can vary. Treat the names in the example queries below as a starting point and confirm them against your own exporter output (see [Step 4](#step-4-verify-the-exposed-metrics)).
9+
:::
10+
11+
For the full list of Runner metrics and what they mean, see the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md). For a server-side Prometheus + Grafana walkthrough, see [Monitor the Rundeck Server with Prometheus and Grafana](/learning/howto/monitor-server-grafana.md).
12+
13+
## Architecture
14+
15+
```text
16+
Runner JVM (pd-runner.jar)
17+
└─ JMX MBeans ─→ jmx_prometheus_javaagent (:9404) ─→ Prometheus (:9090) ─→ Grafana (:3000)
18+
```
19+
20+
The exporter reads the Runner's MBeans from inside the same JVM and serves them in Prometheus format. No JMX remote port needs to be opened.
21+
22+
## Prerequisites
23+
24+
- A running Rundeck Runner (Replica) version 6.0 or later. Metrics export over JMX is enabled by default.
25+
- Access to the Runner's startup command so you can add a JVM argument.
26+
- A Prometheus + Grafana stack. If you do not already have one, follow [Monitor the Rundeck Server with Prometheus and Grafana](/learning/howto/monitor-server-grafana.md) to stand one up, then add the Runner as an extra scrape target.
27+
28+
## Step 1: Download the JMX Exporter
29+
30+
Download the `jmx_prometheus_javaagent` JAR from Maven Central and place it next to the Runner JAR:
31+
32+
```bash
33+
curl -L -o jmx_prometheus_javaagent.jar \
34+
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/1.0.0/jmx_prometheus_javaagent-1.0.0.jar
35+
```
36+
37+
## Step 2: Create the exporter configuration
38+
39+
Create `jmx-config.yml` next to the Runner JAR. This configuration lowercases names, maps standard JVM MBeans to conventional metric names, exposes the Runner's metrics (published under the `metrics` JMX domain), and includes a catch-all so nothing is silently dropped:
40+
41+
```yaml
42+
startDelaySeconds: 0
43+
ssl: false
44+
lowercaseOutputName: true
45+
lowercaseOutputLabelNames: true
46+
47+
rules:
48+
# JVM heap / non-heap memory
49+
- pattern: "java.lang<type=Memory><>(HeapMemoryUsage|NonHeapMemoryUsage)"
50+
name: jvm_memory_bytes
51+
type: GAUGE
52+
labels:
53+
area: "$1"
54+
55+
# Garbage collection
56+
- pattern: "java.lang<type=GarbageCollector,name=(.+)><>(CollectionTime|CollectionCount)"
57+
name: jvm_gc_$2
58+
type: COUNTER
59+
labels:
60+
gc: "$1"
61+
62+
# Threads
63+
- pattern: "java.lang<type=Threading><>(ThreadCount|PeakThreadCount|TotalStartedThreadCount)"
64+
name: jvm_threads_$1
65+
type: GAUGE
66+
67+
# Runner application metrics (runner.operations.*, runner.reporter.*) published via Micrometer's JMX registry
68+
- pattern: 'metrics<name=(.+)><>([A-Za-z0-9_]+)'
69+
name: runner_$1_$2
70+
help: "Runner metric $1 $2"
71+
type: GAUGE
72+
73+
# Catch-all: expose anything not matched above with auto-generated names
74+
- pattern: ".*"
75+
```
76+
77+
## Step 3: Start the Runner with the exporter attached
78+
79+
Add the `-javaagent` argument to the Runner's startup command. The `9404` below is the port the exporter will serve metrics on:
80+
81+
```bash
82+
java \
83+
-javaagent:./jmx_prometheus_javaagent.jar=9404:./jmx-config.yml \
84+
-jar pd-runner.jar
85+
```
86+
87+
Use your actual Runner JAR name (for example `runner-<uuid>.jar`).
88+
89+
For a Docker-based Runner, mount the exporter JAR and config into the container and append the agent to the Java command (or `JAVA_TOOL_OPTIONS`), and publish port `9404`:
90+
91+
```yaml
92+
runner:
93+
ports:
94+
- "9404:9404"
95+
volumes:
96+
- ./jmx_prometheus_javaagent.jar:/app/jmx_prometheus_javaagent.jar:ro
97+
- ./jmx-config.yml:/app/jmx-config.yml:ro
98+
command: >
99+
java
100+
-javaagent:/app/jmx_prometheus_javaagent.jar=9404:/app/jmx-config.yml
101+
-Drunner.credentials.file=/app/.rdrunner-creds
102+
-jar pd-runner.jar
103+
```
104+
105+
## Step 4: Verify the exposed metrics
106+
107+
With the Runner running, scrape the exporter directly to confirm metrics are being published and to discover the exact series names produced by your rules:
108+
109+
```bash
110+
curl http://localhost:9404/metrics | grep -i runner
111+
```
112+
113+
You should see the Runner's operation and report-delivery metrics, for example series derived from `runner.operations.pool.utilization`, `runner.operations.running`, `runner.reporter.queue.size.total`, and `runner.reporter.max_delivery_delay_seconds`. Note the exact names returned here — you will use them in your Grafana queries. If a metric you expect is missing, widen the rules in `jmx-config.yml` (the catch-all should surface it under an auto-generated name).
114+
115+
## Step 5: Add a Prometheus scrape target
116+
117+
Add the exporter to your `prometheus.yml`:
118+
119+
```yaml
120+
scrape_configs:
121+
- job_name: 'rundeck-runner'
122+
static_configs:
123+
- targets: ['runner:9404']
124+
labels:
125+
service: runner
126+
```
127+
128+
Replace `runner:9404` with the address Prometheus uses to reach the exporter. Reload or restart Prometheus, then confirm the `rundeck-runner` target is **UP** under **Status → Targets**.
129+
130+
## Step 6: Build dashboard panels
131+
132+
In Grafana, add panels using the Prometheus data source. Use the names you confirmed in Step 4 — the queries below assume the example rules from Step 2.
133+
134+
**Operation pool utilization (0–100%):**
135+
136+
```promql
137+
runner_operations_pool_utilization * 100
138+
```
139+
140+
**Running vs. queued operations:**
141+
142+
```promql
143+
runner_operations_running
144+
runner_operations_queued
145+
runner_operations_inflight
146+
runner_operations_pool_capacity
147+
```
148+
149+
**Report backlog and delivery delay (leading timeout indicator):**
150+
151+
```promql
152+
runner_reporter_queue_size_total
153+
runner_reporter_max_delivery_delay_seconds
154+
```
155+
156+
**JVM heap usage:**
157+
158+
```promql
159+
jvm_memory_bytes{area="HeapMemoryUsage"}
160+
```
161+
162+
:::tip Diagnosing report-delivery timeouts
163+
`runner.reporter.max_delivery_delay_seconds` is the earliest warning sign: as it approaches `540` seconds, a server-side timeout (at 600 seconds) is imminent. Pair this Runner dashboard with the server-side `runner_server_report_*` metrics from [Monitor the Rundeck Server with Prometheus and Grafana](/learning/howto/monitor-server-grafana.md#runner-report-delivery-metrics) to see both ends of the pipeline. See the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md#diagnosing-report-delivery-timeouts) for the full diagnostic playbook and suggested alerts.
164+
:::
165+
166+
## Alternative: scrape JMX remotely
167+
168+
If you prefer not to run an in-process agent, you can instead enable JMX remote access on the Runner (see [Status & Monitoring → Monitoring Replicas](/administration/runner/runner-management/monitoring-runners.md#monitoring-replicas)) and run the standalone `jmx_prometheus_httpserver` as a separate process pointed at the Runner's JMX port. The in-process `javaagent` approach in this guide is simpler because it does not require opening a remote JMX port.
169+
170+
## Next steps
171+
172+
- Set [alerting rules](https://prometheus.io/docs/alerting/latest/overview/) on `runner_reporter_max_delivery_delay_seconds` and operation pool utilization.
173+
- If a Runner is regularly saturated, review [Performance tuning for high-throughput Runners](/administration/runner/runner-config.md#performance-tuning-for-high-throughput-runners).

‎docs/learning/howto/monitor-server-grafana.md‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
# Monitor the Rundeck Server with Prometheus and Grafana
2+
3+
Rundeck 6.0 exposes application metrics natively in Prometheus format at the [`/monitoring/prometheus`](/administration/monitoring/index.md) endpoint, which is enabled by default. This means you no longer need a third-party exporter to build a metrics dashboard — Prometheus can scrape Rundeck directly.
4+
5+
This guide walks through a working Prometheus + Grafana stack pointed at a Rundeck server, with example queries for the most useful health and performance metrics.
6+
7+
:::tip Looking for the older exporter guide?
8+
The community `rundeck_exporter` approach is now deprecated for Rundeck 6.0+. See [Monitor a Rundeck Instance Using Prometheus and Grafana (legacy exporter)](/learning/howto/rundeck-exporter.md) only if you are running an older version.
9+
:::
10+
11+
For the underlying endpoint reference (formats, available metrics, configuration, and authentication), see [Monitoring overview](/administration/monitoring/index.md), [Monitoring configuration](/administration/monitoring/configuration.md), and [Using monitoring data](/administration/monitoring/monitoring.md). To monitor Runners as well, see the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md).
12+
13+
## Architecture
14+
15+
```text
16+
Rundeck server Prometheus Grafana
17+
:4440/monitoring/prometheus → :9090 (scrapes) → :3000 (dashboards)
18+
```
19+
20+
Prometheus scrapes the Rundeck endpoint on an interval and stores the time series; Grafana queries Prometheus to render dashboards.
21+
22+
## Prerequisites
23+
24+
- Rundeck 6.0 or later, reachable on its HTTP port (default `4440`).
25+
- Docker and Docker Compose (this guide runs Prometheus and Grafana as containers; you can also install them directly).
26+
- The modern monitoring endpoints enabled (the default). If they return HTTP 404, confirm `rundeck.metrics.enabled=true` and `rundeck.metrics.monitoring.enabled=true` — see [Monitoring configuration](/administration/monitoring/configuration.md).
27+
28+
## Step 1: Confirm Rundeck is exposing metrics
29+
30+
Before wiring up Prometheus, verify the endpoint returns Prometheus-format text:
31+
32+
```bash
33+
curl http://localhost:4440/monitoring/prometheus
34+
```
35+
36+
You should see output beginning with metric definitions such as `# HELP jvm_memory_used_bytes ...`. If you get a 404, the endpoints are disabled — see the configuration reference linked above.
37+
38+
:::warning Secure the endpoint
39+
The `/monitoring/*` endpoints are unauthenticated by default so that scrapers and health checks can reach them. On production deployments, restrict access to these paths (for example at your reverse proxy or network layer) so they are only reachable by your monitoring system. See [Monitoring configuration](/administration/monitoring/configuration.md#security-considerations).
40+
:::
41+
42+
## Step 2: Configure the Prometheus scrape target
43+
44+
Create `prometheus/prometheus.yml`:
45+
46+
```yaml
47+
global:
48+
scrape_interval: 15s
49+
50+
scrape_configs:
51+
- job_name: 'rundeck'
52+
metrics_path: /monitoring/prometheus
53+
static_configs:
54+
- targets: ['rundeck:4440']
55+
labels:
56+
service: rundeck
57+
```
58+
59+
Replace `rundeck:4440` with the address Prometheus should use to reach your server. When Prometheus and Rundeck run in the same Docker network, the service name (`rundeck`) resolves automatically; otherwise use the host and port (for example `rundeck.example.com:4440`).
60+
61+
## Step 3: Provision the Grafana data source
62+
63+
Create `grafana/provisioning/datasources/prometheus.yaml` so Grafana connects to Prometheus automatically on startup:
64+
65+
```yaml
66+
apiVersion: 1
67+
68+
datasources:
69+
- name: Prometheus
70+
uid: prometheus
71+
type: prometheus
72+
access: proxy
73+
url: http://prometheus:9090
74+
isDefault: true
75+
editable: true
76+
```
77+
78+
## Step 4: Provision a dashboards folder (optional)
79+
80+
To load dashboard JSON files from disk automatically, create `grafana/provisioning/dashboards/dashboard.yaml`:
81+
82+
```yaml
83+
apiVersion: 1
84+
85+
providers:
86+
- name: 'rundeck dashboards'
87+
orgId: 1
88+
folder: 'Rundeck'
89+
type: file
90+
disableDeletion: false
91+
updateIntervalSeconds: 10
92+
options:
93+
path: /var/lib/grafana/dashboards
94+
```
95+
96+
Place any exported dashboard JSON files in `grafana/dashboards/`. You can also skip provisioning and build panels directly in the Grafana UI (Step 6), then export them later.
97+
98+
## Step 5: Run the stack
99+
100+
Create `docker-compose.yml`:
101+
102+
```yaml
103+
services:
104+
prometheus:
105+
image: prom/prometheus:latest
106+
container_name: rundeck-prometheus
107+
ports:
108+
- "9090:9090"
109+
volumes:
110+
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
111+
command:
112+
- "--config.file=/etc/prometheus/prometheus.yml"
113+
restart: unless-stopped
114+
115+
grafana:
116+
image: grafana/grafana:latest
117+
container_name: rundeck-grafana
118+
ports:
119+
- "3000:3000"
120+
environment:
121+
GF_SECURITY_ADMIN_USER: admin
122+
GF_SECURITY_ADMIN_PASSWORD: admin
123+
volumes:
124+
- ./grafana/provisioning/:/etc/grafana/provisioning/
125+
- ./grafana/dashboards/:/var/lib/grafana/dashboards/
126+
- grafana-data:/var/lib/grafana
127+
depends_on:
128+
- prometheus
129+
restart: unless-stopped
130+
131+
volumes:
132+
grafana-data:
133+
```
134+
135+
Start it:
136+
137+
```bash
138+
docker compose up -d
139+
```
140+
141+
Then open:
142+
143+
- Prometheus at `http://localhost:9090` — under **Status → Targets**, the `rundeck` target should be **UP**.
144+
- Grafana at `http://localhost:3000` (default login `admin` / `admin`).
145+
146+
:::tip Networking
147+
If your Rundeck server runs outside this Compose project, make sure the Prometheus container can reach it (shared Docker network, or a routable host/IP in the scrape target). For a quick local test against a Rundeck on the Docker host, you can target `host.docker.internal:4440`.
148+
:::
149+
150+
## Step 6: Build dashboard panels
151+
152+
In Grafana, create a dashboard and add panels using the Prometheus data source. The following queries cover the most useful server health signals. Metric names are the native Micrometer names exposed at `/monitoring/prometheus`; use **Status → Targets** in Prometheus or the [`/monitoring/metrics`](/administration/monitoring/monitoring.md) endpoint to discover the full set.
153+
154+
**JVM heap usage:**
155+
156+
```promql
157+
sum(jvm_memory_used_bytes{area="heap"})
158+
sum(jvm_memory_max_bytes{area="heap"})
159+
```
160+
161+
**Live threads:**
162+
163+
```promql
164+
jvm_threads_live_threads
165+
```
166+
167+
**Garbage collection pause (mean over 5m):**
168+
169+
```promql
170+
rate(jvm_gc_pause_seconds_sum[5m]) / rate(jvm_gc_pause_seconds_count[5m])
171+
```
172+
173+
**Process and system CPU:**
174+
175+
```promql
176+
process_cpu_usage
177+
system_cpu_usage
178+
```
179+
180+
**HTTP request rate by status:**
181+
182+
```promql
183+
sum(rate(http_server_requests_seconds_count[5m])) by (status)
184+
```
185+
186+
**HTTP error rate (4xx/5xx):**
187+
188+
```promql
189+
sum(rate(http_server_requests_seconds_count{status=~"[45].."}[5m]))
190+
```
191+
192+
### Runner report-delivery metrics
193+
194+
If you use [Runners](/administration/runner/index.md), the server also publishes report-delivery pipeline metrics (for example `runner_server_report_end_to_end_latency_max_seconds` and `runner_server_report_timeout_count`) on the same `/monitoring/prometheus` endpoint. These are bridged from Rundeck's internal metric registry, so their series names follow a specific naming pattern. See the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md#server-side-metric-names-in-prometheus) for the exact names, units, and suggested alerts.
195+
196+
## Next steps
197+
198+
- Import or build richer dashboards; many community Grafana dashboards exist for JVM/Micrometer applications and can be adapted.
199+
- Add [alerting rules](https://prometheus.io/docs/alerting/latest/overview/) in Prometheus or Grafana for the signals above.
200+
- Use [`/monitoring/health`](/administration/monitoring/monitoring.md#load-balancer-health-checks) for load-balancer health checks.
201+
- Monitor your Runners — see the [Runner Metrics Reference](/administration/runner/runner-management/runner-metrics.md) and [Status & Monitoring](/administration/runner/runner-management/monitoring-runners.md).

0 commit comments

Comments
 (0)