Skip to content

Commit d608f52

Browse files
kossakovskyclaude
andcommitted
fix: recreate Grafana Prometheus data source so pinned uid survives upgrades
Upgrading an existing installation to 1.10.0 left Grafana in a restart loop: "Datasource provisioning error: data source not found". The data source already existed with a random uid, and Grafana looks an existing data source up by id AND uid when provisioning updates it, so pinning uid: Prometheus could never match. Delete it by name before recreating it; dashboards and alert rules reference the uid, so nothing is lost. Release 1.10.1. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P9yjkBNtrs5twjnMK2cDFx
1 parent 2a1d4cf commit d608f52

4 files changed

Lines changed: 17 additions & 2 deletions

File tree

CHANGELOG.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,11 @@
22

33
## [Unreleased]
44

5+
## [1.10.1] - 2026-09-02
6+
7+
### Fixed
8+
- **Monitoring** - Upgrading an existing installation to 1.10.0 left Grafana in a restart loop with `Datasource provisioning error: data source not found`. The Prometheus data source already existed with a random uid, and Grafana cannot change the uid of an existing data source when provisioning pins one. The provisioning file now deletes the data source by name before recreating it with the fixed uid `Prometheus`; dashboards and alert rules reference the uid, so nothing is lost. Fresh installations were not affected.
9+
510
## [1.10.0] - 2026-09-02
611

712
### Added

CLAUDE.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -190,7 +190,7 @@ This project uses [Semantic Versioning](https://semver.org/). When updating `CHA
190190
- **In queue mode the workflow counters and the duration histogram are emitted by `n8n` main**, which finalises every top-level execution; workers only report sub-workflow executions, node events and their own process metrics. Panels and rules therefore aggregate across instances (`sum by (...)`) and must not filter on `$instance`
191191
- Scrape targets are not static: `generate_n8n_workers.sh` writes `prometheus/targets/n8n.json` (gitignored) with the `n8n:5678` main target and one `n8n-worker-N:5678` per worker, each group carrying its `job` label, and `prometheus.yml` reads it via `file_sd_configs`. The generator runs unconditionally from `05_configure_services.sh` and removes the file when the n8n profile is inactive, so a monitoring-only install has no n8n targets and no permanently-down alert. Metrics are served on the queue-health port (`QUEUE_HEALTH_CHECK_PORT`, default 5678; 5679 is the task broker)
192192
- `prometheus/rules/*.yml` holds recording rules keyed on `workflow_id` (`n8n:workflow_success:increase5m`, `n8n:workflow_last_success_timestamp_seconds`); names are joined at query time from `n8n_active_workflow_info`, so renames are followed. The whole `prometheus/` directory is bind-mounted read-only to `/etc/prometheus`
193-
- Grafana's datasource, dashboards and alert rules are file-provisioned from `grafana/provisioning/`; contact points and notification policies are not, and no SMTP is configured. The Prometheus datasource has the fixed `uid: Prometheus` that dashboards and alert rules reference - keep them in sync. Use `$__rate_interval`, never a fixed range window, in range queries
193+
- Grafana's datasource, dashboards and alert rules are file-provisioned from `grafana/provisioning/`; contact points and notification policies are not, and no SMTP is configured. The Prometheus datasource has the fixed `uid: Prometheus` that dashboards and alert rules reference - keep them in sync. Grafana cannot change the uid of an existing datasource, so `main.yml` deletes it by name (`deleteDatasources`) and recreates it on every start; pre-1.10.0 installations had a random uid and crash-looped without this. Use `$__rate_interval`, never a fixed range window, in range queries
194194
- No Alertmanager: alert rules are Grafana-managed (`grafana/provisioning/alerting/n8n-workflows.yml`); a malformed file stops Grafana from starting. `make doctor` checks that Grafana/Prometheus run, that the targets file matches `N8N_WORKER_COUNT` and that Prometheus loaded the `n8n-workflows` rule group without errors
195195

196196
### Caddy Reverse Proxy

VERSION

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
1.10.0
1+
1.10.1

grafana/provisioning/datasources/main.yml

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,15 @@
11
apiVersion: 1
22

3+
# Installations older than 1.10.0 hold a "Prometheus" data source with a random
4+
# uid. Grafana cannot change the uid of an existing data source (it looks the row
5+
# up by id AND uid, fails with "data source not found" and refuses to start), so
6+
# the data source is dropped by name and recreated with the fixed uid on every
7+
# start. Dashboards and alert rules reference the uid, not the id, so nothing is
8+
# lost. On a fresh install the delete is a no-op.
9+
deleteDatasources:
10+
- name: Prometheus
11+
orgId: 1
12+
313
datasources:
414
- name: Prometheus
515
uid: Prometheus

0 commit comments

Comments
 (0)