Merge pull request #112 from kossakovsky/worktree-fix-issues-110-111

Workflow execution monitoring, worker scrape fix, runner build fix (release 1.10.0)

Author: kossakovsky

.gitignore

@@ -25,3 +25,7 @@ certs/*
 # Custom Caddy addons (user configurations)
 caddy-addon/*.conf
 !caddy-addon/*.example
+
+# Prometheus worker targets (generated by scripts/generate_n8n_workers.sh)
+prometheus/targets/*.json
+!prometheus/targets/.gitkeep

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## [Unreleased]
 
+## [1.10.0] - 2026-09-02
+
+### Added
+- **Monitoring** - The n8n Grafana dashboard now shows whether workflows actually run, not just whether the n8n process is healthy. n8n is started with `N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS`, `N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL`, `N8N_METRICS_INCLUDE_WORKFLOW_NAME_LABEL` and `N8N_METRICS_INCLUDE_WORKFLOW_INFO`, which expose `n8n_workflow_started/success/failed/cancelled_total` counters labelled per workflow plus id-to-name gauges, and a new "Workflow Executions" section adds three panels: executions by outcome over time, executions per workflow, and time since each active workflow's last successful execution. The last one is backed by Prometheus recording rules in `prometheus/rules/n8n-workflows.yml` that remember the last non-manual success per workflow for 30 days and follow renames. Four Grafana-managed alert rules are provisioned from `grafana/provisioning/alerting/n8n-workflows.yml`: "n8n workflow failed" (a non-manual execution failed in the last 15 minutes - runs from the editor are excluded), "n8n workflow stalled" (an active workflow has had no success for 24 hours), "n8n workflow has no recorded success" (active for 24 hours without ever succeeding since monitoring started - catches workflows that were already broken at upgrade time) and "n8n metrics target down" (Prometheus cannot scrape n8n or a worker). The 24-hour thresholds are global and tunable in that file; provisioned rules are read-only in the UI. Alerts follow Grafana's default notification policy, whose built-in email contact point delivers nothing without SMTP - create a contact point and select it under Alerting > Notification policies. The Prometheus data source now has the fixed uid `Prometheus` that the dashboards and rules reference (#110).
+
+### Fixed
+- **Monitoring** - Prometheus never scraped the n8n workers: the `n8n-worker` job targeted a hostname that does not exist (containers are `n8n-worker-1`, `n8n-worker-2`, ...) on the task-broker port 5679 instead of the metrics port 5678. `scripts/generate_n8n_workers.sh` now writes `prometheus/targets/n8n.json` with the n8n main target and one target per worker, read by Prometheus via `file_sd_configs`, so the target list follows `N8N_WORKER_COUNT` automatically. The generator runs on every install and update and removes the file when n8n is deselected, so a monitoring-only install no longer carries a permanently-down `n8n:5678` target. `make doctor` reports a missing or outdated targets file and a failing recording rule (#110).
+- **Doctor** - `make doctor` could never report Grafana or Prometheus as down: the check was gated on a profile named after the container, but both belong to the `monitoring` profile. The check now takes the enabling profile explicitly.
+- **n8n** - `docker compose build` failed on `n8n/Dockerfile.runner` with `/bin/sh: pnpm: Permission denied` (exit code 126). The upstream `n8nio/runners:stable` image rebuilt on 2026-09-02 ships `pnpm.cjs` without the execute bit (pnpm 11.22.0), so the `pnpm add cheerio axios moment lodash` step could not start even as root. The Dockerfile now restores the bit on the symlink target before running pnpm; the fix is a no-op once upstream republishes a correct image (#111).
+
 ## [1.9.0] - 2026-08-27
 
 ### Added

CLAUDE.md

@@ -32,7 +32,7 @@ This is **Selfhost AI** (repository `selfhost-ai`, formerly `n8n-install`), a Do
 - `scripts/06_run_services.sh`: Starts Docker Compose stack
 - `scripts/07_final_report.sh`: Post-install credential summary
 - `scripts/08_fix_permissions.sh`: Fixes file ownership for non-root access
-- `scripts/generate_n8n_workers.sh`: Generates dynamic worker/runner compose file
+- `scripts/generate_n8n_workers.sh`: Generates dynamic worker/runner compose file and the Prometheus n8n targets file
 - `scripts/generate_ollama_instances.sh`: Generates extra Ollama instances compose file (multi-GPU)
 - `scripts/update.sh`: Update orchestrator (syncs with origin and updates images)
 - `scripts/update_preview.sh`: Preview available updates without applying (dry-run)
@@ -184,6 +184,15 @@ This project uses [Semantic Versioning](https://semver.org/). When updating `CHA
 - No published ports and no Caddy block for extra instances - they are internal (`ollama2:11434`); `caddy-addon/site-*.conf` is the documented extension point
 - The generator is invoked unconditionally from `05_configure_services.sh`, which covers install and `make update`, and self-heals a stale file after a hardware-profile switch
 
+### Monitoring (Prometheus + Grafana)
+
+- n8n metrics are enabled in the `x-n8n` anchor: `N8N_METRICS` plus `N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS` / `_WORKFLOW_ID_LABEL` / `_WORKFLOW_NAME_LABEL` / `_WORKFLOW_INFO`. They expose `n8n_workflow_{started,success,failed,cancelled}_total{workflow_id,workflow_name}` and the `n8n_workflow_info` / `n8n_active_workflow_info` id-to-name gauges (leader main only). The alerts and recording rules also use `n8n_workflow_execution_duration_seconds{status,mode,workflow_id}`, which is on by default (`N8N_METRICS_INCLUDE_WORKFLOW_EXECUTION_DURATION`); turning it off silently disables them. Setting `N8N_METRICS_PREFIX` would break every panel and alert
+- **In queue mode the workflow counters and the duration histogram are emitted by `n8n` main**, which finalises every top-level execution; workers only report sub-workflow executions, node events and their own process metrics. Panels and rules therefore aggregate across instances (`sum by (...)`) and must not filter on `$instance`
+- Scrape targets are not static: `generate_n8n_workers.sh` writes `prometheus/targets/n8n.json` (gitignored) with the `n8n:5678` main target and one `n8n-worker-N:5678` per worker, each group carrying its `job` label, and `prometheus.yml` reads it via `file_sd_configs`. The generator runs unconditionally from `05_configure_services.sh` and removes the file when the n8n profile is inactive, so a monitoring-only install has no n8n targets and no permanently-down alert. Metrics are served on the queue-health port (`QUEUE_HEALTH_CHECK_PORT`, default 5678; 5679 is the task broker)
+- `prometheus/rules/*.yml` holds recording rules keyed on `workflow_id` (`n8n:workflow_success:increase5m`, `n8n:workflow_last_success_timestamp_seconds`); names are joined at query time from `n8n_active_workflow_info`, so renames are followed. The whole `prometheus/` directory is bind-mounted read-only to `/etc/prometheus`
+- Grafana's datasource, dashboards and alert rules are file-provisioned from `grafana/provisioning/`; contact points and notification policies are not, and no SMTP is configured. The Prometheus datasource has the fixed `uid: Prometheus` that dashboards and alert rules reference - keep them in sync. Use `$__rate_interval`, never a fixed range window, in range queries
+- No Alertmanager: alert rules are Grafana-managed (`grafana/provisioning/alerting/n8n-workflows.yml`); a malformed file stops Grafana from starting. `make doctor` checks that Grafana/Prometheus run, that the targets file matches `N8N_WORKER_COUNT` and that Prometheus loaded the `n8n-workflows` rule group without errors
+
 ### Caddy Reverse Proxy
 
 - Automatically obtains Let's Encrypt certificates when `LETSENCRYPT_EMAIL` is set

README.md

@@ -28,7 +28,7 @@
 - **AI Agents & RAG** — Flowise, Dify, LangChain, vector databases (Qdrant, Weaviate)
 - **One Command Install** — Interactive wizard, automatic secrets, zero manual configuration
 - **Auto HTTPS** — Caddy reverse proxy with automatic Let's Encrypt certificates
-- **Built-in Monitoring** — Grafana + Prometheus stack included
+- **Built-in Monitoring** — Grafana + Prometheus stack included, with an n8n dashboard that tracks workflow executions (outcomes, per-workflow volume, time since last success) and ready-made alert rules for failed or stalled workflows
 - **Production Ready** — Scalable workers, health checks, proper service dependencies
 - **Free & Open Source** — No vendor lock-in, Apache 2.0 license
 
@@ -239,6 +239,7 @@ After successful installation, your services are up and running! Here's how to g
 
 4.  **Check Monitoring (Optional):**
     - Visit Grafana (`grafana.yourdomain.com`) to see dashboards monitoring your system's performance (data sourced from Prometheus).
+    - The **n8n Monitoring** dashboard includes a *Workflow Executions* section, and four alert rules are pre-provisioned: *n8n workflow failed* (non-manual executions only, so testing in the editor does not page), *n8n workflow stalled* (an active workflow with no success for 24 hours), *n8n workflow has no recorded success* (active for 24 hours without ever succeeding since monitoring started) and *n8n metrics target down*. The 24-hour thresholds are global, so workflows that run less than daily will alert; tune them in `grafana/provisioning/alerting/n8n-workflows.yml`. Alerts follow Grafana's default notification policy, whose built-in email contact point delivers nothing without SMTP: create a contact point (Telegram, Slack, Email with `GF_SMTP_*`, ...) under **Alerting → Contact points** and select it in **Alerting → Notification policies**.
 
 
 ## Secure Access with Cloudflare Tunnel (Optional)

VERSION

@@ -1 +1 @@
-1.9.0
+1.10.0

docker-compose.yml

@@ -82,6 +82,10 @@ x-n8n: &service-n8n
     N8N_LOG_LEVEL: "${N8N_LOG_LEVEL:-info}"
     N8N_LOG_OUTPUT: "${N8N_LOG_OUTPUT:-console}"
     N8N_METRICS: true
+    N8N_METRICS_INCLUDE_MESSAGE_EVENT_BUS_METRICS: true
+    N8N_METRICS_INCLUDE_WORKFLOW_ID_LABEL: true
+    N8N_METRICS_INCLUDE_WORKFLOW_NAME_LABEL: true
+    N8N_METRICS_INCLUDE_WORKFLOW_INFO: true
     N8N_PAYLOAD_SIZE_MAX: "${N8N_PAYLOAD_SIZE_MAX:-256}"
     N8N_PERSONALIZATION_ENABLED: false
     N8N_PROXY_HOPS: "${N8N_PROXY_HOPS:-1}"
@@ -795,7 +799,7 @@ services:
     profiles: ["monitoring"]
     restart: unless-stopped
     volumes:
-      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus:/etc/prometheus:ro
       - prometheus_data:/prometheus
     extra_hosts:
       - "host.docker.internal:host-gateway"