monitoring: add Cluster Nodes dashboard (per-node up/down via up{job=kubernetes-nodes})

node-exporter metrics vanish when a node is down (just one fewer line); this
dashboard uses the kubelet-scrape up metric, which persists as 0 for an
unreachable node, so a down node shows red instead of disappearing. Verified:
cirrus=UP, thelio=DOWN.

Author: cboettig

monitoring/README.md

@@ -64,6 +64,9 @@ the chart's insecure default — this is a public ingress). Retrieve it:
   **tuned to this cluster's metric labels so they show data** — the community
   dashboards 1860 (node) / 12239 (DCGM) assume different `job` labels and render
   No Data:
+  - `grafana-dashboard-cluster.yaml` — **Cluster Nodes**: per-node UP/DOWN via
+    `up{job="kubernetes-nodes"}` (stays red when a node is unreachable, unlike
+    node-exporter metrics that just vanish) + up/down history.
   - `grafana-dashboard-smart.yaml` — **Drive Health (SMART)**: NVMe wear %, temp,
     spare, errors, SATA attributes.
   - `grafana-dashboard-node.yaml` — **Node Host Health**: CPU, memory, load,

monitoring/grafana-dashboard-cluster.yaml

@@ -0,0 +1,68 @@
+# Cluster Nodes dashboard (as-code) — at-a-glance node up/down. Uses
+# `up{job="kubernetes-nodes"}` (the kubelet scrape): it reports 1/0 PER NODE and,
+# unlike node-exporter metrics, the target PERSISTS as 0 when a node is
+# unreachable (nodes come from k8s service discovery), so a down node shows red
+# rather than just disappearing. Auto-loaded by the Grafana sidecar.
+#
+# Note: this reflects kubelet reachability. A node that's up but cordoned still
+# reads UP here (kubelet is fine) — distinguishing Ready/cordoned/pressure needs
+# kube-state-metrics (currently disabled in prometheus-values.yaml).
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-cluster
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+  annotations:
+    grafana_folder: "Hosts"
+data:
+  cluster.json: |
+    {
+      "uid": "cluster-nodes",
+      "title": "Cluster Nodes",
+      "tags": ["cluster", "nodes"],
+      "timezone": "browser",
+      "schemaVersion": 39,
+      "refresh": "1m",
+      "time": { "from": "now-24h", "to": "now" },
+      "templating": { "list": [] },
+      "annotations": { "list": [] },
+      "panels": [
+        {
+          "id": 1, "type": "stat", "title": "Node status (kubelet reachable)",
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "gridPos": { "h": 8, "w": 18, "x": 0, "y": 0 },
+          "targets": [ { "refId": "A", "expr": "up{job=\"kubernetes-nodes\"}", "legendFormat": "{{instance}}" } ],
+          "fieldConfig": {
+            "defaults": {
+              "mappings": [
+                { "type": "value", "options": { "0": { "text": "DOWN", "color": "red" }, "1": { "text": "UP", "color": "green" } } }
+              ],
+              "thresholds": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 1 } ] }
+            },
+            "overrides": []
+          },
+          "options": { "colorMode": "background", "graphMode": "none", "textMode": "value_and_name", "reduceOptions": { "calcs": ["lastNotNull"] } }
+        },
+        {
+          "id": 2, "type": "stat", "title": "Nodes up / total",
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "gridPos": { "h": 8, "w": 6, "x": 18, "y": 0 },
+          "targets": [
+            { "refId": "A", "expr": "sum(up{job=\"kubernetes-nodes\"})", "legendFormat": "up" },
+            { "refId": "B", "expr": "count(up{job=\"kubernetes-nodes\"})", "legendFormat": "total" }
+          ],
+          "fieldConfig": { "defaults": { "unit": "short", "color": { "mode": "thresholds" }, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null } ] } }, "overrides": [] },
+          "options": { "colorMode": "value", "graphMode": "none", "textMode": "value_and_name", "reduceOptions": { "calcs": ["lastNotNull"] } }
+        },
+        {
+          "id": 3, "type": "timeseries", "title": "Node up/down history (1 = up, 0 = down)",
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 },
+          "targets": [ { "refId": "A", "expr": "up{job=\"kubernetes-nodes\"}", "legendFormat": "{{instance}}" } ],
+          "fieldConfig": { "defaults": { "unit": "short", "min": 0, "max": 1, "custom": { "drawStyle": "line", "lineInterpolation": "stepAfter", "fillOpacity": 15 } }, "overrides": [] },
+          "options": { "legend": { "displayMode": "table", "placement": "bottom", "calcs": ["min"] } }
+        }
+      ]
+    }

monitoring/install.sh

@@ -25,7 +25,8 @@ kubectl apply -f smartctl-exporter.yaml
 # Dashboards (as-code; the Grafana sidecar auto-loads labelled ConfigMaps, so
 # applying before/after Grafana both work). Tuned to this cluster's metric
 # labels so they show data (community dashboards 1860/12239 do not — wrong job).
-kubectl apply -f grafana-dashboard-smart.yaml \
+kubectl apply -f grafana-dashboard-cluster.yaml \
+              -f grafana-dashboard-smart.yaml \
               -f grafana-dashboard-node.yaml \
               -f grafana-dashboard-gpu.yaml