Feature: Observability

[jakubhlavacka]

Problem

There is no observability into Claudie's LB clusters. Each LB cluster consists of nodepools where every node runs Envoy (via docker-compose), forwarding traffic to the downstream K8s cluster.
Envoy exposes Prometheus metrics via its admin interface (/stats/prometheus), but nothing scrapes or visualises them today. Operators have zero visibility into connection counts, error rates, latencies, or whether an LB node is healthy.

Goal

Introduce an observability stack that, at minimum, gives operators insight into LB cluster health and Envoy traffic signals (connections, errors, latency, saturation per role/listener).

Open questions

Scope — what else besides LBs?

• Should the observability cover the management plane as well (Manager, Terraformer, Ansibler, Kuber, Kube-eleven services, NATS, MongoDB)?
• What about downstream K8s clusters — do we monitor them, or leave that to the user to set up on their own?

Architecture

• Where does the observability stack live? On the management cluster? On the downstream cluster? Both?
• Push vs pull? LB nodes sit behind the WireGuard VPN. Pull-based scraping requires Prometheus to be inside the VPN mesh. A push model (remote-write agent on LB nodes) avoids that but adds
  agent lifecycle management.

Tooling

• Metrics: Prometheus? Victoria Metrics? Something else?
• Logs: Do we need log aggregation (Loki, EFK), or is stdout/journald enough for now?
• Dashboards & alerting: Ship Grafana dashboards and alert rules out of the box, or as optional extras?

Target discovery

• Dynamic targets: Adding/removing LB nodes or entire LB clusters must update the monitoring targets. How do we handle this — regenerate config on each reconciliation, use service discovery, or something else?

DNS & reachability

• DNS endpoint monitoring: Should we actively probe the LB's public DNS endpoint for availability (e.g. blackbox exporter), or are Envoy-level metrics sufficient?

Delivery

• Opt-in or default? Is the stack deployed automatically with Claudie, or enabled explicitly by the user?
• Integration with existing monitoring: Should we just expose ServiceMonitor/PodMonitor CRs so users with their own Prometheus Operator can scrape Claudie without us deploying a new stack?
• Retention & resource budget: How long do we keep metrics? What additional resource overhead is acceptable on the cluster hosting the stack?

Next steps

Discuss at the next grooming and agree on direction.