Add Prometheus custom metrics for scaling observability (#241)

* Add Prometheus custom metrics for scaling observability

Operators have no way to ask, from metrics alone, "did this autoscaler
scale, and why?" Reconcile logs answer the question for one event at a
time but cannot be aggregated, and the controller-runtime default
metrics describe reconcile loop health, not scaling outcomes.

Add an internal/observability package that registers custom collectors
on controller-runtime's metrics.Registry, so the same /metrics endpoint
that already serves the standard metrics now also exposes:

- State gauges: current/desired/min/max processing units (including the
  schedule-expanded effective bounds), CPU utilization and target per
  metric type, instance ready, active schedules count and additional PU
  sum, last scale / last sync timestamps.
- Schedule counters: activations (cron firings) and deactivations
  labeled by reason (expired | unregistered).
- Scale event counters and histogram: scale_events_total labeled by
  direction and driver (cpu_high_priority | cpu_total | schedule);
  scale_skipped_total labeled by reason; scale_pu_delta histogram of
  the absolute PU change per event.
- Operational counters and histograms: instance_update_total /
  instance_update_duration_seconds for Spanner UpdateInstance;
  metrics_fetch_total / metrics_fetch_duration_seconds for Cloud
  Monitoring GetInstanceMetrics.

Every business metric carries (namespace, name, project_id, instance_id)
directly rather than via an info-metric join, so the Datadog Agent's
OpenMetrics check and other flat-tag consumers can attribute series
without a PromQL join. Active series scale with the number of
SpannerAutoscaler resources (~80 series per resource); join-style
flexibility is preserved for future via an info metric if needed.

The driver label on scale_events_total is a best-effort attribution
computed at the call site: a desired value pinned to the effective min
floor that exceeds the spec-level min is attributed to "schedule"; in
dual CPU mode the per-metric candidates from calcDesiredPUFromCPU are
recomputed and the larger one wins.

A labelsCache on the reconciler tracks the last observed identity
labels per resource so DeleteSeries can be called on the NotFound
branch (when sa.Spec is gone) and on spec.targetInstance changes that
would otherwise leave orphan series on /metrics indefinitely.

The syncer constructor now takes projectID and instanceID directly so
the Cloud Monitoring fetch wrappers can emit labels without an extra
ctrlClient.Get per call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Add HTTP exposition round-trip test for custom metrics

The existing unit tests prove each Record* helper updates its in-memory
vector, but do not exercise the path the controller binary actually
uses at runtime: promhttp.Handler serializing reg.Gather() into
Prometheus exposition format. A regression in label ordering, label
name typo, or registration omission would slip past the per-helper
tests.

Add a single TestMetricsHTTPExposition that:

- Stands up a fresh prometheus.NewRegistry, calls Register, drives one
  sample through every Record* helper, and serves the registry through
  httptest.NewServer wrapping promhttp.HandlerFor.
- Pulls /metrics over HTTP, parses the body with expfmt.TextParser, and
  cross-checks the parsed metric families against reg.Gather() so the
  expected metric set is derived automatically — adding a new
  collector to allCollectors() and calling its Record* here is the
  only edit needed.
- Verifies every spanner_autoscaler_* series carries the four identity
  labels (namespace, name, project_id, instance_id) with the values the
  test recorded. A future emit site that forgets to thread one of the
  labels will fail this assertion without naming the offending metric.

Specific values, bucket boundaries, and exposition line wording are
intentionally not asserted; those change far more often than the
contract the test is meant to protect.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Defer schedule deactivation side effects until persistence

cleanupActiveSchedules is invoked inside retry.RetryOnConflict in
scheduler.Update; on a conflict-retry the closure re-runs and re-fires
every side effect, so the deactivation counter (and the log line that
landed via PR #240) could be incremented multiple times for a single
expiry. pruneActiveSchedules in the controller reconcile has an
analogous issue: it fires before Status.Update, so a Status.Update
failure that triggers a requeue causes the same orphan ActiveSchedule
entries to be re-observed and re-recorded until the write eventually
succeeds.

Refactor both helpers to be pure: return the entries kept and the
entries removed, with no logging or counter increments inside. The
callers now emit "scheduled scaling deactivated" / "removed currently
active schedule" logs and RecordScheduleDeactivation only after the
status update has been persisted, guaranteeing one side-effect per
actual transition.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Fix golangci-lint failures (gocyclo, nolintlint)

1. Reconcile cyclomatic complexity exceeded the project threshold (31 > 30)
   after this PR added the labelsCache lookup in the NotFound branch and
   the labels-change detection block.

   Extract both into a single updateLabelsCache helper that takes the
   current SpannerAutoscaler (or nil for the deletion path). The
   inline NotFound block now reduces to one method call, and the
   refresh path likewise.  This brings the function back under the
   threshold without dropping any behavior.

2. cmd/main.go carried two //nolint:staticcheck directives suppressing
   a deprecation warning on mgr.GetEventRecorderFor, but staticcheck no
   longer flags those call sites — leaving the directives in place
   makes nolintlint fail with "unused". Remove them; when staticcheck
   begins flagging GetEventRecorderFor again the warning will surface
   naturally and we can address it on its own merits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Address review: record state on every reconcile return path

The previous placement of observability.RecordState at the bottom of
Reconcile was skipped on the early-return branches for
InstanceState != Ready and CurrentProcessingUnits == 0. As nktks
pointed out (#241 r3285973010), a ready -> not-ready transition would
leave spanner_autoscaler_instance_ready stuck at 1, and the same
issue applies to every state gauge.

Move the call into a defer placed immediately after the labels-cache
refresh, where sa is guaranteed populated. The deferred call captures
&sa so it observes the final state at return time, regardless of which
early-return path is taken. RecordState is panic-free with any
non-nil *SpannerAutoscaler (all field accesses use value types or are
nil-guarded), so it is safe to invoke from a defer that runs on every
return.

Also restore the //nolint:staticcheck directives on the two
mgr.GetEventRecorderFor calls in cmd/main.go. They were removed when
local golangci-lint did not flag SA1019, but the CI version does
flag the deprecation, so the directives are still needed. The
migration to GetEventRecorder requires refactoring every Event call
site (which use the old record.EventRecorder type) and remains tracked
separately.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: tkuchiki

README.md

@@ -450,6 +450,82 @@ Following are some other advanced methods which can also be used for GCP authent
 </details>
 
 
+## Metrics
+
+The controller exposes custom Prometheus metrics on the standard controller-runtime `/metrics` endpoint (bound to `--metrics-bind-address`, default `127.0.0.1:8080`). All business metrics carry the four identity labels `namespace`, `name`, `project_id`, `instance_id` so they can be sliced by either the Kubernetes resource identity or the target Spanner instance.
+
+The endpoint can be scraped by Prometheus (`ServiceMonitor` / `PodMonitor`), the Datadog Agent's OpenMetrics check, VictoriaMetrics, Grafana Cloud, or any tool that consumes the Prometheus exposition format. No vendor-specific SDK is required.
+
+### Available metrics
+
+#### State (Gauge)
+
+| Name | Extra labels | Description |
+|---|---|---|
+| `spanner_autoscaler_current_processing_units` | — | Current Spanner processing units. |
+| `spanner_autoscaler_desired_processing_units` | — | Desired processing units computed in the latest reconcile. |
+| `spanner_autoscaler_min_processing_units` | — | Configured `spec.scaleConfig.processingUnits.min`. |
+| `spanner_autoscaler_max_processing_units` | — | Configured `spec.scaleConfig.processingUnits.max`. |
+| `spanner_autoscaler_effective_min_processing_units` | — | Effective lower bound including additions from currently active schedules. |
+| `spanner_autoscaler_effective_max_processing_units` | — | Effective upper bound including additions from currently active schedules. |
+| `spanner_autoscaler_cpu_utilization` | `type=high_priority\|total` | Current CPU utilization percentage (0–100) per metric type. |
+| `spanner_autoscaler_cpu_utilization_target` | `type=high_priority\|total` | Configured target CPU utilization percentage. |
+| `spanner_autoscaler_instance_ready` | — | `1` when the Spanner instance state is `ready`, `0` otherwise. |
+| `spanner_autoscaler_active_schedules` | — | Number of `SpannerAutoscaleSchedule` entries currently in effect. |
+| `spanner_autoscaler_active_schedule_additional_pu` | — | Sum of `AdditionalPU` contributed by currently active schedules. |
+| `spanner_autoscaler_last_scale_timestamp_seconds` | — | Unix timestamp of the last successful processing-units update. |
+| `spanner_autoscaler_last_sync_timestamp_seconds` | — | Unix timestamp of the last successful Cloud Monitoring sync. |
+
+#### Scaling events
+
+| Name | Type | Extra labels | Description |
+|---|---|---|---|
+| `spanner_autoscaler_scale_events_total` | Counter | `direction=up\|down`, `driver=cpu_high_priority\|cpu_total\|schedule` | Successful processing-units updates. The `driver` label is a best-effort attribution to the metric (or schedule floor) that drove the decision. |
+| `spanner_autoscaler_scale_skipped_total` | Counter | `reason` | Reconciles where scaling was skipped. Reasons: `same`, `scale_up_interval`, `scale_down_interval`, `scale_down_window`, `instance_not_ready`, `cpu_not_ready`. |
+| `spanner_autoscaler_scale_pu_delta` | Histogram | `direction=up\|down` | Absolute processing-units change per scale event. Buckets: 100, 200, 500, 1000, 2000, 5000, 10000, 20000. |
+
+#### Scheduled scaling
+
+| Name | Extra labels | Description |
+|---|---|---|
+| `spanner_autoscaler_schedule_activations_total` | — | Number of cron firings that created an `ActiveSchedule` entry. |
+| `spanner_autoscaler_schedule_deactivations_total` | `reason=expired\|unregistered` | Number of `ActiveSchedule` entries removed (by `EndTime` expiry or by schedule resource deletion). |
+
+#### Operational (API quality)
+
+| Name | Type | Extra labels | Description |
+|---|---|---|---|
+| `spanner_autoscaler_instance_update_total` | Counter | `result=success\|error` | Spanner `UpdateInstance` API call count. |
+| `spanner_autoscaler_instance_update_duration_seconds` | Histogram | — | Latency of `UpdateInstance` calls. Buckets: 0.1, 0.5, 1, 2, 5, 10, 30, 60. |
+| `spanner_autoscaler_metrics_fetch_total` | Counter | `result=success\|error` | Cloud Monitoring `GetInstanceMetrics` call count. |
+| `spanner_autoscaler_metrics_fetch_duration_seconds` | Histogram | — | Latency of Cloud Monitoring fetches. Buckets: 0.05, 0.1, 0.25, 0.5, 1, 2, 5. |
+
+The standard controller-runtime metrics (`controller_runtime_reconcile_*`, `workqueue_*`) and Go runtime metrics are also exposed on the same endpoint and are not duplicated here.
+
+### Example PromQL queries
+
+```promql
+# Gap between desired and current PU (scaling lag).
+spanner_autoscaler_desired_processing_units - spanner_autoscaler_current_processing_units
+
+# UpdateInstance error rate over the last 5 minutes.
+sum(rate(spanner_autoscaler_instance_update_total{result="error"}[5m]))
+  /
+sum(rate(spanner_autoscaler_instance_update_total[5m]))
+
+# Scale-up frequency by driver over the last hour.
+sum by (driver) (rate(spanner_autoscaler_scale_events_total{direction="up"}[1h]))
+
+# Cloud Monitoring fetch p99 latency.
+histogram_quantile(0.99, sum by (le) (rate(spanner_autoscaler_metrics_fetch_duration_seconds_bucket[5m])))
+
+# Autoscalers pinned at their effective max for 5+ minutes.
+max_over_time(
+  (spanner_autoscaler_current_processing_units
+    == bool spanner_autoscaler_effective_max_processing_units)[5m:1m]
+) == 1
+```
+
 ## Development and Contribution
 
 See [docs/development.md](docs/development.md) and [CONTRIBUTING.md](.github/CONTRIBUTING.md) respectively.

cmd/main.go

@@ -30,6 +30,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/log/zap"
 	ctrlmanager "sigs.k8s.io/controller-runtime/pkg/manager"
 	ctrlsignals "sigs.k8s.io/controller-runtime/pkg/manager/signals"
+	ctrlmetrics "sigs.k8s.io/controller-runtime/pkg/metrics"
 	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
 	ctrlwebhook "sigs.k8s.io/controller-runtime/pkg/webhook"
 
@@ -38,6 +39,7 @@ import (
 
 	spannerv1alpha1 "github.com/mercari/spanner-autoscaler/api/v1alpha1"
 	"github.com/mercari/spanner-autoscaler/internal/controller"
+	"github.com/mercari/spanner-autoscaler/internal/observability"
 )
 
 var (
@@ -101,6 +103,11 @@ func main() {
 		os.Exit(exitCode)
 	}
 
+	if err := observability.Register(ctrlmetrics.Registry); err != nil {
+		setupLog.Error(err, "failed to register custom metrics")
+		os.Exit(exitCode)
+	}
+
 	options := ctrlmanager.Options{
 		Scheme:                 scheme,
 		LeaderElection:         *enableLeaderElection,

go.mod

@@ -13,6 +13,9 @@ require (
 	github.com/netresearch/go-cron v0.14.0
 	github.com/onsi/ginkgo/v2 v2.28.3
 	github.com/onsi/gomega v1.40.0
+	github.com/prometheus/client_golang v1.23.2
+	github.com/prometheus/client_model v0.6.2
+	github.com/prometheus/common v0.67.5
 	go.uber.org/zap v1.28.0
 	golang.org/x/oauth2 v0.36.0
 	golang.org/x/sync v0.20.0
@@ -57,14 +60,12 @@ require (
 	github.com/googleapis/gax-go/v2 v2.22.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/kylelemons/godebug v1.1.0 // indirect
 	github.com/mailru/easyjson v0.7.7 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
-	github.com/prometheus/client_golang v1.23.2 // indirect
-	github.com/prometheus/client_model v0.6.2 // indirect
-	github.com/prometheus/common v0.67.5 // indirect
 	github.com/prometheus/procfs v0.19.2 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	github.com/x448/float16 v0.8.4 // indirect