feat(metrics): config-apply readiness gauge — the half of #216 that never shipped

[thc1006]

Completes #216. Every claim below was checked against the code before being accepted.

The tracking gap is real

#216's body scopes both gauges, in as many words:

Apply the same readiness pattern to ConfigApplyErrorsTotal (ntn_operators_config_apply_ready) so runtime push and config apply are consistent, rather than leaving an asymmetry (push has readiness, apply does not).

…under a heading that reads "Do this holistically, not as a one-metric bolt-on."

PR #218 — titled "readiness gauges for ephemeris push & GP fetch (#216)" — shipped ephemeris_push_ready and gp_fetch_ready and closed the issue. config_apply_ready was never written. The closure was premature; I've left a correction on #216.

The gap is worse than a missing symmetry

ConfigApplyErrorsTotal is incremented in exactly one place — the ApplyCellConfig failure branch. Three other ways the apply can be broken increment it zero times and return a nil error, so neither NTNConfigApplyErrors (a rate alert on that counter) nor controller_runtime_reconcile_errors_total can see them:

ConfigApplied reason	Cause	Requeues?	Counter?	Reconcile error?
InternalError	provider registry not configured	1m	✗	✗
UnsupportedProvider	spec.provider.type unregistered	no	✗	✗
StatusCheckFailed	post-apply read could not verify	1m	✗	✗
ApplyFailed / OwnershipConflict	ApplyCellConfig failed	1m	✓	✗

UnsupportedProvider is the permanent, non-requeuing case that motivated the gauge for the push path in the first place — and here the counter does not even emit the single sample the push path got. Config apply was therefore strictly worse off than ephemeris push was before #216.

The review's own counter-consideration is fair and I checked it: ProviderRef.Type carries +kubebuilder:validation:Enum=ocudu, so a newly created CR cannot reach UnsupportedProvider. That constrains one row. It does not touch InternalError or StatusCheckFailed, and it does not apply to objects stored before the enum or if the enum widens.

Change

• ntn_operators_config_apply_ready{namespace,config} — 1 only on a verified apply, 0 on all four failure paths, DeletePartialMatch on CR deletion.
• Alert NTNConfigApplyNotReady (== 0 for 15m), companion to the existing rate alert.
• Runbook section that triages by ConfigApplied reason, because the three causes have genuinely different fixes (operator wiring / spec edit / ConfigMap access) — including the jsonpath to read the reason.

Deliberate deviation from the proposal

The proposal was {namespace, config, provider}, matching the counter's labels. I dropped the provider label. Both existing readiness gauges are 2-label (ephemeris_push_ready{namespace,config}, gp_fetch_ready{namespace,ephemeris}), and a label that can change identity needs stale-series management a 2-label gauge simply does not have: a spec.provider.type edit would otherwise strand a series at 0 under the old value and alert forever. That also makes the proposal's "clean up stale series when the provider path is disabled" requirement unnecessary rather than implemented-and-fragile. Joining to the counter on {namespace,config} still works.

SmallCL / Boy Scout notes

One logical change, ~200 lines including tests and docs. I deliberately did not factor out the repeated prometheus.Labels{"namespace": …, "config": …} map even though it now appears nine times in this file: #297, #301 and #304 are all open against ntncellconfig_controller.go, and one of them touches the adjacent EphemerisPushReady metric block. Tidying there would buy a cosmetic win and pay for it in merge conflicts. Left as-is, matching the surrounding idiom.

Verification

• go build, go vet, golangci-lint v2.12.2: 0 issues; gofmt clean.
• go test ./internal/controller/... ./pkg/... green (envtest 1.36.2).
• helm template renders the new rule.
• The tests read gauges without instantiating them. testutil.ToFloat64(g.With(labels)) creates a missing series at 0, so a naive "want 0" assertion passes even with the production code deleted — the first version of these tests had exactly that hole. gaugeValue/counterValue collect and match labels, returning a found bool, which is what makes the assertions load-bearing.
• Mutation-verified per set-point, each reverting only itself:
  • drop the three Set(0) calls → no config_apply_ready series for InternalError: an absent series can never fire == 0
  • drop the success Set(1) → config_apply_ready = 0 (found=true), want 1 once the apply verifies
  • drop the DeletePartialMatch → the finalizer left 1 config_apply_ready series behind