observability: readiness gauges for runtime ephemeris push / config apply (alert on permanent, non-requeuing failures)

[thc1006]

Summary

Add a readiness gauge so a permanent, non-requeuing ephemeris-push failure is alertable. PR #215 fixed ntn_operators_ephemeris_push_errors_total to count every push failure (not once per episode), which restores the NTNEphemerisPushFailing alert for transient outages that tight-requeue every minute. But the four permanent reasons that deliberately do not requeue still increment the counter only once, and a rate/increase() alert cannot sustain on a one-shot sample.

Background

pushEphemerisUpdateIfNeeded classifies failures via ephemerisPushShouldRequeue(reason):

• Requeuing (transient) — e.g. ProviderPushFailed, gNB unreachable, API GET error → RequeueAfter: 1m. The per-failure counter (shipped in feat(controller): WO-20 event & GitOps hygiene — emit after status, gate on transition #215) advances each minute, so increase(ntn_operators_ephemeris_push_errors_total[15m]) > 0 for: 15m fires and stays firing through the outage. ✅ Covered.
• Non-requeuing (permanent) — EphemerisRefNotFound, EphemerisPayloadMissing, EphemerisStale, ProviderPushRejected. These clear only on an external change (spec edit / ephemeris refresh), so the reconcile does not retry. The counter increments once; increase(...[15m]) is true for ~15m then resolves, so a persistent misconfiguration goes unalerted. ❌ Not covered by a rate alert.

The durable EphemerisPushed=False condition already records the permanent failure for kubectl inspection, but there is no alertable metric for it. A misconfiguration that silently prevents ephemeris push is arguably more important to alert on than a transient blip (which often self-heals).

Proposal

Add a readiness gauge:

ntn_operators_ephemeris_push_ready{namespace, config}  0|1

• 1 on a successful push (or when the push is already up to date / not configured — "not currently failing").
• 0 on any push failure, regardless of reason.
• DeletePartialMatch{namespace, config} on finalizer cleanup (mirrors the counter, prevents cardinality leak on CR delete).

Alert:

- alert: NTNEphemerisPushNotReady
  expr: ntn_operators_ephemeris_push_ready == 0
  for: 15m

A gauge value persists in the TSDB between scrapes, so == 0 for 15m fires even when the reconcile does not requeue — covering the permanent reasons that the counter alert cannot. It is complementary to, not a replacement for, the existing NTNEphemerisPushFailing counter-rate alert (which measures failure rate during a churning transient outage).

Scope decision (design, before code)

Do this holistically, not as a one-metric bolt-on:

• Apply the same readiness pattern to ConfigApplyErrorsTotal (ntn_operators_config_apply_ready) so runtime push and config apply are consistent, rather than leaving an asymmetry (push has readiness, apply does not).
• Keep the existing *_errors_total counters and their rate alerts (failure rate ≠ currently-broken); the gauges add the "currently broken" signal.
• Update dist/chart/templates/monitoring/prometheusrule.yaml, the docs/runbooks/alerts.md runbook, and optionally the Grafana dashboard.
• Tests: gauge → 0 on failure, → 1 on success/recovery/up-to-date, deleted on CR delete; and that the readiness alert expression behaves for a non-requeuing reason.

Origin

Surfaced in the PR #215 (WO-20) adversarial review. The minimal per-failure counter fix shipped in #215; this gauge is the tracked follow-up for the permanent-reason coverage the counter fundamentally cannot provide.

[thc1006]

Closure correction: this was closed by #218, which shipped ephemeris_push_ready and gp_fetch_ready — two of the three gauges scoped here. The third, ntn_operators_config_apply_ready, was never written, so the asymmetry this issue's scope section explicitly set out to avoid ("push has readiness, apply does not") is exactly what remained.

The remaining half turned out to matter more than the symmetry argument suggested. ConfigApplyErrorsTotal is incremented in exactly one place — the ApplyCellConfig failure branch — so three other ways the apply can be broken increment it zero times and return a nil error, making them invisible to both NTNConfigApplyErrors and controller_runtime_reconcile_errors_total:

• InternalError — provider registry not configured (requeues 1m)
• UnsupportedProvider — spec.provider.type unregistered (no requeue at all)
• StatusCheckFailed — post-apply verification could not confirm the write (requeues 1m)

UnsupportedProvider is the same permanent, non-requeuing shape that motivated this issue for the push path — except the counter does not even emit the single sample the push path got.

PR #306 adds ntn_operators_config_apply_ready, the NTNConfigApplyNotReady alert and a runbook section that triages by ConfigApplied reason. Leaving this issue closed and tracking the completion in #306; noting it here so the record is not misleading.