[flyte2] Instrument the runs service reconcilers (abort-reconciler) with Prometheus metrics

[pingsutw]

Part of #7445. Depends on #7446 (the /metrics endpoint + Scope must exist first).

Summary

Add Prometheus metrics to the runs service background reconcilers (starting with the abort reconciler) to observe queue depth, processing throughput, retries, and failures.

Background

runs/service/abort_reconciler.go runs as a background worker (registered in runs/setup.go via sc.AddWorker("abort-reconciler", ...)). It has a worker pool, a bounded queue (QueueSize: 1000), and retry logic (MaxAttempts, InitialDelay, MaxDelay). None of this is currently observable via metrics.

What to do

1. Thread the metrics Scope (from [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446) into service.NewAbortReconciler(...) (extend its config/constructor).
2. Emit metrics such as:
   • current queue depth / pending items (gauge)
   • items processed (counter, labeled by success/failure)
   • retries / attempts (counter)
   • per-item processing latency (timer/histogram)

Acceptance criteria

• /metrics exposes abort-reconciler queue depth, processed count (success/failure), retry count, and processing latency.
• Metrics use a dedicated sub-scope, e.g. scope.NewSubScope("abort_reconciler"), created once.
• A unit test verifies that processing an item updates the relevant counters/gauges.

Pointers

• runs/service/abort_reconciler.go — the reconciler implementation and its run loop.
• runs/setup.go:64-73 — where NewAbortReconciler is constructed and registered as a worker.
• flytestdlib/promutils/scope.go — Scope helpers (MustNewGauge, MustNewCounter, MustNewStopWatch, NewSubScope).

Notes for contributors

• The gauge for queue depth should be updated as items are enqueued/dequeued (or sampled periodically).
• This is independent of [flyte2] Add gRPC/Connect RPC metrics interceptor to the runs service #7447 and [flyte2] Instrument the runs service DB repository layer with Prometheus metrics #7448; all three consume the same Scope from [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446.

[runllm]

To instrument the abort reconciler with Prometheus metrics, thread a promutils.Scope (preferably a sub-scope like scope.NewSubScope("abort_reconciler")) into the reconciler's constructor. Use MustNewGauge for queue depth, MustNewCounter for processed items (label by success/failure) and retries, and MustNewStopWatch or MustNewHistogram for per-item processing latency. Update the gauge as items are enqueued/dequeued, increment counters on processing/retry, and record latency per item. Unit tests should verify metric updates during processing. See flytestdlib/promutils/scope.go for metric helpers and sub-scope usage.

Would you like a step-by-step code example or more detail on metric registration and updates?

Sources:

• flytestdlib/promutils/scope.go

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!