[flyte2] Instrument the actions service (watcher metrics + dropped-updates counter)

[pingsutw]

Part of #7445. Depends on #7446 (the /metrics endpoint + initialized Scope must exist first).

Summary

Instrument the actions service with Prometheus metrics: implement the existing dropped-updates counter TODO, and add throughput / latency / queue-depth metrics for the TaskAction watcher.

Background

The actions service is already partly wired for metrics — it just has nothing to plug into yet:

• actions/setup.go:39 already passes sc.Scope into NewActionsClient(...).
• actions/k8s/client.go:91 already uses scope.NewSubScope("actions_filter") for the dedup bloom filter.
• actions/k8s/client.go:65 has an explicit TODO: // TODO: add a prometheus counter for dropped updates when metrics are wired up.

Note on the metrics scope: When run via the unified manager (manager/cmd/main.go:75), sc.Scope is already initialized (promutils.NewScope("flyte")) before actions.Setup runs, so the bloom-filter sub-scope at client.go:91 works and there is no panic. The dependency on #7446 is because #7446 mounts the /metrics endpoint — without it, the metrics you add here are registered into the default registry but never exposed to a scrape. (#7446 also initializes sc.Scope at the framework level, which additionally makes the standalone actions/cmd/main.go binary safe — that path currently leaves sc.Scope nil, so client.go:90-91's scope.NewSubScope(...) would panic there, since RecordFilterSize defaults to 1 << 23 > 0.)

What to do

Using the Scope available on ActionsClient (passed in via NewActionsClient), add metrics under a dedicated sub-scope (e.g. scope.NewSubScope("watcher")):

1. Dropped updates counter — implement the TODO at actions/k8s/client.go:65. Increment a counter whenever a watch update is dropped (e.g. buffer full / channel send would block).
2. Watcher throughput — counter of TaskAction events processed, labeled by result (success/error).
3. Processing latency — a timer/histogram around per-event handling in the watch worker loop.
4. Queue/buffer depth — a gauge for the watch buffer occupancy (config WatchBufferSize), updated as events are enqueued/dequeued (or sampled periodically).

Acceptance criteria

• /metrics exposes a dropped-updates counter, watcher event throughput (by result), processing latency, and buffer depth for the actions service.
• The TODO at actions/k8s/client.go:65 is implemented and removed.
• Metrics are created once under a dedicated sub-scope (no Prometheus duplicate-registration panics).
• A unit test verifies the dropped-updates counter increments when an update is dropped, and that the throughput counter increments on event processing.

Pointers

• actions/k8s/client.go — the watcher, worker loop, buffer, and the dropped-updates TODO (line 65); constructor NewActionsClient (line 77) already receives a promutils.Scope.
• actions/setup.go:31-40 — where NewActionsClient is constructed with sc.Scope.
• flytestdlib/promutils/scope.go — Scope helpers (MustNewCounter, MustNewGauge, MustNewStopWatch, NewSubScope).

Notes for contributors

• Keep label cardinality bounded — label by result/status, never by action/run IDs or other user input.
• This is independent of the runs-service instrumentation issues ([flyte2] Add gRPC/Connect RPC metrics interceptor to the runs service #7447, [flyte2] Instrument the runs service DB repository layer with Prometheus metrics #7448, [flyte2] Instrument the runs service reconcilers (abort-reconciler) with Prometheus metrics #7449); all consume the same Scope from [flyte2] Add /metrics endpoint and initialize metrics Scope in the app framework #7446.

[runllm]

To instrument the actions service with Prometheus metrics, use the promutils.Scope passed to NewActionsClient. Create a dedicated sub-scope (e.g., scope.NewSubScope("watcher")) and define the following metrics: a dropped-updates counter (MustNewCounter), a throughput counter labeled by result (MustNewCounterVec), a processing latency timer/histogram (MustNewStopWatch or MustNewHistogram), and a queue/buffer depth gauge (MustNewGauge). Update these metrics at the appropriate points in the watcher logic, ensuring metrics are registered only once to avoid duplicate-registration panics. Remove the TODO at actions/k8s/client.go:65 by incrementing the dropped-updates counter when an update is dropped. Unit tests should verify counter increments for dropped updates and throughput.

For implementation details, refer to the Scope API in flytestdlib/promutils/scope.go, which provides helpers for counters, gauges, histograms, and sub-scopes. Ensure label cardinality is bounded (e.g., label by result/status only). Do not start until #7446 is merged, as it initializes the Scope and /metrics endpoint. See the acceptance criteria and pointers in your issue for exact file locations and requirements. Would you like a step-by-step code example for these metrics?

Sources:

• flytestdlib/promutils/scope.go
• flytestdlib/promutils/scope_test.go

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!