feat(diagnostics): expose topology, SSE and cache metrics in the bundle

[arlenvasconcelos]

Description

Issue #1303 was slow to diagnose because the data needed to locate the bottleneck never reached us. The phase-level numbers were gated behind --dev and printed to stderr, which no bug reporter sends, and the one always-on store answered only "how long did a topology build take" with every build shape averaged into a single percentile.

Everything here lands in pkg/perfstats and surfaces through /api/diagnostics and the overlay. No new stdout logging, no new flags, and every line is conditional so a healthy cluster grows the report by about five lines.

• Split topology build stats by scope (full / scoped / refused). A namespace filter is what bounds the cost, so one duration window can no longer average a cluster-wide build with a namespace-scoped one.
• Record the broadcast cycle: wall time, client groups, auth groups and marshal time. Auth groups are the fan-out multiplier that turns one slow build into a slow cycle, and were previously invisible. Recorded from a defer so cycles abandoned for a cluster switch report the time they spent instead of looking free.
• Count abandoned, coalesced and retried cycles, plus the active debounce rung.
• Sample the resource-change queue. Drops already surfaced once the channel overflowed; depth and the high-water mark show the approach to that cliff, and the received count gives the rate.
• Time full topology rebuilds that run on a request goroutine because the relationship cache was dirty, and the edge-index builds with them.
• Record how long the critical and deferred informer phases took.

Per-kind stats deliberately carry no payload size: payload is measured at marshal time, downstream of the build and with no record of which kind produced it, so the field could only ever report zero.

Type of change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

How has this been tested?

Describe the tests you ran to verify your changes.

• Tested locally with minikube/kind
• Tested against a remote cluster
• Added/updated unit tests

Checklist

• My code follows the project's coding standards
• I have performed a self-review of my code
• I have added comments where necessary
• My changes generate no new warnings
• Any dependent changes have been merged

Related issues

Fixes #(issue number)

---

Note

Low Risk
Observability-only: atomic counters and sampled ring buffers on existing paths; no auth, data, or broadcast logic changes beyond recording metrics.

Overview
Adds always-on performance instrumentation in pkg/perfstats and wires it through the SSE broadcaster, topology builder, and informer cache so bug reports and /api/diagnostics can show where time goes—not a single blended topology-build percentile.

Topology builds are recorded as full / scoped / refused (BuildKind), so a multi-second cluster-wide build no longer hides inside namespace-scoped p95s. The builder tags refused large-cluster guard paths separately from completed graphs.

SSE gains counters for coalesced broadcast requests, abandoned cycles after cluster switches, retries when the worker re-arms, and the active debounce ladder rung. Each broadcast cycle records wall time, client-group vs auth-group fan-out (clone/strip/marshal multiplier), and aggregate marshal time. The change watcher samples queue depth (throttled) and tracks high-water on the resource-change channel.

Relationship cache paths time on-demand full rebuilds on request goroutines and IndexByResource builds.

Informer sync exposes critical and deferred phase wall times in cache sync status. The web diagnostics overlay and GitHub export show these fields conditionally so quiet clusters stay short.

^{Reviewed by Cursor Bugbot for commit 52dfe52. Bugbot is set up for automated code reviews on this repo. Configure here.}