Is your feature request related to a problem? Please describe.
Investigating #10730 (serviceResolver.Lookup() routing to draining/stopping members during rolling restarts) and #11108 (no operator-facing way to evict a gray-failed host) surfaced a related gap: the membership package currently emits almost no metrics, so operators have no visibility into ring health before it turns into a routing incident like these two.
Confirmed by reading common/membership/: the only existing metric, MembershipChangedCounter, is emitted from outside the package (service/history/shard/ownership.go), not from membership itself. Neither monitor.go nor service_resolver.go accept a metrics.Handler today. There's no gauge for how many hosts are reachable, how many are actually accepting traffic (non-draining), or how many are draining, per service.
Describe the solution you'd like
Add three per-service gauges, emitted on every ring refresh in serviceResolver.refreshLocked():
membership_reachable_members — from the existing MemberCount()
membership_available_members — from the existing AvailableMemberCount() (non-draining)
membership_draining_members — the difference between the two
All tagged with the existing metrics.ServiceNameTag(service) for a per-service (frontend/history/matching/worker) breakdown.
Implementation shape:
Add MetricsHandler metrics.Handler to factoryParams in common/membership/ringpop/factory.go (fx already provides this elsewhere, so it's just a new field)
Thread it through newMonitor → newServiceResolver
Emit the three gauges inside refreshLocked() right after the new ringAndHosts is stored, following the existing metrics.NewGaugeDef(...) + .With(handler).Record(...) pattern used elsewhere (e.g. NumShardsGauge in service/history/shard/controller_impl.go)
This is purely additive — no change to Lookup()/routing behavior — so it doesn't collide with #10730's fix landing in parallel, and gives operators (and dashboards/alerts) the missing signal to catch ring-health degradation before it causes a routing incident.
Describe alternatives you've considered
Waiting for #10730/#11108 to land first — but those solve routing correctness, not observability; this is complementary, not redundant, and can land independently.
Logging only (current state) — the existing "Current reachable members" log line in service_resolver.go isn't queryable/alertable the way a metric is.
Additional context
Happy to contribute a PR for this if the team is open to the approach — I've already mapped the exact call sites and confirmed the existing patterns to follow (gauge defs, service name tagging, refresh-cycle hook point).
Is your feature request related to a problem? Please describe.
Investigating #10730 (serviceResolver.Lookup() routing to draining/stopping members during rolling restarts) and #11108 (no operator-facing way to evict a gray-failed host) surfaced a related gap: the membership package currently emits almost no metrics, so operators have no visibility into ring health before it turns into a routing incident like these two.
Confirmed by reading common/membership/: the only existing metric, MembershipChangedCounter, is emitted from outside the package (service/history/shard/ownership.go), not from membership itself. Neither monitor.go nor service_resolver.go accept a metrics.Handler today. There's no gauge for how many hosts are reachable, how many are actually accepting traffic (non-draining), or how many are draining, per service.
Describe the solution you'd like
Add three per-service gauges, emitted on every ring refresh in serviceResolver.refreshLocked():
membership_reachable_members — from the existing MemberCount()
membership_available_members — from the existing AvailableMemberCount() (non-draining)
membership_draining_members — the difference between the two
All tagged with the existing metrics.ServiceNameTag(service) for a per-service (frontend/history/matching/worker) breakdown.
Implementation shape:
Add MetricsHandler metrics.Handler to factoryParams in common/membership/ringpop/factory.go (fx already provides this elsewhere, so it's just a new field)
Thread it through newMonitor → newServiceResolver
Emit the three gauges inside refreshLocked() right after the new ringAndHosts is stored, following the existing metrics.NewGaugeDef(...) + .With(handler).Record(...) pattern used elsewhere (e.g. NumShardsGauge in service/history/shard/controller_impl.go)
This is purely additive — no change to Lookup()/routing behavior — so it doesn't collide with #10730's fix landing in parallel, and gives operators (and dashboards/alerts) the missing signal to catch ring-health degradation before it causes a routing incident.
Describe alternatives you've considered
Waiting for #10730/#11108 to land first — but those solve routing correctness, not observability; this is complementary, not redundant, and can land independently.
Logging only (current state) — the existing "Current reachable members" log line in service_resolver.go isn't queryable/alertable the way a metric is.
Additional context
Happy to contribute a PR for this if the team is open to the approach — I've already mapped the exact call sites and confirmed the existing patterns to follow (gauge defs, service name tagging, refresh-cycle hook point).