Membership: provide an operator-facing way to evict a "gray-failed" host (SWIM-reachable but unable to serve) from the ring

[soohunee]

Is your feature request related to a problem? Please describe.

We hit a production incident where a single History node entered a "gray failure" state and there was no in-band way to remove it from the membership ring.

Timeline:

• The Kubernetes node hosting a History pod suffered a disk failure and became NotReady / unschedulable.
• kubectl drain on that node failed, so the pod's container kept running.
• The History process was still healthy at the network layer — it kept answering ringpop/SWIM pings on its membership (tchannel) port and could still reach the persistence layer — but it could no longer serve its shards.

Because SWIM liveness effectively means "responds to a ping on the membership port", the failure detector kept the node marked reachable, so the Frontend kept routing the shards owned by that host to it, and those shards were effectively unavailable.

Root cause, as far as I can tell from the code:

• The routing ring for shard ownership is built purely from SWIM-reachable members:
  serviceResolver.getReachableMembers() calls rp.GetReachableMemberObjects(...)
  (common/membership/ringpop/service_resolver.go), and shard ownership goes through
  ServiceResolver.Lookup() (service/history/shard/ownership.go), which uses the full hash ring. There is no health gate beyond SWIM ping-reachability.
• SWIM is designed so that a live, still-gossiping node refutes suspicion (incarnation numbers), so peers cannot durably evict it while it keeps gossiping.
• There is no operator-facing way to force-remove another host:
  Monitor.EvictSelf() / EvictSelfAt() (common/membership/ringpop/monitor.go)
  are self-only (rp.SelfEvict()).
• Deleting the cluster_membership row does not help on a running cluster: the DB is only read at bootstrap (GetClusterMembers is only called from
  fetchCurrentBootstrapHostports → bootstrapRingPop → Start()); the live ring is 100% SWIM. On top of that, the node re-inserts its own row every ~10–15s via
  startHeartbeatUpsertLoop, and the read-side liveness filter is only
  healthyHostLastHeartbeatCutoff = 20s, so the row always looks fresh (the 48h RecordExpiry is documented as being only for human inspection).
• The existing draining label doesn't help: it only affects AvailableMembers() /
  AvailableMemberCount() (used by the gRPC resolver), not the Lookup() hash ring used for shard ownership.

Net effect: the only ways to recover were out-of-band — terminate the underlying instance at the cloud/hypervisor level, or network-isolate the host's membership
port. Neither is available from Temporal itself, and both require infra access the on-call operator may not have.

Describe the solution you'd like

An operator-facing, cluster-consistent way to force-evict a specific host from the membership ring.

Concretely, a cluster-wide "membership denylist":

• An AdminService RPC (e.g. RemoveClusterMember{host_id} / a banned-hosts list) that persists the ban in shared state (dynamic config or a small persistence table).
• getReachableMembers() filters out any member whose host_id (the per-process UUID created in monitor.go) is on the denylist, before building the hash ring.
• Because every node reads the same denylist, all rings exclude the host consistently — there is no window where some nodes still route to it — so shard ownership moves to healthy hosts and rangeID fencing keeps data safe. This sidesteps the SWIM refutation problem entirely: we filter on top of SWIM, deterministically, rather than trying to convince the failure detector that a still-gossiping node is dead.
• Keying on the per-process host_id means the ban auto-clears when a fresh process restarts with a new UUID, so it targets this specific bad incarnation rather than banning an address forever.

Describe alternatives you've considered

• Kill the process / kubectl delete pod — impossible here: the k8s node was NotReady, so the container kept running.
• Terminate the instance at the cloud layer — works, but out-of-band and requires infra access.
• Network-isolate the host's membership port — works, but out-of-band and requires network/CNI or firewall access.
• Delete the cluster_membership row — no-op on a running cluster (DB read only at bootstrap), and the node re-inserts it within ~10–15s anyway.
• tdbg shard close-shard — pointless while the host is still in the ring; consistent hashing just reassigns the shard back to the same zombie.
• Mark the node draining — doesn't gate the Lookup() ring, so shard ownership is unaffected.

Complementary options (possibly separate issues):

• A self-health watchdog that calls SelfEvict() when the process detects it can no longer serve (e.g. persistence writes failing). This would auto-handle self-detectable failures like ours, but can't cover a fully wedged process — so an operator override is still needed as a backstop.
• Making draining (or a new unhealthy label) actually gate the Lookup() routing ring, not just AvailableMembers().

Additional context

• Reported on Temporal Server v1.31.2. Code paths referenced below (symbol names) were confirmed against a recent server checkout; this membership code is materially unchanged on the v1.31.x line:
  • common/membership/ringpop/monitor.go — startHeartbeatUpsertLoop,
    EvictSelf/EvictSelfAt, fetchCurrentBootstrapHostports,
    healthyHostLastHeartbeatCutoff = 20s, upsertMembershipRecordExpiryDefault = 48h
  • common/membership/ringpop/service_resolver.go — getReachableMembers
    (GetReachableMemberObjects), Lookup, isDraining / AvailableMembers
  • service/history/shard/ownership.go — shard ownership via Lookup
• Environment: Temporal Server v1.31.2; Kubernetes; Cassandra persistence.

I'd be happy to contribute a PR for this if the team is open to the approach.

[dnr]

• Because every node reads the same denylist, all rings exclude the host consistently — there is no window where some nodes still route to it

This is a pretty critical point and it seems wrong: neither dynamic config nor persistence (polled on some interval) can cause all hosts to start excluding some specific node at exactly the same time. There will definitely be a window where they're out of sync, and any more than a few seconds in that window will be pretty chaotic for availability.

It seems like this sort of feature needs to be built into ringpop, so any denylist is propagated through the same mechanism at least as fast as the rest of the membership information. Unfortunately ringpop is more or less in maintenance mode.

I think the watchdog approach may be the most promising overall. Or a less intrusive ringpop integration: a health check from ringpop to the application in the ping path, so it can't respond to pings without at least some of the application being alive.