Is your feature request related to a problem? Please describe.
Investigating #50 (surface "Unhealthy" version after rollout acceptance test / worker healthcheck fails), I found that today's per-version HealthySince field (api/v1alpha1/workerdeployment_types.go) is computed only from Kubernetes Deployment readiness — internal/controller/state_mapper.go calls k8s.IsDeploymentHealthy(deployment), which checks pod/replica status. It has no visibility into whether Temporal is actually receiving polls from those pods. A Deployment can look perfectly healthy to Kubernetes (all replicas Ready) while its workers are misconfigured, stuck, or unable to reach Temporal — and today the controller has no way to surface that.
Separately, the controller already fetches real poller data from Temporal: internal/temporal/worker_deployment.go's getPollers() calls client.DescribeTaskQueue(...) and gets back the actual poller list per task queue. Today this is used only to compute one internal boolean (AllTaskQueuesHaveUnversionedPoller, a versioning-migration heuristic) — the poller data itself is discarded, never surfaced to the user.
@carlydf's comment on #50 confirms this is in scope now (not gated on the post-GA rollout-acceptance-testing work):
We should surface worker healthcheck error and Gate workflow failure in the TemporalWorkerDeployment events for sure.
Describe the solution you'd like
- Extend
VersionInfo with a poller-health summary derived from the poller data already fetched in getPollers() (no new Temporal API calls needed for the base case).
- Add a new condition type on
WorkerDeploymentStatus (following the existing Reason* constant pattern already in workerdeployment_types.go), e.g. ConditionWorkersHealthy, with reasons:
ReasonPollersHealthy — all expected task queues have active pollers
ReasonNoActivePollers — a version has zero pollers on one or more task queues
ReasonPollerStatusUnknown — couldn't determine (e.g. transient API error) — should not regress to "unhealthy" on a fetch failure
- Emit K8s Events on health-state transitions (not every reconcile, to avoid event spam) using the existing recorder pattern (
r.Recorder.Eventf(workerDeploy, corev1.EventTypeWarning/Normal, reason, "%s", message), already used in worker_controller.go/execplan.go):
Warning when a version loses all pollers on a task queue
Normal when a version's pollers become healthy again
- Gate workflow failure surfacing (explicitly called out by @carlydf above): the controller already tracks
TestWorkflows []WorkflowExecution with status including Failed/Canceled/Terminated/TimedOut. Emit a Warning event referencing the workflow ID when a test/gate workflow transitions into one of these terminal-failure states.
This is purely additive — new condition type, new event emissions, one new struct field — with no change to rollout/reconciliation decision logic (internal/planner/planner.go). Health reporting, not health gating; the post-GA rollout-acceptance-testing work in #50 is a separate, later step that could consume this data but isn't required for it.
Describe alternatives you've considered
Additional context
Also sets up naturally for #61 (add URL field to version metadata) — once real poller/Gate-failure diagnostics exist, encoding workflow URLs to link directly to the failing execution becomes a much smaller follow-up. Not attempting #61's CRD/metadata change in this issue, since it's separately milestoned Post-GA.
Happy to contribute a PR for this if the team is open to the approach.
Is your feature request related to a problem? Please describe.
Investigating #50 (surface "Unhealthy" version after rollout acceptance test / worker healthcheck fails), I found that today's per-version
HealthySincefield (api/v1alpha1/workerdeployment_types.go) is computed only from Kubernetes Deployment readiness —internal/controller/state_mapper.gocallsk8s.IsDeploymentHealthy(deployment), which checks pod/replica status. It has no visibility into whether Temporal is actually receiving polls from those pods. A Deployment can look perfectly healthy to Kubernetes (all replicas Ready) while its workers are misconfigured, stuck, or unable to reach Temporal — and today the controller has no way to surface that.Separately, the controller already fetches real poller data from Temporal:
internal/temporal/worker_deployment.go'sgetPollers()callsclient.DescribeTaskQueue(...)and gets back the actual poller list per task queue. Today this is used only to compute one internal boolean (AllTaskQueuesHaveUnversionedPoller, a versioning-migration heuristic) — the poller data itself is discarded, never surfaced to the user.@carlydf's comment on #50 confirms this is in scope now (not gated on the post-GA rollout-acceptance-testing work):
Describe the solution you'd like
VersionInfowith a poller-health summary derived from the poller data already fetched ingetPollers()(no new Temporal API calls needed for the base case).WorkerDeploymentStatus(following the existingReason*constant pattern already inworkerdeployment_types.go), e.g.ConditionWorkersHealthy, with reasons:ReasonPollersHealthy— all expected task queues have active pollersReasonNoActivePollers— a version has zero pollers on one or more task queuesReasonPollerStatusUnknown— couldn't determine (e.g. transient API error) — should not regress to "unhealthy" on a fetch failurer.Recorder.Eventf(workerDeploy, corev1.EventTypeWarning/Normal, reason, "%s", message), already used inworker_controller.go/execplan.go):Warningwhen a version loses all pollers on a task queueNormalwhen a version's pollers become healthy againTestWorkflows []WorkflowExecutionwith status includingFailed/Canceled/Terminated/TimedOut. Emit aWarningevent referencing the workflow ID when a test/gate workflow transitions into one of these terminal-failure states.This is purely additive — new condition type, new event emissions, one new struct field — with no change to rollout/reconciliation decision logic (
internal/planner/planner.go). Health reporting, not health gating; the post-GA rollout-acceptance-testing work in #50 is a separate, later step that could consume this data but isn't required for it.Describe alternatives you've considered
HealthySince— insufficient, since it can't detect a worker that's running but not actually polling Temporal.Additional context
Also sets up naturally for #61 (add URL field to version metadata) — once real poller/Gate-failure diagnostics exist, encoding workflow URLs to link directly to the failing execution becomes a much smaller follow-up. Not attempting #61's CRD/metadata change in this issue, since it's separately milestoned Post-GA.
Happy to contribute a PR for this if the team is open to the approach.