Watchdog reports a schedule healthy while its run is wedged forever

[jorgecuesta]

Watchdog reports a schedule healthy while its run is wedged forever

The blind spot

evaluateLiveness returns early on the first condition it checks after the pause guard:

// packages/temporal/src/scheduleWatchdog.ts:167
if (desc.info.runningActions.length > 0) return 'healthy' // a run is in flight -> scheduler alive

A workflow whose workflow task fails permanently stays Running forever — Temporal retries the task indefinitely rather than failing the execution. That keeps runningActions non-empty, so the schedule is judged healthy for as long as the stall lasts, and the heal ladder is never entered.

Because mapScheduleToHealth derives the UI state from the same verdict (workflowView.ts:298), the Workflows page shows the schedule green throughout.

What it cost

On mainnet, VerifyPendingTransactions-scheduled sat wedged from 2026-08-10 13:34:30Z for three days. With ScheduleOverlapPolicy: SKIP, every 30s firing in that window was skipped — SkippedOverlap reached 9533 — so no transaction was verified at all. Five stake transactions stayed pending, their supplier rows were never created, and the outage surfaced only because a user asked why their supplier still showed as pending.

Throughout those three days the watchdog considered that schedule healthy, and nothing paged.

Why #338 does not close this

#338 removes one cause: workflows now throw ApplicationFailure instead of WorkflowError, so a systemic failure ends the execution instead of wedging the task. Its regression guard greps for new WorkflowError(.

The blind spot is wider than that one cause. Any of these reproduce the identical silent stall, and none is covered:

• a nondeterminism error after a workflow-code deploy changes a workflow's shape mid-run,
• an activity name that no longer resolves in the worker,
• a payload that exceeds the blob size limit,
• any other permanently-failing workflow task.

Proposal

Have evaluateLiveness stop treating the presence of a running action as proof of life. A running action older than K × the schedule interval should be judged stale rather than healthy, so the existing heal ladder and the existing UI state both start reacting.

Worth deciding as part of it:

• what K should be, given a legitimate sweep can now run for several minutes (see the run-time bound discussed in fix: stop transient RPC errors from wedging the tx verifier #338),
• whether a stalled run should be surfaced distinctly from a schedule that stopped firing, since the remedy differs: a wedged run needs terminating, a dead scheduler needs re-arming,
• whether TemporalReportedProblems (the search attribute that carried category=WorkflowTaskFailed on the wedged run) is a cheaper signal than an age heuristic.

Context

Found while investigating the mainnet incident of 2026-08-10. The immediate cause is fixed in #338; this is the detection gap that let it run for three days.