Repository navigation
Report poller target on each poll and in the worker heartbeat - #1688
Open
veeral-patel wants to merge 2 commits into
Open
veeral-patel wants to merge 2 commits into
veeral-patel wants to merge 2 commits into
Conversation
An autoscaling poller pool now sends its current target with every workflow and activity poll, so the server can spread scaling suggestions across a fleet by share rather than by whoever happens to receive the most tasks. Suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's share of suggestions tracks its share of pollers -- growth is proportional to current size, which preserves imbalance between identical workers instead of correcting it. A reported target gives the server the one number it needs to compare a worker against the fleet mean. The target is read off the report handle at the poll call site, alongside the timeout override already derived from it. Fixed-size pools report nothing, which the server reads as "no report" and excludes from fairness.
Sushisource
approved these changes
Oct 20, 2026
Comment on lines
+638
to
+643
| /// The pool's current scaling target, reported to the server on each poll so it can | ||
| /// spread poller-scaling suggestions across a fleet by share rather than letting them | ||
| /// land on whichever worker happens to receive the most tasks. | ||
| /// | ||
| /// `None` for a fixed-size pool: it has no target to balance, and the server leaves | ||
| /// such a worker out of fairness entirely rather than assuming anything about it. |
Member
There was a problem hiding this comment.
Suggested change
| /// The pool's current scaling target, reported to the server on each poll so it can | |
| /// spread poller-scaling suggestions across a fleet by share rather than letting them | |
| /// land on whichever worker happens to receive the most tasks. | |
| /// | |
| /// `None` for a fixed-size pool: it has no target to balance, and the server leaves | |
| /// such a worker out of fairness entirely rather than assuming anything about it. | |
| /// The pool's current scaling target. `None` for a fixed-size pool. |
Comment on lines
+449
to
+452
| /// The poller pool's current scaling target, reported to the server so it can spread | ||
| /// poller-scaling suggestions across a fleet by share rather than letting them land on | ||
| /// whichever worker happens to receive the most tasks. `None` for a fixed-size pool, | ||
| /// which has no target to balance; the server then leaves this worker out of fairness. |
Member
There was a problem hiding this comment.
Suggested change
| /// The poller pool's current scaling target, reported to the server so it can spread | |
| /// poller-scaling suggestions across a fleet by share rather than letting them land on | |
| /// whichever worker happens to receive the most tasks. `None` for a fixed-size pool, | |
| /// which has no target to balance; the server then leaves this worker out of fairness. | |
| /// The pool's current scaling target. `None` for a fixed-size pool. |
veeral-patel
marked this pull request as ready for review
October 20, 2026 20:11
The poller target is already sent on each poll, but matching consumes it inside the scaling decision and discards it -- nothing persists it and nothing can observe it. Reporting it on the heartbeat too lets the server read it per worker and export it as a metric, making fleet-wide poller distribution visible without instrumenting every SDK. WorkerHeartbeat already carries four WorkerPollerInfo sub-messages, so this is per-task-type for free, including the normal-vs-sticky workflow split. The heartbeat closure is constructed before any poller exists, so it cannot capture the scaling handles. That is the same ordering problem last_successful_poll_time already solves, and this copies its solution: a cell per poller type is created up front, cloned into both HeartbeatMetrics and the poller, and the report handle publishes its target into it on every change. Nexus is included even though its poll request has no field for a target -- the heartbeat covers every autoscaling pool, not only the pools fairness acts on. Fixed-size pools publish None, which the proto defines as unknown.
veeral-patel
force-pushed
the
vpatel/poller-load-balancing
branch
from
October 22, 2026 03:41
3d75d2b to
464e07f
Compare
Sushisource
requested changes
Oct 22, 2026
Comment on lines
801
to
+811
| let wf_last_suc_poll_time = Arc::new(AtomicCell::new(None)); | ||
| let wf_sticky_last_suc_poll_time = Arc::new(AtomicCell::new(None)); | ||
| let act_last_suc_poll_time = Arc::new(AtomicCell::new(None)); | ||
| let nexus_last_suc_poll_time = Arc::new(AtomicCell::new(None)); | ||
| // Autoscaler target per poller type, reported on the heartbeat so the server can | ||
| // export poller distribution as a metric. Pre-created here for the same reason as | ||
| // the poll-time cells above: the heartbeat closure is built before any poller exists. | ||
| let wf_target = Arc::new(AtomicCell::new(None)); | ||
| let wf_sticky_target = Arc::new(AtomicCell::new(None)); | ||
| let act_target = Arc::new(AtomicCell::new(None)); | ||
| let nexus_target = Arc::new(AtomicCell::new(None)); |
Member
There was a problem hiding this comment.
There are enough of these now that all get passed together that it would be good to roll this up into one struct and avoid passing a bajillion of these at each call site.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An autoscaling poller pool now reports its current target in two places:
PollerScalingInfo.poller_targeton the workflow and activity poll requests.WorkerPollerInfo.poller_target, for all four poller types.Why
Scaling suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's share of suggestions tracks its share of pollers. Growth is proportional to current size, which preserves imbalance between identical workers instead of correcting it — measured at 74/12/14 percent across three identical workers, with a different winner each run.
Reporting the target gives the server the one number it needs to compare a worker against the fleet mean and withhold suggestions that would push it further from its share.
The poll-request copy is consumed inside the matching decision and discarded. Reporting it on the heartbeat too lets a follow-up read it per worker and export it as a metric, making fleet-wide poller distribution visible without instrumenting every SDK.
How
Poll path —
PollScalerReportHandle::reportable_target()is read at the poll call site inpoll_buffer.rs, alongside the timeout override already derived from the same handle, and threaded throughpoll_fnintoPollOptions.Heartbeat — the heartbeat closure is built in
WorkerHeartbeatManager::newbefore any poller exists, since pollers are created in a deferred closure once namespace capabilities are known. It therefore cannot capture the scaling handles.That is the same ordering problem
last_successful_poll_timealready solves, and this copies its solution: a cell per poller type is created up front inWorker::new_with_pollers, cloned into bothHeartbeatMetricsand the poller, and the report handle publishes its target into it on construction and on every change.Design notes
reportable_target()returnsNonefor a fixed-size pool rather than its static size: such a pool has no target to balance, and the server leaves that worker out of fairness entirely rather than assuming anything about it. That reads as 0 on the heartbeat, which the proto defines as unknown.poll_nexus_taskleavespoller_targetunused because the nexus request has no such field, but nexus has its ownPollScalerReportHandleand a heartbeat slot, so its target is reported there.Dependency
Mirrors the proto change in temporalio/api#887. Because the API protos are vendored in-tree under
crates/protos/protos/api_upstream, the.protoedits are included here sobuild.rsregenerates and the branch compiles standalone. Those vendored changes should come from a normal api sync before merge, not from this PR.Testing
cargo check -p temporalio-sdk-core --all-targetspasses and the 15pollers::unit tests pass. Not yet exercised end to end against a server — that has been done via the Go SDK (temporalio/sdk-go#2799), which shares the wire contract.