Skip to content

Report poller target on each poll and in the worker heartbeat - #1688

Open
veeral-patel wants to merge 2 commits into
mainfrom
vpatel/poller-load-balancing
Open

veeral-patel wants to merge 2 commits into
mainfrom
vpatel/poller-load-balancing

Conversation

@veeral-patel

@veeral-patel veeral-patel commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

What

An autoscaling poller pool now reports its current target in two places:

  1. On each poll — PollerScalingInfo.poller_target on the workflow and activity poll requests.
  2. On the worker heartbeat — WorkerPollerInfo.poller_target, for all four poller types.

Why

Scaling suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's share of suggestions tracks its share of pollers. Growth is proportional to current size, which preserves imbalance between identical workers instead of correcting it — measured at 74/12/14 percent across three identical workers, with a different winner each run.

Reporting the target gives the server the one number it needs to compare a worker against the fleet mean and withhold suggestions that would push it further from its share.

The poll-request copy is consumed inside the matching decision and discarded. Reporting it on the heartbeat too lets a follow-up read it per worker and export it as a metric, making fleet-wide poller distribution visible without instrumenting every SDK.

How

Poll path — PollScalerReportHandle::reportable_target() is read at the poll call site in poll_buffer.rs, alongside the timeout override already derived from the same handle, and threaded through poll_fn into PollOptions.

Heartbeat — the heartbeat closure is built in WorkerHeartbeatManager::new before any poller exists, since pollers are created in a deferred closure once namespace capabilities are known. It therefore cannot capture the scaling handles.

That is the same ordering problem last_successful_poll_time already solves, and this copies its solution: a cell per poller type is created up front in Worker::new_with_pollers, cloned into both HeartbeatMetrics and the poller, and the report handle publishes its target into it on construction and on every change.

Design notes

  • reportable_target() returns None for a fixed-size pool rather than its static size: such a pool has no target to balance, and the server leaves that worker out of fairness entirely rather than assuming anything about it. That reads as 0 on the heartbeat, which the proto defines as unknown.
  • Nexus is in the heartbeat but not the poll path. poll_nexus_task leaves poller_target unused because the nexus request has no such field, but nexus has its own PollScalerReportHandle and a heartbeat slot, so its target is reported there.

Dependency

Mirrors the proto change in temporalio/api#887. Because the API protos are vendored in-tree under crates/protos/protos/api_upstream, the .proto edits are included here so build.rs regenerates and the branch compiles standalone. Those vendored changes should come from a normal api sync before merge, not from this PR.

Testing

cargo check -p temporalio-sdk-core --all-targets passes and the 15 pollers:: unit tests pass. Not yet exercised end to end against a server — that has been done via the Go SDK (temporalio/sdk-go#2799), which shares the wire contract.

An autoscaling poller pool now sends its current target with every workflow and
activity poll, so the server can spread scaling suggestions across a fleet by
share rather than by whoever happens to receive the most tasks.

Suggestions ride on dispatched tasks and the matcher is FIFO, so a worker's
share of suggestions tracks its share of pollers -- growth is proportional to
current size, which preserves imbalance between identical workers instead of
correcting it. A reported target gives the server the one number it needs to
compare a worker against the fleet mean.

The target is read off the report handle at the poll call site, alongside the
timeout override already derived from it. Fixed-size pools report nothing,
which the server reads as "no report" and excludes from fairness.

@Sushisource Sushisource left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

De llm-ify

Comment on lines +638 to +643
/// The pool's current scaling target, reported to the server on each poll so it can
/// spread poller-scaling suggestions across a fleet by share rather than letting them
/// land on whichever worker happens to receive the most tasks.
///
/// `None` for a fixed-size pool: it has no target to balance, and the server leaves
/// such a worker out of fairness entirely rather than assuming anything about it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The pool's current scaling target, reported to the server on each poll so it can
/// spread poller-scaling suggestions across a fleet by share rather than letting them
/// land on whichever worker happens to receive the most tasks.
///
/// `None` for a fixed-size pool: it has no target to balance, and the server leaves
/// such a worker out of fairness entirely rather than assuming anything about it.
/// The pool's current scaling target. `None` for a fixed-size pool.

Comment on lines +449 to +452
/// The poller pool's current scaling target, reported to the server so it can spread
/// poller-scaling suggestions across a fleet by share rather than letting them land on
/// whichever worker happens to receive the most tasks. `None` for a fixed-size pool,
/// which has no target to balance; the server then leaves this worker out of fairness.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/// The poller pool's current scaling target, reported to the server so it can spread
/// poller-scaling suggestions across a fleet by share rather than letting them land on
/// whichever worker happens to receive the most tasks. `None` for a fixed-size pool,
/// which has no target to balance; the server then leaves this worker out of fairness.
/// The pool's current scaling target. `None` for a fixed-size pool.

@veeral-patel
veeral-patel marked this pull request as ready for review October 20, 2026 20:11
@veeral-patel
veeral-patel requested a review from a team as a code owner October 20, 2026 20:11
@veeral-patel veeral-patel changed the title Report poller target on each poll Report poller target on each poll and in the worker heartbeat Oct 22, 2026
The poller target is already sent on each poll, but matching consumes it inside
the scaling decision and discards it -- nothing persists it and nothing can
observe it. Reporting it on the heartbeat too lets the server read it per worker
and export it as a metric, making fleet-wide poller distribution visible without
instrumenting every SDK.

WorkerHeartbeat already carries four WorkerPollerInfo sub-messages, so this is
per-task-type for free, including the normal-vs-sticky workflow split.

The heartbeat closure is constructed before any poller exists, so it cannot
capture the scaling handles. That is the same ordering problem
last_successful_poll_time already solves, and this copies its solution: a cell
per poller type is created up front, cloned into both HeartbeatMetrics and the
poller, and the report handle publishes its target into it on every change.

Nexus is included even though its poll request has no field for a target -- the
heartbeat covers every autoscaling pool, not only the pools fairness acts on.
Fixed-size pools publish None, which the proto defines as unknown.
@veeral-patel
veeral-patel force-pushed the vpatel/poller-load-balancing branch from 3d75d2b to 464e07f Compare October 22, 2026 03:41
Comment on lines 801 to +811
let wf_last_suc_poll_time = Arc::new(AtomicCell::new(None));
let wf_sticky_last_suc_poll_time = Arc::new(AtomicCell::new(None));
let act_last_suc_poll_time = Arc::new(AtomicCell::new(None));
let nexus_last_suc_poll_time = Arc::new(AtomicCell::new(None));
// Autoscaler target per poller type, reported on the heartbeat so the server can
// export poller distribution as a metric. Pre-created here for the same reason as
// the poll-time cells above: the heartbeat closure is built before any poller exists.
let wf_target = Arc::new(AtomicCell::new(None));
let wf_sticky_target = Arc::new(AtomicCell::new(None));
let act_target = Arc::new(AtomicCell::new(None));
let nexus_target = Arc::new(AtomicCell::new(None));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are enough of these now that all get passed together that it would be good to roll this up into one struct and avoid passing a bajillion of these at each call site.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants