Skip to content

fix(batch): mask serving workers with version mismatch - #26559

Open
zwang28 wants to merge 9 commits into
mainfrom
nukualofa
Open

fix(batch): mask serving workers with version mismatch#26559
zwang28 wants to merge 9 commits into
mainfrom
nukualofa

Conversation

@zwang28

@zwang28 zwang28 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This PR improves availability during rolling upgrades when serving nodes run differing RW versions.

Performance overhead justification:

This PR adds scheduling-time overhead only. For non-barrier-read worker selection, it now scans the current serving worker list and builds a small local HashSet<WorkerId> combining temporary masks with workers whose reported rw_version differs from RW_VERSION.

The added cost is O(number_of_serving_workers) per affected selector call, plus one read lock on the existing temporary mask. There is no new persistent cached mask and no repeated write-lock churn. Query execution inner loops are unaffected; the extra work happens during planning/scheduling paths such as fragment_mapping, schedule_unit_count, worker_node_count, and next_random_worker.

When a mismatch exists, the existing place_vnode remapping path is triggered, which is heavier but intentional to avoid scheduling serving reads to incompatible workers.

Details:

  • Mask serving workers whose reported RisingWave version differs from the current RW_VERSION before remapping serving fragment mappings.
  • Build the effective mask locally from the temporary failure mask and current serving-worker versions, avoiding repeated write-lock churn.
  • Add unit coverage for current-version workers, mismatched-version workers, missing or empty versions, and temporary-mask coexistence.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

@github-actions github-actions Bot added the type/fix Type: Bug fix. Only for pull requests. label Aug 4, 2026
@zwang28 zwang28 added the A-batch Area: Batch engine. label Aug 4, 2026
@zwang28
zwang28 requested review from Li0k and chenzl25 August 4, 2026 04:01
@zwang28
zwang28 requested a review from hzxa21 August 4, 2026 04:22

@chenzl25 chenzl25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found two blocking issues on the current head:

origin: Vec<WorkerNode>,
worker_node_mask: &HashSet<WorkerId>,
) -> Vec<WorkerNode> {
if origin.iter().all(|w| worker_node_mask.contains(&w.id)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this fallback now also bypasses the version check. If every serving worker has a different/empty version, or the only matching-version worker is temporarily masked, all(...) is true and we return the original list, making incompatible workers schedulable again. Please keep version mismatch fail-closed and apply the “all masked” fallback only to the temporary mask within the matching-version subset. Please also cover both cases in tests.

@zwang28 zwang28 Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW for a typical serving node (frontend + serving role compute node), at least 1 matched serving node is guaranteed.

fn effective_worker_node_mask(&self, serving_worker_nodes: &[WorkerNode]) -> HashSet<WorkerId> {
let mut mask = self.worker_node_mask.read().unwrap().clone();
mask.extend(serving_worker_nodes.iter().filter_map(|worker| {
(worker.resource.as_ref().map(|r| r.rw_version.as_str()) != Some(RW_VERSION))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: resource.rw_version is not guaranteed to be current in the frontend snapshot. WorkerNode.resource is not persisted; after meta restarts, existing workers are rebuilt with default resource data, while heartbeat updates only meta memory and the frontend handles only Add/Delete node notifications. A frontend snapshot taken before the first heartbeat can therefore retain an empty version and mask healthy workers indefinitely. Please make version updates durable/observable before using this field as a scheduling filter.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will make WorkerNode.resource durable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zwang28 zwang28 added A-meta Area: Meta node. A-frontend Area: Protocol, parsing, binder. ci/run-e2e-single-node-tests labels Aug 4, 2026

@chenzl25 chenzl25 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zwang28

zwang28 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Will merge this PR after #26568

# Conflicts:
#	src/batch/src/worker_manager/worker_node_manager.rs
#	src/meta/src/controller/cluster.rs
@zwang28
zwang28 enabled auto-merge August 4, 2026 12:57
@zwang28
zwang28 disabled auto-merge August 4, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-batch Area: Batch engine. A-frontend Area: Protocol, parsing, binder. A-meta Area: Meta node. ci/run-e2e-single-node-tests need-cherry-pick-since-release-3.0 type/fix Type: Bug fix. Only for pull requests.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants