Skip to content

feat(frontend): support await-tree dump - #26552

Open
chenzl25 wants to merge 1 commit into
mainfrom
dylan/frontend-await-tree-dump
Open

feat(frontend): support await-tree dump#26552
chenzl25 wants to merge 1 commit into
mainfrom
dylan/frontend-await-tree-dump

Conversation

@chenzl25

@chenzl25 chenzl25 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

This is stacked on #26515, which adds await-tree roots for batch tasks on compute nodes. This PR completes the frontend side of the dump path so risectl await-tree dump can show the query driver and local-mode frontend execution as well.

  • Create an optional frontend await-tree registry using the existing streaming.async_stack_trace setting; no new configuration is introduced.
  • Register the full local query execution future as a Local Query root. Await spans from the local executor, including local exchange waits added by feat(batch): support await-tree dump for batch tasks #26515, are therefore attached to the same tree.
  • Register the distributed query scheduler driver as a Distributed Query root.
  • Implement MonitorService::stack_trace on frontend nodes and return the active query trees in a new frontend_traces field.
  • Have meta enumerate frontend workers, collect their monitor responses through the existing frontend internal RPC address, and merge them into the cluster dump.
  • Render the result as a new --- Frontend Traces --- section in risectl await-tree dump.

When async stack tracing is disabled, the frontend monitor service returns an empty trace map.

Testing

  • cargo check -p risingwave_pb
  • cargo check -p risingwave_frontend -p risingwave_meta -p risingwave_ctl
  • cargo check -p risingwave_compute
  • cargo clippy -p risingwave_frontend -p risingwave_meta -p risingwave_ctl -p risingwave_compute --lib -- -D warnings
  • cargo sort --check --workspace --grouped
  • cargo fmt --all
  • git diff --check

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

  • My PR needs documentation updates.
Release note

@chenzl25
chenzl25 requested a review from a team as a code owner August 3, 2026 09:05
@chenzl25
chenzl25 requested review from wenym1 and removed request for a team August 3, 2026 09:05

chenzl25 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@chenzl25 chenzl25 added A-frontend Area: Protocol, parsing, binder. A-meta Area: Meta node. user-facing-changes Contains changes that are visible to users labels Aug 3, 2026

@wenym1 wenym1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that we don't have any newly added instrument logic. Do we already have instrument_await in frontend code, but just not enabled because we didn't register the await tree root in frontend?

Comment thread src/frontend/src/scheduler/local.rs Outdated
@chenzl25
chenzl25 force-pushed the dylan/frontend-await-tree-dump branch from f54fe03 to f801fa8 Compare August 3, 2026 11:09
@chenzl25

chenzl25 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

@wenym1 You are right—there was no existing instrument_await in frontend. I pushed f801fa8afa to add the missing instrumentation:

  • query drivers now show wait_stage_event;
  • every StageRunner is registered as its own frontend await-tree root ({query_id}/stage/{stage_id}), since the task-local context does not cross the compute-runtime spawn;
  • stage scheduling/status/shutdown, frontend root/local executor build and poll, result-channel backpressure, and batch exchange execute/get-data RPCs are instrumented.

The local query root can also reuse existing batch/storage child spans. Thanks for catching this.

Base automatically changed from dylan/batch-await-tree-dump to main August 3, 2026 11:34
@chenzl25
chenzl25 force-pushed the dylan/frontend-await-tree-dump branch from f801fa8 to 2691f56 Compare August 4, 2026 06:25
@chenzl25
chenzl25 requested a review from wenym1 August 4, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-frontend Area: Protocol, parsing, binder. A-meta Area: Meta node. ci/run-e2e-single-node-tests type/feature Type: New feature. user-facing-changes Contains changes that are visible to users

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants