feat(frontend): support sink backfill rate limit #24075

kwannoel · 2025-12-10T10:39:56Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Decouple ThrottleTarget into ThrottleTarget and ThrottleType. ThrottleType refers to the type of rate limit we want to apply. ThrottleTarget refers to the actual object we are targeting (sink, index, fragment, etc...).
This change is required so we can alter both backfill and sink rate limit for sinks.
This is because the sink executor and the scan executor can lie on the same fragment, so we need to provide the throttle type at runtime, through the barrier mutation, to make sure only the rate limit type we are targeting gets altered.
Proto breaking changes are just for the RPC layer, and do not change the persisted state of rate limit in fragments.

I avoided doing mass renaming of Throttle -> RateLimit, but we should make this change subsequently for consistency
I also avoided specializing alter fragment rate limit. After this PR we can have another one to do the specialization to different rate limit types.

Checklist

I have written necessary rustdoc comments.
I have added necessary unit tests and integration tests.
I have added test labels as necessary.
I have added fuzzing tests or opened an issue to track them.
My PR contains breaking changes.
My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

Documentation

My PR needs documentation updates.

Release note

Copilot

Pull request overview

This PR introduces support for separate backfill rate limiting on sinks by decoupling ThrottleTarget (what to throttle) from ThrottleType (how to throttle). This allows users to independently control both the sink execution rate and the sink backfill rate.

Key Changes:

Introduced ThrottleType enum to distinguish between DML, Backfill, Source, and Sink rate limit types
Added ALTER SINK SET BACKFILL_RATE_LIMIT SQL syntax support
Updated all executors to check throttle_type before applying rate limits
Modified RPC interfaces to accept both throttle_target and throttle_type parameters

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
proto/meta.proto	Added `ThrottleType` enum and updated `ApplyThrottleRequest` to include both target and type
proto/stream_plan.proto	Added nested `ThrottleType` enum in `ThrottleMutation` and added `throttle_type` field
src/sqlparser/src/ast/ddl.rs	Added `SetBackfillRateLimit` variant to `AlterSinkOperation` enum
src/sqlparser/src/parser.rs	Added parsing logic for `ALTER SINK SET BACKFILL_RATE_LIMIT` syntax
src/frontend/src/meta_client.rs	Updated `apply_throttle` method signature to accept both target and type
src/frontend/src/test_utils.rs	Updated mock implementation to match new signature
src/frontend/src/handler/mod.rs	Added handler routing for sink backfill rate limit operations
src/frontend/src/handler/alter_streaming_rate_limit.rs	Updated logic to handle different throttle target/type combinations
src/rpc_client/src/meta_client.rs	Updated `apply_throttle` to pass both target and type parameters
src/meta/service/src/stream_service.rs	Updated throttle application logic to match on (type, target) tuples
src/meta/service/src/ddl_service.rs	Updated to pass `ThrottleType::Source` when creating throttle commands
src/meta/src/barrier/command.rs	Modified `Command::Throttle` to include `ThrottleType` parameter
src/meta/src/barrier/info.rs	Updated pattern matching for throttle commands
src/meta/src/barrier/context/context_impl.rs	Updated pattern matching for throttle commands
src/stream/src/executor/mod.rs	Updated `Mutation::Throttle` to include `throttle_type` field and conversion logic
src/stream/src/executor/sink.rs	Added throttle type check to only apply sink rate limits
src/stream/src/executor/dml.rs	Added throttle type check to only apply DML rate limits
src/stream/src/executor/source/source_executor.rs	Added throttle type check to only apply source rate limits
src/stream/src/executor/source/source_backfill_executor.rs	Added throttle type check to only apply backfill rate limits
src/stream/src/executor/source/iceberg_fetch_executor.rs	Added throttle type check to only apply source rate limits
src/stream/src/executor/source/fs_fetch_executor.rs	Added throttle type check to only apply source rate limits
src/stream/src/executor/backfill/no_shuffle_backfill.rs	Added throttle type check to only apply backfill rate limits
src/stream/src/executor/backfill/arrangement_backfill.rs	Added throttle type check to only apply backfill rate limits
src/stream/src/executor/backfill/cdc/cdc_backfill.rs	Added throttle type check to only apply backfill rate limits
src/stream/src/executor/backfill/cdc/cdc_backill_v2.rs	Added throttle type check to only apply backfill rate limits
src/ctl/src/cmd_impl/throttle.rs	Added logic to infer throttle type from target for backward compatibility

proto/stream_plan.proto

Copilot · 2025-12-10T13:14:31Z

src/stream/src/executor/backfill/cdc/cdc_backill_v2.rs

    CdcOffset, ExternalCdcTableType, ExternalTableReaderImpl,
 };
 use risingwave_connector::source::{CdcTableSnapshotSplit, CdcTableSnapshotSplitRaw};
+use risingwave_pb::meta::ThrottleType;


For consistency with other executor files in this PR, consider importing ThrottleType from crate::executor rather than directly from risingwave_pb::meta. Most other executors use use crate::executor::ThrottleType; (see source_executor.rs, sink.rs, etc.).

Suggested change

use risingwave_pb::meta::ThrottleType;

use crate::executor::ThrottleType;

kwannoel · 2025-12-10T15:04:27Z

@copilot please submit a pr to fix the review comments.

Copilot · 2025-12-10T15:04:35Z

@kwannoel I've opened a new pull request, #24083, to work on those changes. Once the pull request is ready, I'll request review from you.

kwannoel · 2025-12-12T08:16:11Z

src/ctl/src/cmd_impl/throttle.rs

 // limitations under the License.

+use risingwave_pb::common::PbThrottleType;
 use risingwave_pb::meta::PbThrottleTarget;

 use crate::ThrottleCommandArgs;
 use crate::common::CtlContext;

 pub async fn apply_throttle(
    context: &CtlContext,
    kind: PbThrottleTarget,
    params: ThrottleCommandArgs,
 ) -> anyhow::Result<()> {
    let meta_client = context.meta_client().await?;
+    // Infer throttle type from target for backward compatibility.
+    let throttle_type = match kind {
+        PbThrottleTarget::Source => PbThrottleType::Source,
+        PbThrottleTarget::Mv => PbThrottleType::Backfill,
+        PbThrottleTarget::Sink => PbThrottleType::Sink,
+        PbThrottleTarget::Table | PbThrottleTarget::Fragment | PbThrottleTarget::Unspecified => {
+            // Default to Backfill for unspecified/unsupported combinations; user should use SQL for table throttling
+            PbThrottleType::Backfill
+        }
+    };
    meta_client
-        .apply_throttle(kind, params.id, params.rate)
+        .apply_throttle(kind, throttle_type, params.id, params.rate)
        .await?;
    Ok(())


This should be fixed, we should supply both throttle type and throttle target.

@copilot please help to update the ctl interface to support this behaviour. Add a new cli option for this.

Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: kwannoel <[email protected]>

Copilot · 2025-12-15T05:38:28Z

@kwannoel I've opened a new pull request, #24111, to work on those changes. Once the pull request is ready, I'll request review from you.

github-actions bot added type/feature Type: New feature. ci/run-e2e-single-node-tests ci/run-e2e-test-other-backends labels Dec 10, 2025

kwannoel changed the title ~~feat(frontend): support alter sink rate limit~~ feat(frontend): support sink backfill rate limit Dec 10, 2025

github-actions bot added ci/run-s3-source-tests ci/main-cron/run-selected ci/run-e2e-iceberg-tests labels Dec 10, 2025

kwannoel requested review from BugenZhao, Copilot, tabVersion, wenym1 and zwang28 December 10, 2025 12:48

Copilot started reviewing on behalf of kwannoel December 10, 2025 13:06 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

Copilot AI mentioned this pull request Dec 10, 2025

refactor: consolidate ThrottleType enum to common.proto #24083

Merged

8 tasks

kwannoel marked this pull request as draft December 12, 2025 08:15

kwannoel added the ci/run-e2e-tests label Dec 12, 2025

kwannoel commented Dec 12, 2025

View reviewed changes

kwannoel and others added 10 commits December 15, 2025 12:18

decouple throttle target and type in the frontend

1d19125

support alter backfill rate limit for sink

3ed16c2

explicitly match on the throttle type in every executor

7f46061

permit altering backfill rate limit for sink into table

db579d4

reuse alter handler

fdc9c26

fix id for throttle table with source

69acd91

refactor: consolidate ThrottleType enum to common.proto (#24083)

f3a0cee

Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: kwannoel <[email protected]>

fmt

cb54c60

avoid manual conversion

ced4a66

normalize rate limit proto

3c0155a

kwannoel force-pushed the kwannoel/refactor-rate-limit branch from 8757617 to 3c0155a Compare December 15, 2025 05:29

Copilot AI mentioned this pull request Dec 15, 2025

feat(ctl): add required throttle type option for independent target/type control #24111

Draft

8 tasks

fix test

0f37591

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(frontend): support sink backfill rate limit #24075

feat(frontend): support sink backfill rate limit #24075

Uh oh!

kwannoel commented Dec 10, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Dec 10, 2025

Uh oh!

kwannoel commented Dec 10, 2025

Uh oh!

Copilot AI commented Dec 10, 2025

Uh oh!

kwannoel Dec 12, 2025

Uh oh!

kwannoel Dec 15, 2025

Uh oh!

Copilot AI commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	use risingwave_pb::meta::ThrottleType;
	use crate::executor::ThrottleType;

feat(frontend): support sink backfill rate limit #24075

Are you sure you want to change the base?

feat(frontend): support sink backfill rate limit #24075

Uh oh!

Conversation

kwannoel commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's changed and what's your intention?

Checklist

Documentation

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Dec 10, 2025

Choose a reason for hiding this comment

Uh oh!

kwannoel commented Dec 10, 2025

Uh oh!

Copilot AI commented Dec 10, 2025

Uh oh!

kwannoel Dec 12, 2025

Choose a reason for hiding this comment

Uh oh!

kwannoel Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kwannoel commented Dec 10, 2025 •

edited

Loading