Skip to content

feat: add pk-index compaction resolver and apply stream operators - #26564

Open
silver-ymz wants to merge 1 commit into
mainfrom
silver/pk-index-compaction-operators
Open

feat: add pk-index compaction resolver and apply stream operators#26564
silver-ymz wants to merge 1 commit into
mainfrom
silver/pk-index-compaction-operators

Conversation

@silver-ymz

@silver-ymz silver-ymz commented Aug 4, 2026

Copy link
Copy Markdown
Member

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

Stack: PR 3 of 6 for iceberg pk-index compaction (← #26563 | → #26565)

Adds two new stream executors that form the core pk-index compaction operator pipeline, plus the proto definitions they need.

  • CompactionResolverExecutor: resolves which position delete files to compact based on delete file count thresholds and partition grouping
  • CompactionApplyExecutor: receives compaction task outputs and rewrites data files by merging position deletes into new data files
  • from_proto/compaction_resolver.rs: proto deserialization for the new executor
  • proto/stream_plan.proto: add IcebergPkIndexBarrierMutation and new stream node types
  • proto/stream_service.proto: add PB_ICEBERG_PK_INDEX_SINK_ROLE_COMPACTION_RESOLVER
  • executor/mod.rs: register new NodeBody variants. IcebergPkIndexBarrierMutation is used for the next PR.

The executors are defined but not yet wired into the task system — PR 4 does that.

Checklist

  • I have written necessary rustdoc comments.
  • I have added necessary unit tests and integration tests.
  • I have added test labels as necessary.
  • I have added fuzzing tests or opened an issue to track them.
  • My PR contains breaking changes.
  • My PR changes performance-critical code, so I will run (micro) benchmarks and present the results.
  • I have checked the Release Timeline and Currently Supported Versions to determine which release branches I need to cherry-pick this PR into.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@silver-ymz
silver-ymz changed the base branch from silver/pk-index-compaction-refactor to graphite-base/26564 August 5, 2026 04:46
@chenzl25
chenzl25 self-requested a review August 5, 2026 06:09
@silver-ymz
silver-ymz force-pushed the graphite-base/26564 branch from 192b7bc to 1a9401a Compare August 5, 2026 06:41
@silver-ymz
silver-ymz requested a review from a team as a code owner August 5, 2026 06:41
@silver-ymz
silver-ymz force-pushed the silver/pk-index-compaction-operators branch from 7bebb6e to bd91e4d Compare August 5, 2026 06:41
@silver-ymz
silver-ymz changed the base branch from graphite-base/26564 to silver/pk-index-compaction-refactor August 5, 2026 06:41
Base automatically changed from silver/pk-index-compaction-refactor to main August 5, 2026 06:42
@chenzl25
chenzl25 requested review from Li0k and wenym1 August 5, 2026 07:19
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
@silver-ymz
silver-ymz force-pushed the silver/pk-index-compaction-operators branch from bd91e4d to cc13ded Compare August 5, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant