Skip to content

Add partition-local exact semi/anti filter stage - #13

Draft
VibhuJawa wants to merge 2 commits into
sync/nvidia-main-6fa52e3efrom
agent/partitioned-exact-reducers
Draft

Add partition-local exact semi/anti filter stage#13
VibhuJawa wants to merge 2 commits into
sync/nvidia-main-6fa52e3efrom
agent/partitioned-exact-reducers

Conversation

@VibhuJawa

Copy link
Copy Markdown
Owner

Summary

Adds a focused PartitionedExactFilterStage for exact bucket-local exclusion or membership after two datasets have been partitioned with the existing ShuffleStage.

  • Supports cuDF leftanti and leftsemi joins on one or more exact key columns.
  • Matches each left shuffle task to the reference partition with the same partition_index.
  • Preserves left-row multiplicity even when reference keys are duplicated.
  • Validates complete-partition metadata and schemas before reading data.
  • Rejects row-subsetting reads, multi-file writes, destructive path aliases, and missing reference partitions.
  • Publishes per-partition input, reference, and output row counts in task metadata.

Why this is the missing primitive

Current Curator already provides the compatible hash shuffle. Campaign-specific grouping can be composed from that shuffle plus workload-specific reducers, but Curator does not currently provide the reusable bucket-local exact semi/anti operation needed for scalable exclusion without broadcasting the complete reference set. This PR deliberately adds only that missing reducer and no scheduler, checkpoint, or workload-specific control plane.

Composition

left ShuffleStage(key_fields, N) ---------+
                                           +--> PartitionedExactFilterStage(mode="leftanti" | "leftsemi")
reference ShuffleStage(key_fields, N) ----+

Both shuffles must use identical keys, hashing, and partition count.

Validation

  • 37 focused CPU tests passed; 2 GPU semantics cases are separately marked.
  • git diff --check passed.
  • The three-file diff is compared against exact NVIDIA main commit 6fa52e3e597206849acb2ac119413fa6cbfe644d through non-default review base sync/nvidia-main-6fa52e3e; the stale fork default main was not modified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant