Skip to content

feat: unify data stats and partition values in DataSkippingFilter#1948

Open
DrakeLin wants to merge 2 commits intodelta-io:mainfrom
DrakeLin:stack/unify-data-skipping-stats-partition
Open

feat: unify data stats and partition values in DataSkippingFilter#1948
DrakeLin wants to merge 2 commits intodelta-io:mainfrom
DrakeLin:stack/unify-data-skipping-stats-partition

Conversation

@DrakeLin
Copy link
Collaborator

@DrakeLin DrakeLin commented Feb 25, 2026

🥞 Stacked PR

Use this link to review incremental changes.


What changes are proposed in this pull request?

Restructure DataSkippingFilter to combine data stats and partition values into a unified schema ({ stats_parsed, partitionValues_parsed }) for predicate evaluation.

Enables usage of partitionValues_parsed

How was this change tested?

@codecov
Copy link

codecov bot commented Feb 25, 2026

Codecov Report

❌ Patch coverage is 76.53333% with 88 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.45%. Comparing base (630db20) to head (4c1e6b5).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
uc-catalog/src/lib.rs 0.00% 67 Missing ⚠️
kernel/src/log_segment.rs 88.00% 7 Missing and 2 partials ⚠️
kernel/src/scan/data_skipping.rs 91.26% 7 Missing and 2 partials ⚠️
kernel/src/scan/log_replay.rs 97.46% 2 Missing ⚠️
kernel/src/engine/arrow_utils.rs 85.71% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1948      +/-   ##
==========================================
- Coverage   86.53%   86.45%   -0.08%     
==========================================
  Files         147      147              
  Lines       45547    45802     +255     
  Branches    45547    45802     +255     
==========================================
+ Hits        39414    39599     +185     
- Misses       4471     4540      +69     
- Partials     1662     1663       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions github-actions bot added the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Feb 25, 2026
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch 11 times, most recently from f7b29dd to d8c2ddf Compare February 28, 2026 06:22
@github-actions github-actions bot removed the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Feb 28, 2026
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch 3 times, most recently from c29e550 to 0cac46f Compare March 3, 2026 00:13
@github-actions github-actions bot added the breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump. label Mar 3, 2026
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch 5 times, most recently from 71ee728 to 46ce3ac Compare March 4, 2026 00:38
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch 2 times, most recently from 37ed3f0 to 98afd42 Compare March 4, 2026 01:42
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch from 98afd42 to 1733c63 Compare March 4, 2026 02:26
@DrakeLin DrakeLin force-pushed the stack/unify-data-skipping-stats-partition branch from 1733c63 to 4c1e6b5 Compare March 4, 2026 02:48
Ok(())
}

// Validates partition-based data skipping works end-to-end against a real UC table.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For testing only

@DrakeLin DrakeLin marked this pull request as ready for review March 4, 2026 02:49
@DrakeLin DrakeLin changed the title unify feat: unify data stats and partition values in DataSkippingFilter Mar 4, 2026
@DrakeLin DrakeLin requested review from dengsh12 and nicklan March 4, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-change Public API change that could cause downstream compilation failures. Requires a major version bump.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant