Generation of Snapshot Summaries #724
Assignees
Comments
|
Can I pick this up? |
|
@barronw gladly! Assigned the issue to you :) |
|
@c-thiel Feel free to re-assign this to someone else (maybe @jonathanc-n). I won't be able to complete this soon, so I pushed what I have to my fork for reference (link). |
liurenjie1024
pushed a commit
that referenced
this issue
Mar 21, 2025
## Which issue does this PR close? - Related to #724 ## What changes are included in this PR? This is the building block to implementing snapshot summaries. Most of the implementation code is built off of @barronw 's very nice commit. Most of the added lines are also just tests. A follow up pr will be included integrating it with the rest of the codebase. ## Are these changes tested? Yes, unit test
liurenjie1024
added a commit
that referenced
this issue
Mar 25, 2025
…mmary` (#1122) ## Which issue does this PR close? - Part of #724 . ## What changes are included in this PR? Added functionality for merging summary and adding manifest file information. ## Are these changes tested? One unit test for merge functionality and add manifest --------- Co-authored-by: Renjie Liu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With each snapshot comes a summary map, optional in V1, required in V2 and later:
The summary contains information such as what kind of files the snapshot contains (data/delete), and what the changes are in rows and bytes. The best way to replicate this metrics collection is by looking at the Java code: https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/SnapshotSummary.java
This class works closely alongside the
SnapshotProducerand tracks what happens with the snapshot.The text was updated successfully, but these errors were encountered: