New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: support merge append action #902

Open

ZENOTME wants to merge 3 commits into apache:main from ZENOTME:merge_append_new

+698 −31

Contributor

ZENOTME commented Jan 20, 2025

This PR complete #736

Contributor Author

ZENOTME commented Jan 20, 2025

cc @Fokko @liurenjie1024 @Xuanwo @sdd

kevinjqliu reviewed

View reviewed changes

crates/iceberg/src/spec/manifest.rs Outdated

@@ @@ -1182,6 +1182,12 @@ impl ManifestEntry { @@
                   pub fn data_file(&self) -> &DataFile {
                       &self.data_file
                   }
+                  /// get file sequence number

Contributor

kevinjqliu Jan 20, 2025

Suggested change

      
                /// get file sequence number
          
                /// File sequence number indicating when the file was added. Inherited when null and status is 1 (added).

from https://iceberg.apache.org/spec/#manifests

jonathanc-n reviewed

View reviewed changes

Contributor

jonathanc-n left a comment

Overall just some small nits. implementing the defaults values for new fields can be done after this pr (#737)

crates/integration_tests/tests/merge_append_test.rs Outdated

+                  // Enable merge append for table
+                  let tx = Transaction::new(&table);
+                  table = tx
+                      .set_properties(HashMap::from([

Contributor

jonathanc-n Feb 14, 2025

Should we also try adding a test here for the MANIFEST_TARGET_SIZE_BYTES property?

crates/iceberg/src/transaction.rs Outdated

+                  /// Finished building the action and apply it to the transaction.
+                  pub async fn apply(self) -> Result<Transaction<'a>> {
+                      if self.merge_enabled {
+                          let process = MergeManifsetProcess {

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                        let process = MergeManifsetProcess {
          
                        let process = MergeManifestProcess {

crates/iceberg/src/transaction.rs Outdated

+                  }
+              }
+              impl ManifestProcess for MergeManifsetProcess {

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
            impl ManifestProcess for MergeManifsetProcess {
          
            impl ManifestProcess for MergeManifestProcess {

crates/iceberg/src/transaction.rs Outdated

Comment on lines 388 to 393

+              struct MergeManifsetProcess {
+                  target_size_bytes: u32,
+                  min_count_to_merge: u32,
+              }
+              impl MergeManifsetProcess {

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
            struct MergeManifsetProcess {
          
                target_size_bytes: u32,
          
                min_count_to_merge: u32,
          
            }
          
            impl MergeManifsetProcess {
          
            struct MergeManifestProcess {
          
                target_size_bytes: u32,
          
                min_count_to_merge: u32,
          
            }
          
            impl MergeManifestProcess {

crates/iceberg/src/transaction.rs Outdated

Comment on lines 419 to 420

		for manifset_file in manifest_bin {
		let manifest_file = manifset_file.load_manifest(&file_io).await?;

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                    for manifset_file in manifest_bin {
          
                        let manifest_file = manifset_file.load_manifest(&file_io).await?;
          
                    for manifest_file in manifest_bin {
          
                        let manifest_file = manifest_file.load_manifest(&file_io).await?;

crates/iceberg/src/transaction.rs Outdated

+                      Ok(merged_bins.into_iter().flatten().collect())
+                  }
+                  async fn merge_manifeset<'a>(

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                async fn merge_manifeset<'a>(
          
                async fn merge_manifest<'a>(

crates/iceberg/src/transaction.rs Outdated

@@ @@ -267,13 +376,174 @@ trait SnapshotProduceOperation: Send + Sync { @@
               struct DefaultManifestProcess;
               impl ManifestProcess for DefaultManifestProcess {
-                  fn process_manifeset(&self, manifests: Vec<ManifestFile>) -> Vec<ManifestFile> {
-                      manifests
+                  async fn process_manifeset<'a>(

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                async fn process_manifeset<'a>(
          
                async fn process_manifest<'a>(

crates/iceberg/src/transaction.rs Outdated

                   }
               }
               trait ManifestProcess: Send + Sync {
-                  fn process_manifeset(&self, manifests: Vec<ManifestFile>) -> Vec<ManifestFile>;
+                  fn process_manifeset<'a>(

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                fn process_manifeset<'a>(
          
                fn process_manifest<'a>(

crates/iceberg/src/transaction.rs Outdated

+              }
+              impl ManifestProcess for MergeManifsetProcess {
+                  async fn process_manifeset<'a>(

Contributor

jonathanc-n Feb 14, 2025

Suggested change

      
                async fn process_manifeset<'a>(
          
                async fn process_manifest<'a>(

crates/iceberg/src/transaction.rs

+                          return Ok(manifests);
+                      }
+                      let first_manifest = manifests[0].clone();

Contributor

jonathanc-n Feb 14, 2025

Would this be an expensive clone (not wrapped in arc)? is there a way to avoid?

Contributor Author

ZENOTME Mar 13, 2025

async fn merge_group<'a>(
        &self,
        snapshot_produce: &mut SnapshotProduceAction<'a>,
        first_manifest: &ManifestFile,
        group_manifests: Vec<ManifestFile>,
    )

This function will take ownership of the manifests, so we have to clone the first one out. Looks like hard to avoid it. Welcome for any suggestion to avoid this.

ZENOTME added 2 commits

March 13, 2025 14:49


          support merge append action

5bd4b64


          refine code and test

82d73d4

ZENOTME force-pushed the merge_append_new branch from 2e6c75e to 82d73d4 Compare

March 13, 2025 08:02

Contributor Author

ZENOTME commented Mar 13, 2025

Thanks for review! @jonathanc-n @kevinjqliu. I have refined the code and fixed the test. I think this PR is ready to review again. also cc @Fokko @liurenjie1024 @Xuanwo @sdd

ZENOTME commented

View reviewed changes

crates/integration_tests/tests/shared_tests/merge_append_test.rs

+                      // For this first manifest, it will be pack with the first additional manifest and
+                      // the count(2) is less than the min merge count(4), so these two will not merge.
+                      // See detail: `MergeManifestProcess::merge_group`
+                      if idx == 0 {

Contributor Author

ZENOTME Mar 13, 2025 •

edited

Loading

I'm not sure whether it's expected behaviour. This manifest will not be merged, so the status will remain Added. cc @Fokko @kevinjqliu

ZENOTME mentioned this pull request

feat: support merge append risingwavelabs/iceberg-rust#29

Merged


          add manifest merge manager and only process manifest for data type

69e6a40

ZENOTME requested review from jonathanc-n and kevinjqliu

March 13, 2025 09:32

This was referenced Mar 14, 2025

Tracking: sync changes from upstream and drop commits in our fork risingwavelabs/iceberg-rust#31

Open

Support RowDeltaAction #1104

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet