chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 #1563

andygrove · 2025-03-21T19:55:51Z

Which issue does this PR close?

Rationale for this change

Use latest DataFusion. I will create a follow-on PR next week to change to the official 47.0.0 release once it is available.

What changes are included in this PR?

How are these changes tested?

andygrove · 2025-03-21T19:56:49Z

native/core/src/parquet/schema_adapter.rs

-    /// Unlike `map_batch` this method also preserves the columns that
-    /// may not appear in the final output (`projected_table_schema`) but may
-    /// appear in push down predicates
-    fn map_partial_batch(&self, batch: RecordBatch) -> datafusion::common::Result<RecordBatch> {


map_partial_batch has been removed in DataFusion

@mbutrovich @parthchandra fyi, I am not sure of the impact, but I figured I would first see if any tests fail in CI

codecov-commenter · 2025-03-21T20:53:56Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 58.75%. Comparing base (f09f8af) to head (da0cc73).
Report is 146 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1563      +/-   ##
============================================
+ Coverage     56.12%   58.75%   +2.62%     
- Complexity      976     1081     +105     
============================================
  Files           119      125       +6     
  Lines         11743    12581     +838     
  Branches       2251     2360     +109     
============================================
+ Hits           6591     7392     +801     
- Misses         4012     4015       +3     
- Partials       1140     1174      +34

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

andygrove · 2025-04-07T22:21:27Z

I ran TPC-H benchmarks locally and do not see any noticeable difference in performance

andygrove · 2025-04-10T17:44:04Z

Some aggregate tests started to fail. I filed an issue in DataFusion apache/datafusion#15676

andygrove · 2025-04-14T13:41:59Z

native/core/src/execution/expressions/bloom_filter_might_contain.rs

+    fn fmt_sql(&self, _: &mut Formatter<'_>) -> std::fmt::Result {
+        unimplemented!()
+    }


DataFusion now supports a new EXPLAIN output, but this is not exposed in Comet, so we do not need to implement these new methods.

andygrove · 2025-04-14T13:42:40Z

native/core/src/parquet/parquet_exec.rs

-        .with_table_partition_cols(partition_fields),
-        _ => get_file_config(
+        (Some(data_schema), Some(projection_vector), Some(partition_fields)) => {
+            get_file_config_builder(


this now uses a builder

andygrove · 2025-04-14T14:55:17Z

@comphead Could you review the HDFS changes? There was a change in objectstore to use Range<u64> instead of Range<usize>

comphead · 2025-04-14T23:45:22Z

@comphead Could you review the HDFS changes? There was a change in objectstore to use Range<u64> instead of Range<usize>

Thanks @andygrove I think these HDFS changes LGTM

parthchandra

lgtm. I don't think the schema adapter change is likely to affect our readers

comphead

lgtm thanks @andygrove are you planning to refer to rc1 or wait for the official release?

andygrove · 2025-04-16T22:12:08Z

I'd like to merge this one and then switch to the official release next week

YanivKunda · 2025-04-20T15:05:05Z

I'd like to merge this one and then switch to the official release next week

@andygrove here's the switch:
#1663

use latest DF from main branch

cbce217

andygrove commented Mar 21, 2025

View reviewed changes

andygrove changed the title ~~chore: Prepare for update to DataFusion 47~~ chore: Prepare for DataFusion 47.0.0 Mar 21, 2025

remove unused schema

e5e49d9

andygrove added 4 commits April 7, 2025 15:22

upmerge

ea66d82

bump df rev

5896128

tidy up

74ae825

update test

fe549cb

andygrove mentioned this pull request Apr 7, 2025

Release DataFusion 47.0.0 (April 2025) apache/datafusion#15072

Closed

39 tasks

andygrove added 3 commits April 8, 2025 07:30

bump df rev

0ac9529

bump df rev

727a3fb

Merge remote-tracking branch 'apache/main' into prep-datafusion-47

2544107

andygrove mentioned this pull request Apr 10, 2025

Regression in last_value functionality apache/datafusion#15676

Closed

andygrove added 8 commits April 11, 2025 11:26

ignore first last

84896e1

bump rev

748644a

Merge remote-tracking branch 'apache/main' into prep-datafusion-47

2658f7c

ignore more tests

e6f9e92

bump rev ignore tests

51b93ff

Merge remote-tracking branch 'apache/main' into prep-datafusion-47

49f6cf2

ignore test

7a8b316

bump rev to include arrow-rs 55

baa5faa

andygrove changed the title ~~chore: Prepare for DataFusion 47.0.0~~ chore: Prepare for datafusion 47.0.0 and arrow-rs 55.0.0 Apr 14, 2025

andygrove commented Apr 14, 2025

View reviewed changes

andygrove added 3 commits April 14, 2025 09:51

add link to follow-on issue

0cfb757

update deps

eb90625

enable a test

633d1de

update hdfs crate to use Range<u64> instead of Range<usize>

0b4efe5

andygrove mentioned this pull request Apr 14, 2025

Investigate unstable benchmark results on macOS #1648

Closed

andygrove added 2 commits April 15, 2025 09:58

Merge remote-tracking branch 'apache/main' into prep-datafusion-47

3ce6bfc

bump DF rev

3afd646

mbutrovich mentioned this pull request Apr 15, 2025

fix: better int96 support for experimental native scans #1652

Merged

andygrove changed the title ~~chore: Prepare for datafusion 47.0.0 and arrow-rs 55.0.0~~ chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 Apr 16, 2025

andygrove marked this pull request as ready for review April 16, 2025 21:03

andygrove added 2 commits April 16, 2025 17:06

use rc

6dc3384

Merge remote-tracking branch 'apache/main' into prep-datafusion-47

da0cc73

parthchandra approved these changes Apr 16, 2025

View reviewed changes

comphead approved these changes Apr 16, 2025

View reviewed changes

andygrove merged commit b8be7b7 into apache:main Apr 17, 2025
78 checks passed

andygrove deleted the prep-datafusion-47 branch April 17, 2025 02:12

YanivKunda mentioned this pull request Apr 20, 2025

chore: Upgrade to datafusion 47.0.0 #1663

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 #1563

chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 #1563

Uh oh!

andygrove commented Mar 21, 2025 •

edited

Loading

Uh oh!

andygrove Mar 21, 2025

Uh oh!

andygrove Mar 21, 2025

Uh oh!

codecov-commenter commented Mar 21, 2025 •

edited

Loading

Uh oh!

andygrove commented Apr 7, 2025

Uh oh!

andygrove commented Apr 10, 2025

Uh oh!

andygrove Apr 14, 2025 •

edited

Loading

Uh oh!

andygrove Apr 14, 2025

Uh oh!

andygrove commented Apr 14, 2025

Uh oh!

comphead commented Apr 14, 2025 •

edited

Loading

Uh oh!

parthchandra left a comment

Uh oh!

comphead left a comment

Uh oh!

andygrove commented Apr 16, 2025

Uh oh!

Uh oh!

YanivKunda commented Apr 20, 2025

Uh oh!

Uh oh!

chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 #1563

chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 #1563

Uh oh!

Conversation

andygrove commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

andygrove Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

andygrove Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

codecov-commenter commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

andygrove commented Apr 7, 2025

Uh oh!

andygrove commented Apr 10, 2025

Uh oh!

andygrove Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andygrove Apr 14, 2025

Choose a reason for hiding this comment

Uh oh!

andygrove commented Apr 14, 2025

Uh oh!

comphead commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parthchandra left a comment

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove commented Apr 16, 2025

Uh oh!

Uh oh!

YanivKunda commented Apr 20, 2025

Uh oh!

Uh oh!

andygrove commented Mar 21, 2025 •

edited

Loading

codecov-commenter commented Mar 21, 2025 •

edited

Loading

andygrove Apr 14, 2025 •

edited

Loading

comphead commented Apr 14, 2025 •

edited

Loading