Weekly Plan (Andrew Lamb) Feb 24, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• work with @xudong963 to get Release DataFusion 46.0.0 #14123 ready (mostly testing / herding bugs)
• chore: upgrade to DataFusion 46.0.0 delta-io/delta-rs#3261
• publish 45 blog post from @Omega359 : DataFusion 45 blog post datafusion-site#57
• Make a sql-parser-rs release Release sqlparser-rs version 0.55.0 datafusion-sqlparser-rs#1671
• Review a bunch of PRs (naturally)

Other projects I plan to help along with priority

(aka look for PRs related to these first)

• Splitting out datasources with @AdamGS and @logan-keede [Epic] Split datasources out from datafusion crate (datafusion/core) #14444
• Statistics / Analysis with @clflushopt @ozankabak and others StatisticsV2: initial statistics framework redesign #14699 / Examples: boundary analysis example for AND/OR conjunctions #14735

Weekly Stretch / Nice to have goals

• Make a new "this week in datafusion"
• Improve new documentation for DataSourceExec / Config files
• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

Related:

• alamb's review queue #14698

[alamb]

Review Queue:

DataFusion: Bugs/UX/Performance

• fix duplicated schema name error from count wildcard #14824
• fix(substrait): Do not add implicit groupBy expressions in LogicalPlanBuilder or when building logical plans from Substrait #14860

DataFusion: New Features/ Functions

• StatisticsV2: initial statistics framework redesign #14699

[alamb]

Review Queue:

DataFusion: Bugs/UX/Performance

• Move FileFormat and related pieces to datafusion-datasource #14873
• Refactor SortPushdown using the standard top-down visitor and using EquivalenceProperties #14821
• Datafusion-cli: Redesign the datafusion-cli execution and print, make it totally streaming printing without memory overhead #14877
• feat: Add tree / pretty explain mode #14677
• Do not unescape backslashes in datafusion-cli #14844
• Cancellation benchmark #14818
• fix: fetch is missed during EnforceDistribution #14207
• Examples: boundary analysis example for AND/OR conjunctions #14735

[alamb]

DataFusion: Bugs/UX/Performance

• Allow setting the recursion limit for sql parsing #14756
• Fix sequential metadata fetching in ListingTable causing high latency #14918

[alamb]

DataFusion: Bugs/UX/Performance

• feat: Add tree / pretty explain mode #14677
• feat: introduce JoinSetTracer trait for tracing context propagation in spawned tasks #14547

DataFusion: New Features/ Functions

• Add support for DISTINCT + ORDER BY in ARRAY_AGG #14413
• support simple/cross lateral joins #14595
• Support multiple results comparison in benchmark scripts #14196
• Add union_tag scalar function #14687
• STRING_AGG missing functionality #14412
• Introduce Async User Defined Functions  #14837

arrow-rs

• Avro codec enhancements arrow-rs#6965
• Add Parquet Modular decryption (read) support + encryption flag arrow-rs#6637
• Test : move tests for parse_string_decimal_native to parse_decimal  arrow-rs#7177
• feat: add rounding logic and scale zero fix parse_decimal to match parse_string_to_decimal_native behavior arrow-rs#7179
• feat:use parse_decimal for generic_string_to_decimal conversion. arrow-rs#7191

Stuck PRs that need help pushing to completion

• Substrait support for propagating TableScan.filters to Substrait ReadRel.filter #14194
• hash join: add build-side join keys to memory accounting #14222
• add manual trigger for extended tests in pull requests #14331
• Support marking columns as system columns via Field's metadata #14362 / feat: metadata columns #14057

[alamb]

• Next week: Weekly Plan (Andrew Lamb) March 3, 2025 #14978