v0.7.4 · Eventual-Inc Daft · Discussion #6293 · GitHub

desmondcheongzx
Feb 25, 2026
Maintainer

What's Changed 🚀

💥 Breaking Changes

refactor(arrow2)!: remaining arrow2 from daft-core @universalmind303 (refactor(arrow2)!: remaining arrow2 from daft-core #6284)
refactor(arrow2)!: use arrow-rs casting for strftime function @universalmind303 (refactor(arrow2)!: use arrow-rs casting for strftime function #6263)
refactor(arrow2)!: migrate interval arithmetic to arrow-rs @rohitkulshreshtha (refactor(arrow2)!: migrate interval arithmetic to arrow-rs #6186)

✨ Features

feat(ai): Add HTTP URL direct passthrough for images and videos in prompt function @huleilei (feat(ai): Add HTTP URL direct passthrough for images and videos in prompt function #6182)
feat(sql): add DATE_TRUNC function support @desmondcheongzx (feat(sql): add DATE_TRUNC function support #6258)
feat(swordfish): Streaming sources @colin-ho (feat(swordfish): Streaming sources #5978)
feat(metrics): add metrics docs @cckellogg (feat(metrics): add metrics docs #6253)
feat(io/av): enhance time-interval sampling with comprehensive tests and improved @huleilei (feat(io/av): enhance time-interval sampling with comprehensive tests and improved #6088)
feat: Add Tencent Cloud COS (Cloud Object Storage) support @XuQianJin-Stars (feat: Add Tencent Cloud COS (Cloud Object Storage) support #6140)
feat(metrics): consolidate naming and add node.type attribute @cckellogg (feat(metrics): consolidate naming and add node.type attribute #6236)
feat: add Flight shuffle to Flotilla @srilman (feat: add Flight shuffle to Flotilla #6123)
feat: Apache OpenDAL™ compatible backends @universalmind303 (feat: Apache OpenDAL™ compatible backends #6177)
feat(observability): Split duration into separate column in metrics DF @srilman (feat(observability): Split duration into separate column in metrics DF #6235)
feat: add support for pyiceberg 0.11.0 @gweaverbiodev (feat: add support for pyiceberg 0.11.0 #6200)
feat: add support for SQL ORDER BY column position @Lucas61000 (feat: add support for SQL ORDER BY column position #6211)
feat: Supports running dashboard in daemon mode @plotor (feat: Supports running dashboard in daemon mode #5993)
feat: json_write support for time stamps @gpathak128 (feat: json_write support for time stamps #6214)
feat: .as_T cast methods @aaron-ang (feat: .as_T cast methods #6100)

🐛 Bug Fixes

fix: handle case where join keys are different for sort-merge multi-partition join @gweaverbiodev (fix: handle case where join keys are different for sort-merge multi-partition join #6243)
fix(arrow2): reinterpret physical array as logical type after cast fallback @desmondcheongzx (fix(arrow2): reinterpret physical array as logical type after cast fallback #6291)
fix(sql): resolve GROUP BY ambiguous column names for derived expressions @desmondcheongzx (fix(sql): resolve GROUP BY ambiguous column names for derived expressions #6286)
fix(flight): Add a check for flight_shuffle_dirs arg and change default @srilman (fix(flight): Add a check for flight_shuffle_dirs arg and change default #6266)
fix: Map Literal <-> Python Dict conversion @w2ais (fix: Map Literal <-> Python Dict conversion #6084)
fix: NaN-aware comparator for multi-column search_sorted and sort @desmondcheongzx (fix: NaN-aware comparator for multi-column search_sorted and sort #6242)
fix: add ignore_empty_and_null parameter to .explode() @singularityDLW (fix: add ignore_empty_and_null parameter to .explode() #6047)
fix: Broadcast literal expressions in aggregations to match input length @desmondcheongzx (fix: Broadcast literal expressions in aggregations to match input length #6155)
fix: Cleanup imports from dashboard daemon PR @srilman (fix: Cleanup imports from dashboard daemon PR #6222)
fix: Fix compilation on main @srilman (fix: Fix compilation on main #6221)
fix: canonicalize negative NaN in multi-column sort comparator @ykdojo (fix: canonicalize negative NaN in multi-column sort comparator #6215)
fix: delta version parsing @aaron-ang (fix: delta version parsing #6156)
fix: Register extension type before reading from Lance @ykdojo (fix: Register extension type before reading from Lance #6058)
fix: use union instead of append_column in window agg to fix schema mismatch @ykdojo (fix: use union instead of append_column in window agg to fix schema mismatch #6178)
fix: into_batches should not allow downstream shuffle elision @desmondcheongzx (fix: into_batches should not allow downstream shuffle elision #6170)

♻️ Refactor

refactor(arrow2): refactor index_bitmap and time_unit @universalmind303 (refactor(arrow2): refactor index_bitmap and time_unit #6287)
refactor(arrow2)!: remaining arrow2 from daft-core @universalmind303 (refactor(arrow2)!: remaining arrow2 from daft-core #6284)
refactor(arrow2): remove series::try_from(name, arrow2_arr) @universalmind303 ( refactor(arrow2): remove series::try_from(name, arrow2_arr) #6283)
refactor(arrow2): switch to arrow-rs backed arrays @universalmind303 (refactor(arrow2): switch to arrow-rs backed arrays #6280)
refactor(arrow2): misc deprecation warnings @universalmind303 (refactor(arrow2): misc deprecation warnings #6270)
refactor(arrow2): remove from_arrow2 @universalmind303 (refactor(arrow2): remove from_arrow2 #6267)
refactor(arrow2): fully remove toarrow2 from arrays & series @universalmind303 (refactor(arrow2): fully remove toarrow2 from arrays & series #6265)
refactor(arrow2)!: use arrow-rs casting for strftime function @universalmind303 (refactor(arrow2)!: use arrow-rs casting for strftime function #6263)
refactor(arrow2): use arrow-rs for filtering on python arrays @universalmind303 (refactor(arrow2): use arrow-rs for filtering on python arrays #6262)
refactor(arrow2): migrate cast.rs from arrow2 to arrow-rs @desmondcheongzx (refactor(arrow2): migrate cast.rs from arrow2 to arrow-rs #6239)
refactor(arrow2): migrate BooleanArray bitmap access to arrow-rs @desmondcheongzx (refactor(arrow2): migrate BooleanArray bitmap access to arrow-rs #6256)
refactor(arrow2): migrate concat.rs to arrow-rs @desmondcheongzx (refactor(arrow2): migrate concat.rs to arrow-rs #6255)
refactor(arrow2): remove misc arrow2 references in csv and json @universalmind303 (refactor(arrow2): remove misc arrow2 references in csv and json #6248)
refactor(arrow2): remove more arrow2 usages in daft-core @universalmind303 (refactor(arrow2): remove more arrow2 usages in daft-core #6249)
refactor(arrow2): remove arrow2 from daft-recordbatch @universalmind303 (refactor(arrow2): remove arrow2 from daft-recordbatch #6231)
refactor(arrow2): replace arrow2 based buffer with custom impl @universalmind303 (refactor(arrow2): replace arrow2 based buffer with custom impl #6247)
refactor(arrow2)!: migrate interval arithmetic to arrow-rs @rohitkulshreshtha (refactor(arrow2)!: migrate interval arithmetic to arrow-rs #6186)
refactor(arrow2): remove daft_arrow from arrow_growable @universalmind303 (refactor(arrow2): remove daft_arrow from arrow_growable #6251)
refactor(arrow2): migrate len.rs to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate len.rs to arrow-rs #6171)
refactor(arrow2): migrate image.rs to arrow-rs @rohitkulshreshtha (refactor(arrow2): migrate image.rs to arrow-rs #6191)
refactor(arrow2): migrate dyn_compare and probeable to arrow-rs @desmondcheongzx (refactor(arrow2): migrate dyn_compare and probeable to arrow-rs #6227)
refactor(arrow2): migrate if_else kernel from arrow2 to arrow-rs @desmondcheongzx (refactor(arrow2): migrate if_else kernel from arrow2 to arrow-rs #6240)
refactor(arrow2): migrate array serdes from arrow2 to arrow-rs @desmondcheongzx (refactor(arrow2): migrate array serdes from arrow2 to arrow-rs #6238)
refactor(arrow2): migrate growable internals from arrow2 to arrow-rs @desmondcheongzx (refactor(arrow2): migrate growable internals from arrow2 to arrow-rs #6228)
refactor(arrow-rs): Remove basic growable usages @srilman (refactor(arrow-rs): Remove basic growable usages #5779)
refactor(arrow2): misc arrow2 cleanups in daft-core @universalmind303 (refactor(arrow2): misc arrow2 cleanups in daft-core #6232)
refactor(arrow2): remove daft_arrow from daft-functions-utf8 @universalmind303 (refactor(arrow2): remove daft_arrow from daft-functions-utf8 #6229)
refactor(arrow2): fully remove daft-arrow from functions-list @universalmind303 (refactor(arrow2): fully remove daft-arrow from functions-list #6230)
refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs @universalmind303 (refactor(arrow2): migrates hashing kernel from arrow2 to arrow-rs #6166)
refactor(arrow2): replace arrow2 iterators with custom daft-native iterators @universalmind303 (refactor(arrow2): replace arrow2 iterators with custom daft-native iterators #6220)
refactor(arrow2): list kernels @universalmind303 (refactor(arrow2): list kernels #6219)
refactor(arrow2): migrate groups to arrow-rs @cckellogg (refactor(arrow2): migrate groups to arrow-rs #6185)
refactor(arrow2): second attempt at offsets @universalmind303 (refactor(arrow2): second attempt at offsets #6162)
refactor(arrow2): update image_array to not use arrow2 @universalmind303 (refactor(arrow2): update image_array to not use arrow2 #6217)
refactor(arrow2): Migrate arithmetic kernels @desmondcheongzx (refactor(arrow2): Migrate arithmetic kernels #6193)
refactor(arrow2): Migrate concat agg @desmondcheongzx (refactor(arrow2): Migrate concat agg #6190)

📖 Documentation

docs: Fix broken links to the url modality and various datatypes @desmondcheongzx (docs: Fix broken links to the url modality and various datatypes #6212)

✅ Tests

test(benchmarking): add some read_json python benchmarks @universalmind303 (test(benchmarking): add some read_json python benchmarks #6288)
test(parquet): add benchmarks for nested types, codecs, and filter pushdown @desmondcheongzx (test(parquet): add benchmarks for nested types, codecs, and filter pushdown #6285)
test: Filter null bytes from generated column names in property-based tests @desmondcheongzx (test: Filter null bytes from generated column names in property-based tests #6213)
test(postmerge): Minor fix to OpenAI integration tests @desmondcheongzx (test(postmerge): Minor fix to OpenAI integration tests #6209)
test: Fix incomplete metrics migration in OpenAI integration tests @desmondcheongzx (test: Fix incomplete metrics migration in OpenAI integration tests #6204)

👷 CI

ci: include dashboard assets in maturin sdist for manylinux builds @desmondcheongzx (ci: include dashboard assets in maturin sdist for manylinux builds #6281)
ci: fix restore-mtime exit code when last file is deleted on macos runners @desmondcheongzx (ci: fix restore-mtime exit code when last file is deleted on macos runners #6268)
ci: cache workspace crates and share rust caches across all workflows @desmondcheongzx (ci: cache workspace crates and share rust caches across all workflows #6264)
ci: cache workspace crate artifacts in integration build @desmondcheongzx (ci: cache workspace crate artifacts in integration build #6261)
ci: share rust cache across branches and restore file mtimes @desmondcheongzx (ci: share rust cache across branches and restore file mtimes #6246)
ci: bump all timeouts from 30 -> 45 @universalmind303 (ci: bump all timeouts from 30 -> 45 #6250)
ci: increase integration-test-build timeout from 45 to 90 minutes @desmondcheongzx (ci: increase integration-test-build timeout from 45 to 90 minutes #6244)
ci: increase integration-test-build timeout from 30 to 45 minutes @desmondcheongzx (ci: increase integration-test-build timeout from 30 to 45 minutes #6241)
ci: increase integration-test-sql timeout from 30 to 45 minutes @desmondcheongzx (ci: increase integration-test-sql timeout from 30 to 45 minutes #6237)
ci: Reduce xdist workers on macOS to fix Ray actor timeout @desmondcheongzx (ci: Reduce xdist workers on macOS to fix Ray actor timeout #6210)
ci: make repository secrets optional for CI workflows @jeevb (ci: make repository secrets optional for CI workflows #6199)
ci: Improve CI reliability via disk space reclamation and coverage optimization @desmondcheongzx (ci: Improve CI reliability via disk space reclamation and coverage optimization #6205)

🔧 Maintenance

chore: Update codeowners @colin-ho (chore: Update codeowners #6183)
chore: Adding Issue requirement for PRs, Updating maintainers @madvart (chore: Adding Issue requirement for PRs, Updating maintainers #6196)
chore(observability): Split dashboard cli into separate start / stop subcommands @srilman (chore(observability): Split dashboard cli into separate start / stop subcommands #6234)
chore(deps): Resolve Dependabot security alerts @desmondcheongzx (chore(deps): Resolve Dependabot security alerts #6226)
chore: address review feedback from chore(deps): Bump dependency group with conflict resolution #6208 @desmondcheongzx (chore: address review feedback from #6208 #6225)
chore(deps): Bump dependency group with conflict resolution @desmondcheongzx (chore(deps): Bump dependency group with conflict resolution #6208)
chore: Replace Bun with Node / NPM @srilman (chore: Replace Bun with Node / NPM #6202)
chore(arrow2): simplify deprecation markers @universalmind303 (chore(arrow2): simplify deprecation markers #6218)
chore: add .values method for utf8Array @universalmind303 (chore: add .values method for utf8Array #6216)
chore: add mypy-boto3-glue to aws optional dependencies @Killua7163 (chore: add mypy-boto3-glue to aws optional dependencies #6080)

Full Changelog: v0.7.3...v0.7.4

This discussion was created from the release v0.7.4.

Replies: 0 comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment