Skip to content

feat: split Parquet files into row-group-sized morsels #59

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
adriangb wants to merge 12 commits into
base: main
Choose a base branch
from
Draft

feat: split Parquet files into row-group-sized morsels #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
c470bb1
chore: backport version from `branch-53`, update some dependencies (#…
comphead Apr 21, 2026
e524f49
fix: array_concat widens container variant for mixed List/LargeList i…
hcrosse Apr 21, 2026
9b5e43e
feat: Expose used `MemoryPool` details in `ResourcesExhausted` error …
erenavsarogullari Apr 21, 2026
526f0cb
perf: Reduce `Box` and `Arc` allocation churn during tree rewriting (…
neilconway Apr 21, 2026
a737c27
feat: estimate cardinality for semi and anti-joins using distinct cou…
buraksenn Apr 21, 2026
5baa6ef
chore(deps): bump astral-sh/setup-uv from 8.0.0 to 8.1.0 (#21759)
dependabot[bot] Apr 21, 2026
9a851d6
fix: Fix local `datafusion-cli` test failure (#21761)
2010YOUY01 Apr 21, 2026
c641262
chore(deps): bump aws-config from 1.8.15 to 1.8.16 in the all-other-c…
dependabot[bot] Apr 21, 2026
af67cdd
chore(deps): bump github/codeql-action from 4.35.1 to 4.35.2 (#21758)
dependabot[bot] Apr 21, 2026
9a1ed57
chore(deps): bump taiki-e/install-action from 2.75.10 to 2.75.18 (#21…
dependabot[bot] Apr 21, 2026
ff805cf
feat: split Parquet files into row-group-sized morsels
adriangb Apr 20, 2026
8ef53b7
refactor: defer morsel decoder build to Morsel::into_stream
adriangb Apr 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/audit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ jobs:
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Install cargo-audit
uses: taiki-e/install-action@85b24a67ef0c632dfefad70b9d5ce8fddb040754 # v2.75.10
uses: taiki-e/install-action@055f5df8c3f65ea01cd41e9dc855becd88953486 # v2.75.18
with:
tool: cargo-audit
- name: Run audit check
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/codeql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,11 @@ jobs:
persist-credentials: false

- name: Initialize CodeQL
uses: github/codeql-action/init@c10b8064de6f491fea524254123dbe5e09572f13 # v4
uses: github/codeql-action/init@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4
with:
languages: actions

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@c10b8064de6f491fea524254123dbe5e09572f13 # v4
uses: github/codeql-action/analyze@95e58e9a2cdfd71adc6e0353d5c52f41a045d225 # v4
with:
category: "/language:actions"
2 changes: 1 addition & 1 deletion .github/workflows/docs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
path: asf-site

- name: Setup uv
uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0

- name: Install dependencies
run: uv sync --package datafusion-docs
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs_pr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
submodules: true
fetch-depth: 1
- name: Setup uv
uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
- name: Install doc dependencies
run: uv sync --package datafusion-docs
- name: Install dependency graph tooling
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ jobs:
sudo apt-get update -qq
sudo apt-get install -y -qq clang
- name: Setup wasm-pack
uses: taiki-e/install-action@85b24a67ef0c632dfefad70b9d5ce8fddb040754 # v2.75.10
uses: taiki-e/install-action@055f5df8c3f65ea01cd41e9dc855becd88953486 # v2.75.18
with:
tool: wasm-pack
- name: Run tests with headless mode
Expand Down Expand Up @@ -770,7 +770,7 @@ jobs:
- name: Setup Rust toolchain
uses: ./.github/actions/setup-builder
- name: Install cargo-msrv
uses: taiki-e/install-action@85b24a67ef0c632dfefad70b9d5ce8fddb040754 # v2.75.10
uses: taiki-e/install-action@055f5df8c3f65ea01cd41e9dc855becd88953486 # v2.75.18
with:
tool: cargo-msrv

Expand Down
Loading
Loading