Skip to content

feat(arro3): Freethreading - module markers + CI #275

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kylebarron merged 29 commits into from
Oct 3, 2025
Merged

feat(arro3): Freethreading - module markers + CI #275

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
8e030ba
Opt-in to gil_used = false, pin to rust-numpy git dep (for now)
H-Plus-Time Dec 12, 2024
13f5bc2
Add 3.13t to all builds
H-Plus-Time Dec 12, 2024
3e4d851
Minimum CI changes to fix 3.13t builds in CI
H-Plus-Time Dec 16, 2024
9f7ef44
Pin to main branch of rust-numpy, boost pyo3 to match (NB: does not c…
H-Plus-Time Feb 23, 2025
fd1726b
Merge remote-tracking branch 'upstream/main' into freethreading
H-Plus-Time Feb 24, 2025
a1a904d
Merge branch 'main' into freethreading
kylebarron Mar 25, 2025
e4a6f49
use upstream actions/setup-python
kylebarron Mar 25, 2025
0c01ce7
Add pytest-freethreaded, boost (hopefully just dev/test-related) pyth…
H-Plus-Time Mar 26, 2025
728a8d2
Unmark io as freethreading-safe, exclude io tests from concurrent tests
H-Plus-Time Mar 26, 2025
37b9f69
Wrap filesystem access in context-managed temp directories in tests.
H-Plus-Time Mar 28, 2025
e78f219
Re-mark io as freethreading safe
H-Plus-Time Mar 28, 2025
12abd0f
ruff format
H-Plus-Time Mar 28, 2025
bd0fe7a
cargo fmt
H-Plus-Time Mar 28, 2025
9708633
Merge branch 'main' into freethreading
kylebarron Sep 9, 2025
d55c441
Add comments to tests
kylebarron Sep 9, 2025
b146b39
Split docs deps into separate group
kylebarron Sep 9, 2025
20e59e8
Add pre-commit as dev dep
kylebarron Sep 9, 2025
711724e
update cargo lockfile
kylebarron Sep 9, 2025
ea1bb3f
regen lockfile
kylebarron Sep 9, 2025
5457247
Add test with --require-gil-disabled
kylebarron Sep 9, 2025
1876172
install python with uv in ci
kylebarron Sep 9, 2025
b489b0a
typo
kylebarron Sep 9, 2025
97e4b97
reduce requires-python
kylebarron Sep 9, 2025
1db2474
pytest-freethreaded only on 3.13+
kylebarron Sep 9, 2025
1125fff
Merge branch 'main' into freethreading
kylebarron Oct 3, 2025
154b9f6
bump maturin
kylebarron Oct 3, 2025
4739800
bump maturin in lock
kylebarron Oct 3, 2025
9c4ddbc
version reordering
kylebarron Oct 3, 2025
b692107
lock
kylebarron Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 11 additions & 9 deletions .github/workflows/wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,15 +56,15 @@ jobs:
version: "0.4.x"

- name: Install Python versions
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10 3.13t

- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
# As of Nov 2024, it was necessary to manually specify -i 3.13 to get
# maturin to find the executable. --find-interpreter did not find it.
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 --manifest-path ${{ matrix.module }}/Cargo.toml
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 -i 3.13t --manifest-path ${{ matrix.module }}/Cargo.toml
sccache: "true"
manylinux: ${{ matrix.platform.manylinux }}

Expand Down Expand Up @@ -101,13 +101,13 @@ jobs:
version: "0.4.x"

- name: Install Python versions
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10 3.13t

- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 --manifest-path ${{ matrix.module }}/Cargo.toml
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 -i 3.13t --manifest-path ${{ matrix.module }}/Cargo.toml
sccache: "true"
manylinux: musllinux_1_2
- name: Upload wheels
Expand All @@ -130,15 +130,17 @@ jobs:
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
- uses: Quansight-Labs/setup-python@v5
with:
python-version: 3.13
python-version: |
3.13t
3.13

- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 --manifest-path ${{ matrix.module }}/Cargo.toml
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i 3.13t --manifest-path ${{ matrix.module }}/Cargo.toml
sccache: "true"
- name: Upload wheels
uses: actions/upload-artifact@v4
Expand Down Expand Up @@ -169,13 +171,13 @@ jobs:
version: "0.4.x"

- name: Install Python versions
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10
run: uv python install 3.9 3.10 3.11 3.12 3.13 pypy3.10 3.13t

- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: ${{ matrix.platform.target }}
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 --manifest-path ${{ matrix.module }}/Cargo.toml
args: --release --out dist -i 3.9 -i 3.10 -i 3.11 -i 3.12 -i 3.13 -i pypy3.10 -i 3.13t --manifest-path ${{ matrix.module }}/Cargo.toml
sccache: "true"
- name: Upload wheels
uses: actions/upload-artifact@v4
Expand Down
6 changes: 3 additions & 3 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ arrow-select = "54"
bytes = "1.7.0"
half = "2"
indexmap = "2"
numpy = "0.23"
# numpy = "0.23"
numpy = { git = "https://github.com/PyO3/rust-numpy.git", rev = "8dd4f8e2934126a8323b8c8f00e69b0182b8ecee" }
object_store = "0.11"
parquet = "54"
pyo3 = { version = "0.23", features = ["macros", "indexmap"] }
Expand Down
2 changes: 1 addition & 1 deletion arro3-compute/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ fn check_debug_build(py: Python) -> PyResult<()> {
Ok(())
}

#[pymodule]
#[pymodule(gil_used = false)]
fn _compute(py: Python, m: &Bound<PyModule>) -> PyResult<()> {
check_debug_build(py)?;

Expand Down
2 changes: 1 addition & 1 deletion arro3-core/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ fn check_debug_build(py: Python) -> PyResult<()> {
}

/// A Python module implemented in Rust.
#[pymodule]
#[pymodule(gil_used = false)]
fn _core(py: Python, m: &Bound<PyModule>) -> PyResult<()> {
check_debug_build(py)?;

Expand Down
2 changes: 1 addition & 1 deletion arro3-io/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ fn check_debug_build(py: Python) -> PyResult<()> {
Ok(())
}

#[pymodule]
#[pymodule(gil_used = false)]
Copy link
Owner

@kylebarron kylebarron Mar 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this will always be true for the IO module. E.g. I might want to allow a Python HTTP library like aiohttp to be used for making the parquet file reads

Copy link
Contributor Author

@H-Plus-Time H-Plus-Time Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

True, if unmodified (as in, rust locking structures cfg-gated on Py_GIL_DISABLED) - there isn't code that allows that quite yet, right?

I should definitely take a crack at pytest-freethread in general before this is merged (at the very least CI for a single test in pyo3-arrow or core).

Thinking about it, pretty much all users of the freethreading build at the moment are other library maintainers porting their package to freethreading - this is probably a rare instance (especially given the main dependant is geoarrow.rust.io) where hard concurrency errors are preferred over RuntimeWarning .

fn _io(py: Python, m: &Bound<PyModule>) -> PyResult<()> {
check_debug_build(py)?;

Expand Down
Loading
Loading