feat(arro3): Freethreading - module markers + CI #275

H-Plus-Time · 2024-12-16T09:38:53Z

Yet another fork (this time in setup-python), what exactly was it about setup-uv in windows that broke builds?

Verified wheel building run over at https://github.com/H-Plus-Time/arro3/actions/runs/12349504140.

kylebarron · 2024-12-16T14:12:13Z

I was hitting this #263. If you find a way to use uv on Windows I'd be happy to use it. Maybe it's an issue with PATH?

kylebarron · 2024-12-16T15:30:56Z

It's also an issue that the buffer protocol is even less sound in Python 3.13t.

See discussion: https://discord.com/channels/1209263839632424990/1310572181138178129/1310572181138178129, https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/, PyO3/pyo3#2824

I'm not sure what to do here. It's very beneficial to have the buffer protocol as an option. So perhaps just document to Python users that it's unsafe to mutate the Python buffer after passing it to rust?

H-Plus-Time · 2024-12-17T14:58:15Z

Yeah - I'm actually having a great deal of difficulty thinking of buffer protocol scenarios where freethreading would be unsound but not GIL-enabled; there's definitely a difference, but the complete lack of immutability guarantees is 99% of the problem (and serialized python execution (the GIL) addresses the remaining 1%).

I think the safety note on into_arrow_array, surfaced prominently should do the trick.

By the sounds of (amongst other parts of that document) https://py-free-threading.github.io/porting/#dealing-with-thread-unsafe-objects and the comments on how numpy is going ahead with this, pathological cases (like resizing an array while it's being read) can just be left as "you're holding it wrong, enjoy the segfault", right up until the first bug report.

kylebarron · 2024-12-17T16:00:22Z

Hmm it looks like I can't trigger the wheel builds on this repo from your PR.

I think the safety note on into_arrow_array, surfaced prominently should do the trick.

Ideally this would also be surfaced to Python to be seen by Python users, but I'm not sure the right level of obtrusiveness for that.

kylebarron · 2024-12-17T16:01:23Z

It would be nice if we didn't have to use a fork of numpy. Do you know if there's progress towards getting that merged?

H-Plus-Time · 2024-12-20T12:25:15Z

Hmm it looks like I can't trigger the wheel builds on this repo from your PR.

Yeah, I noticed that in my own fork, though that seems to be fine now 🤔. It's likely a matter of: jobs can be triggered manually via workflow_dispatch, but not on pull requests (the wheels would need on: pull_request, which might be ok with cancel-in-progress); there doesn't seem to be a non-convoluted way to do manually triggered PR checks.

H-Plus-Time · 2024-12-20T13:21:09Z

Mm, not yet, probably mid-January (given the whole Christmas/NYE period is upon us) - it looks like it's down to a single mutex, probably test reconfiguration too.

Obtrusiveness-wise, I can stick a (concise) spiel in the docstring of Array::frombuffer; I'd be hesitant to log anything at runtime higher than trace or debug level.

kylebarron · 2025-02-11T17:25:00Z

I'm inclined not to merge this until numpy gets a stable release including the free-threading feature

H-Plus-Time · 2025-02-12T13:44:45Z

Yep, I'll stick this one (and related) in draft.

kylebarron · 2025-02-21T18:35:33Z

Numpy merged its free-threading pr: PyO3/rust-numpy#471 (comment)

H-Plus-Time · 2025-02-23T00:25:35Z

Excellent, hopefully shouldn't be too much longer until that gets a release. I'll re-pin to the tip of main and see if anything strange crops up in the build in the meantime.

H-Plus-Time · 2025-02-23T01:07:37Z

Hrm, blocked by the pyO3 breaking change in 0.23.4 involving the IntoPyObject constraint and Chrono Tz. With the suggested workaround (map(|v| v.fixed_offset()) specifically), seems to compile fine. By the looks of apache/arrow-rs#7173 and associates, everything should pan out roughly around the same time (the numpy release + patch to obviate the fixed_offset workaround)

…ompile without subsequent commit (avoid keeping it though, since it erases timezone info))

kylebarron · 2025-02-23T02:08:39Z

Hrm, blocked by the pyO3 breaking change in 0.23.4 involving the IntoPyObject constraint and Chrono Tz.

Can you merge in latest main? It should be fixed (with a temporary fixed offset workaround)

.github/workflows/wheels.yml

kylebarron · 2025-03-25T16:24:55Z

With #304 it looks like this might actually be unblocked?

kylebarron · 2025-03-25T17:05:17Z

arro3-io/src/lib.rs

 }

-#[pymodule]
+#[pymodule(gil_used = false)]


I'm not sure this will always be true for the IO module. E.g. I might want to allow a Python HTTP library like aiohttp to be used for making the parquet file reads

True, if unmodified (as in, rust locking structures cfg-gated on Py_GIL_DISABLED) - there isn't code that allows that quite yet, right?

I should definitely take a crack at pytest-freethread in general before this is merged (at the very least CI for a single test in pyo3-arrow or core).

Thinking about it, pretty much all users of the freethreading build at the moment are other library maintainers porting their package to freethreading - this is probably a rare instance (especially given the main dependant is geoarrow.rust.io) where hard concurrency errors are preferred over RuntimeWarning .

H-Plus-Time · 2025-03-25T23:28:46Z

With #304 it looks like this might actually be unblocked?

Oh, excellent - I gather the obstore py03 0.24.0 + object store 0.11 release unblocks this too (via #304 probably), right? That was my last blocker on this (the parquet crate's dependency on object store was involved in there somewhere).

Thought I'd have to wait for the April parquet release.

kylebarron · 2025-03-25T23:55:05Z

Yeah I made compatibility releases https://github.com/developmentseed/obstore/tree/main/pyo3-object_store#version-compatibility

…on version

bschoenmaeckers · 2025-07-03T07:34:48Z

I can take a closer look if anything is still depending on the GIL if you want.

there are some functions that take in a py: Python token, and so those presumably would change this.

FYI, Using the Python token does not mean that a function is GIL dependent. It means it requires the python thread state to be initialised.

bschoenmaeckers · 2025-09-09T16:19:08Z

pyproject.toml

 description = "Add your description here"
 readme = "README.md"
-requires-python = ">=3.11"
+requires-python = ">=3.13"


This should be reverted before merging.

The top-level pyproject.toml is only for development. This change only means that to run the tests you need to be using 3.13+.

I renamed this name to "dev-environment" to make that clearer.

Regardless, might not need to pin 3.13 here.

kylebarron · 2025-09-09T16:46:56Z

According to pytest-freethreaded it's also important to run with pytest --require-gil-disabled

When I run this locally with the GIL disabled I get a segfault, so something is wrong? @bschoenmaeckers do you have an idea?

> uv run --python 3.13t pytest --require-gil-disabled
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.13.6, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/kyle/github/kylebarron/arro3
configfile: pyproject.toml
plugins: freethreaded-0.1.0
collecting ... Fatal Python error: Segmentation fault

Current thread 0x00000002043560c0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1320 in create_module
  File "<frozen importlib._bootstrap>", line 813 in module_from_spec
  File "<frozen importlib._bootstrap>", line 921 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/kyle/github/kylebarron/arro3/arro3-compute/python/arro3/compute/__init__.py", line 2 in <module>
  File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1026 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "/Users/kyle/github/kylebarron/arro3/tests/compute/test_aggregate.py", line 3 in <module>
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/assertion/rewrite.py", line 186 in exec_module
  File "<frozen importlib._bootstrap>", line 935 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1331 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1360 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1387 in _gcd_import
  File "/Users/kyle/.local/share/uv/python/cpython-3.13.6+freethreaded-macos-aarch64-none/lib/python3.13t/importlib/__init__.py", line 88 in import_module
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/pathlib.py", line 587 in import_path
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 498 in importtestmodule
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 551 in _getobj
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 280 in obj
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 567 in _register_setup_module_fixture
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/python.py", line 554 in collect
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 389 in collect
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 344 in from_call
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 391 in pytest_make_collect_report
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/runner.py", line 567 in collect_one_node
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 839 in _collect_one_node
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 974 in genitems
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 979 in genitems
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 979 in genitems
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 979 in genitems
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 813 in perform_collect
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 353 in pytest_collection
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 342 in _main
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/Users/kyle/github/kylebarron/arro3/.venv/lib/python3.13t/site-packages/_pytest/config/__init__.py", line 201 in console_main
  File "/Users/kyle/github/kylebarron/arro3/.venv/bin/pytest", line 10 in <module>

kylebarron · 2025-09-09T17:07:54Z

Now I'm able to reproduce the segfault on CI as well https://github.com/kylebarron/arro3/actions/runs/17590097733/job/49968516771?pr=275

cc @H-Plus-Time as well

bschoenmaeckers · 2025-09-09T17:13:37Z

I’m not entirely sure what happens here without a debug build of python. Does this error also occur on 3.14?

kylebarron · 2025-09-09T18:38:21Z

Does this error also occur on 3.14?

I tried

uv run --python 3.14t pytest --require-gil-disabled

But that doesn't work out of the box because pyarrow doesn't have a built wheel for 3.14t and I don't want to figure out how to build it. We could split out the tests to not require pyarrow but I don't have time to do that now.

kylebarron · 2025-10-03T13:28:45Z

I updated package versions (esp maturin) and now CI seems to pass 🤷‍♂️.

Thanks all for your work here!

H-Plus-Time added 3 commits December 12, 2024 18:36

Opt-in to gil_used = false, pin to rust-numpy git dep (for now)

8e030ba

Add 3.13t to all builds

13f5bc2

Minimum CI changes to fix 3.13t builds in CI

3e4d851

H-Plus-Time marked this pull request as draft February 12, 2025 13:44

Pin to main branch of rust-numpy, boost pyo3 to match (NB: does not c…

9f7ef44

…ompile without subsequent commit (avoid keeping it though, since it erases timezone info))

Merge remote-tracking branch 'upstream/main' into freethreading

fd1726b

H-Plus-Time force-pushed the freethreading branch from b4caf36 to fd1726b Compare February 24, 2025 01:20

kylebarron reviewed Mar 25, 2025

View reviewed changes

.github/workflows/wheels.yml Outdated Show resolved Hide resolved

kylebarron added 2 commits March 25, 2025 13:00

Merge branch 'main' into freethreading

a1a904d

use upstream actions/setup-python

e4a6f49

kylebarron marked this pull request as ready for review March 25, 2025 17:01

kylebarron reviewed Mar 25, 2025

View reviewed changes

H-Plus-Time added 2 commits March 26, 2025 19:54

Add pytest-freethreaded, boost (hopefully just dev/test-related) pyth…

0c01ce7

…on version

Unmark io as freethreading-safe, exclude io tests from concurrent tests

728a8d2

kylebarron mentioned this pull request Jul 11, 2025

Python: Missing methods on async_tiff.store classes developmentseed/async-tiff#111

Closed

kylebarron mentioned this pull request Sep 9, 2025

feat(release): Enable sdist artifacts #398

Merged

Merge branch 'main' into freethreading

9708633

bschoenmaeckers suggested changes Sep 9, 2025

View reviewed changes

kylebarron added 2 commits September 9, 2025 12:35

Add comments to tests

d55c441

Split docs deps into separate group

b146b39

kylebarron changed the title ~~Freethreading - module markers + CI~~ feat(arro3): Freethreading - module markers + CI Sep 9, 2025

github-actions bot added the feat label Sep 9, 2025

kylebarron added 4 commits September 9, 2025 12:38

Add pre-commit as dev dep

20e59e8

update cargo lockfile

711724e

regen lockfile

ea1bb3f

Add test with --require-gil-disabled

5457247

kylebarron added 4 commits September 9, 2025 12:50

install python with uv in ci

1876172

typo

b489b0a

reduce requires-python

97e4b97

pytest-freethreaded only on 3.13+

1db2474

kylebarron added 5 commits October 3, 2025 15:03

Merge branch 'main' into freethreading

1125fff

bump maturin

154b9f6

bump maturin in lock

4739800

version reordering

9c4ddbc

lock

b692107

kylebarron approved these changes Oct 3, 2025

View reviewed changes

kylebarron merged commit 5c3d865 into kylebarron:main Oct 3, 2025
8 checks passed

kylebarron mentioned this pull request Oct 13, 2025

Free-threading & free-threaded wheels developmentseed/obstore#580

Open

feat(arro3): Freethreading - module markers + CI #275

feat(arro3): Freethreading - module markers + CI #275

Uh oh!

Conversation

H-Plus-Time commented Dec 16, 2024

Uh oh!

kylebarron commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylebarron commented Dec 16, 2024

Uh oh!

H-Plus-Time commented Dec 17, 2024

Uh oh!

kylebarron commented Dec 17, 2024

Uh oh!

kylebarron commented Dec 17, 2024

Uh oh!

H-Plus-Time commented Dec 20, 2024

Uh oh!

H-Plus-Time commented Dec 20, 2024

Uh oh!

kylebarron commented Feb 11, 2025

Uh oh!

H-Plus-Time commented Feb 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylebarron commented Feb 21, 2025

Uh oh!

H-Plus-Time commented Feb 23, 2025

Uh oh!

H-Plus-Time commented Feb 23, 2025

Uh oh!

kylebarron commented Feb 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

kylebarron commented Mar 25, 2025

Uh oh!

kylebarron Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

H-Plus-Time Mar 26, 2025

Choose a reason for hiding this comment

Uh oh!

H-Plus-Time commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylebarron commented Mar 25, 2025

Uh oh!

bschoenmaeckers commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bschoenmaeckers Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

kylebarron Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

kylebarron commented Oct 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kylebarron commented Dec 16, 2024 •

edited

Loading

H-Plus-Time commented Feb 12, 2025 •

edited

Loading

kylebarron commented Feb 23, 2025 •

edited

Loading

H-Plus-Time commented Mar 25, 2025 •

edited

Loading

bschoenmaeckers commented Jul 3, 2025 •

edited

Loading