feat(release): Enable sdist artifacts #398

bschoenmaeckers · 2025-09-09T12:47:31Z

While testing various packages on free-threaded Python (3.13t and 3.14t), I needed to install arro3. Because there are no prebuilt wheels or a published sdist, I currently have to install arro3 directly from Git. To make this easier, I’d like to re-enable publishing of the sdist distribution.

I noticed this was previously disabled in #113. I tested the sdist locally and did not encounter any issues. If there were specific problems that led to disabling it, could you elaborate on them and confirm whether they’re now resolved?

kylebarron · 2025-09-09T14:28:24Z

I've had issues with sdists specifically in the context of workspaces. Where building the sdist for arro3-core would only include the Cargo.toml inside of the arro3-core directory, which then wouldn't compile correctly because it would be missing the top-level Cargo.toml of the workspace.

Unless I remember the problem incorrectly, it looks like maturin has been fixed to support this use case. When I build the sdist it now includes these files:

> uv run maturin build -m arro3-core/Cargo.toml --sdist -o dist
> extract dist/arro3_core-0.6.1.tar.gz
x arro3_core-0.6.1/pyo3-arrow/Cargo.toml
x arro3_core-0.6.1/pyo3-arrow/CHANGELOG.md
x arro3_core-0.6.1/pyo3-arrow/Cargo.lock
x arro3_core-0.6.1/pyo3-arrow/README.md
x arro3_core-0.6.1/pyo3-arrow/src/array.rs
x arro3_core-0.6.1/pyo3-arrow/src/array_reader.rs
x arro3_core-0.6.1/pyo3-arrow/src/buffer.rs
x arro3_core-0.6.1/pyo3-arrow/src/chunked.rs
x arro3_core-0.6.1/pyo3-arrow/src/datatypes.rs
x arro3_core-0.6.1/pyo3-arrow/src/error.rs
x arro3_core-0.6.1/pyo3-arrow/src/export.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/array.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/array_reader.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/chunked.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/datatypes.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/ffi_stream.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/field.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/input.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/mod.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/record_batch.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/record_batch_reader.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/scalar.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/schema.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/table.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/from_python/utils.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/mod.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/to_python/chunked.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/to_python/ffi_stream.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/to_python/mod.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/to_python/nanoarrow.rs
x arro3_core-0.6.1/pyo3-arrow/src/ffi/to_python/utils.rs
x arro3_core-0.6.1/pyo3-arrow/src/field.rs
x arro3_core-0.6.1/pyo3-arrow/src/input.rs
x arro3_core-0.6.1/pyo3-arrow/src/interop/mod.rs
x arro3_core-0.6.1/pyo3-arrow/src/interop/numpy/from_numpy.rs
x arro3_core-0.6.1/pyo3-arrow/src/interop/numpy/mod.rs
x arro3_core-0.6.1/pyo3-arrow/src/interop/numpy/to_numpy.rs
x arro3_core-0.6.1/pyo3-arrow/src/lib.rs
x arro3_core-0.6.1/pyo3-arrow/src/record_batch.rs
x arro3_core-0.6.1/pyo3-arrow/src/record_batch_reader.rs
x arro3_core-0.6.1/pyo3-arrow/src/scalar.rs
x arro3_core-0.6.1/pyo3-arrow/src/schema.rs
x arro3_core-0.6.1/pyo3-arrow/src/table.rs
x arro3_core-0.6.1/pyo3-arrow/src/utils.rs
x arro3_core-0.6.1/arro3-core/Cargo.toml
x arro3_core-0.6.1/arro3-core/README.md
x arro3_core-0.6.1/arro3-core/python/arro3/core/__init__.py
x arro3_core-0.6.1/arro3-core/python/arro3/core/_array.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_array_reader.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_buffer.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_chunked_array.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_core.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_data_type.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_field.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_record_batch.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_record_batch_reader.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_scalar.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_schema.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/_table.pyi
x arro3_core-0.6.1/arro3-core/python/arro3/core/py.typed
x arro3_core-0.6.1/arro3-core/python/arro3/core/types.py
x arro3_core-0.6.1/arro3-core/src/accessors/dictionary.rs
x arro3_core-0.6.1/arro3-core/src/accessors/list_flatten.rs
x arro3_core-0.6.1/arro3-core/src/accessors/list_offsets.rs
x arro3_core-0.6.1/arro3-core/src/accessors/mod.rs
x arro3_core-0.6.1/arro3-core/src/accessors/struct_field.rs
x arro3_core-0.6.1/arro3-core/src/constructors.rs
x arro3_core-0.6.1/arro3-core/src/lib.rs
x arro3_core-0.6.1/Cargo.lock
x arro3_core-0.6.1/Cargo.toml
x arro3_core-0.6.1/pyproject.toml
x arro3_core-0.6.1/python/arro3/core/_record_batch_reader.pyi
x arro3_core-0.6.1/python/arro3/core/_buffer.pyi
x arro3_core-0.6.1/python/arro3/core/_array.pyi
x arro3_core-0.6.1/python/arro3/core/_scalar.pyi
x arro3_core-0.6.1/python/arro3/core/_field.pyi
x arro3_core-0.6.1/python/arro3/core/_core.pyi
x arro3_core-0.6.1/python/arro3/core/__init__.py
x arro3_core-0.6.1/python/arro3/core/_schema.pyi
x arro3_core-0.6.1/python/arro3/core/types.py
x arro3_core-0.6.1/python/arro3/core/_chunked_array.pyi
x arro3_core-0.6.1/python/arro3/core/_array_reader.pyi
x arro3_core-0.6.1/python/arro3/core/_table.pyi
x arro3_core-0.6.1/python/arro3/core/py.typed
x arro3_core-0.6.1/python/arro3/core/_record_batch.pyi
x arro3_core-0.6.1/python/arro3/core/_data_type.pyi
x arro3_core-0.6.1/PKG-INFO

So it looks like Maturin is now

keeping the original folder structure, including the top-level Cargo.toml
Figuring out which dependencies are local path dependencies, and including pyo3-arrow in the sdist

Creating a new top-level pyproject.toml (overwriting the one already in the repo) that mostly matches arro3-core/pyproject.toml but with manifest-path injected into the [tool.maturin] section so that it knows to look into that subfolder.

[build-system]
requires = ["maturin>=1.4.0,<2.0"]
build-backend = "maturin"

[project]
name = "arro3-core"
requires-python = ">=3.9"
dependencies = ["typing-extensions; python_version < '3.12'"]
classifiers = [
    "Programming Language :: Rust",
    "Programming Language :: Python :: Implementation :: CPython",
    "Programming Language :: Python :: Implementation :: PyPy",
]
dynamic = ["version"]

[tool.maturin]
features = ["pyo3/extension-module"]
module-name = "arro3.core._core"
python-source = "python"
strip = true
manifest-path = "arro3-core/Cargo.toml"

I think at a minimum we need to add a step to the sdist CI that verifies we can install each of the sdists generated for each module.

kylebarron · 2025-09-09T14:30:17Z

I'm also open to merging #275 but don't know enough about freethreading to know what else needs to be fixed there.

bschoenmaeckers · 2025-09-09T14:38:22Z

I think at a minimum we need to add a step to the sdist CI that verifies we can install each of the sdists generated for each module.

I will try to add a test to verify.

I'm also open to merging #275 but don't know enough about freethreading to know what else needs to be fixed there.

All the tests seem to pass without obvious issues. I haven’t looked deeply enough into the codebase to say for certain, but I think experimental support should be fine. Most of the heavy lifting is done in PyO3's internals. So using safe api's is thread-safe.

kylebarron · 2025-09-09T14:41:23Z

I will try to add a test to verify.

I think you can just add a pip install ./sdist step?

It was mostly this comment about adding pytest-freethread before merging it.

If we declare

#[pymodule(gil_used = false)]

but then use Python::attach somewhere, will pyo3 fail to compile? Or do we have to remember to fix gil_used = true?

bschoenmaeckers · 2025-09-09T14:46:38Z

If we declare
#[pymodule(gil_used = false)]
but then use Python::attach somewhere, will pyo3 fail to compile? Or do we have to remember to fix gil_used = true?

Python::attach has no interaction with the GIL anymore on the free-threaded build, that's why it is not called Python::with_gil anymore. It is still used the attach the python state, and synchronize with the garbage collector. For more info see the PyO3 docs at https://pyo3.rs/v0.26.0/free-threading.html#attaching-to-the-runtime

bschoenmaeckers · 2025-09-09T14:53:19Z

I think you can just add a pip install ./sdist step?

Where should I add this test, in the test-python workflow? And should it run for every MR, just for main or on release?

kylebarron · 2025-09-09T14:55:34Z

I think you can just add a pip install ./sdist step?

Where should I add this test, in the test-python workflow? And should it run for every MR, just for main or on release?

No I think you can just add it as a step between building and uploading the sdist in the release workflow. I think the CI is too long to test on every PR. I'm happy to just have that verify before it gets pushed to PyPI

bschoenmaeckers · 2025-09-09T15:03:15Z

Done, tested the job here https://github.com/kylebarron/arro3/actions/runs/17586733255/job/49956641911

kylebarron · 2025-09-09T15:48:38Z

Thanks!

feat(release): Enable sdist artifacts

ecdc386

github-actions bot added the feat label Sep 9, 2025

Test install sdist

85d817c

bschoenmaeckers force-pushed the sdist branch from 6922569 to 85d817c Compare September 9, 2025 15:02

kylebarron merged commit 6e2bc06 into kylebarron:main Sep 9, 2025
6 checks passed

This was referenced Sep 9, 2025

chore: Prepare 0.6.2 release #399

Merged

Publish sdist to PyPI developmentseed/obstore#560

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(release): Enable sdist artifacts #398

feat(release): Enable sdist artifacts #398

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025 •

edited

Loading

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025 •

edited

Loading

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025 •

edited

Loading

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat(release): Enable sdist artifacts #398

feat(release): Enable sdist artifacts #398

Uh oh!

Conversation

bschoenmaeckers commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

bschoenmaeckers commented Sep 9, 2025

Uh oh!

Uh oh!

kylebarron commented Sep 9, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kylebarron commented Sep 9, 2025 •

edited

Loading

bschoenmaeckers commented Sep 9, 2025 •

edited

Loading

bschoenmaeckers commented Sep 9, 2025 •

edited

Loading