Pyodide updates: add 314.0 alpha, switch to pyemscripten platform tag (PEP 783), and remove hardcoded Emscripten location + update dependencies #2812
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -94,29 +94,28 @@ def ensure_node(major_version: str) -> Path: | |
| return path | ||
|
|
||
|
|
||
| def install_emscripten(env: dict[str, str], version: str, xbuildenv_cache_path: Path) -> Path: | ||
| """Install Emscripten via pyodide-build, which also applies Pyodide-specific patches.""" | ||
| emscripten_dir = Path( | ||
| call("pyodide", "config", "get", "emscripten_dir", env=env, capture_stdout=True).strip() | ||
| ) | ||
| with FileLock(CIBW_CACHE_PATH / "emscripten.lock"): | ||
| if emscripten_dir.exists(): | ||
| return emscripten_dir | ||
| call( | ||
| "pyodide", | ||
| "xbuildenv", | ||
| "install-emscripten", | ||
| "--force", | ||
| "--version", | ||
| version, | ||
| "--path", | ||
| str(xbuildenv_cache_path), | ||
| env=env, | ||
| cwd=CIBW_CACHE_PATH, | ||
| ) | ||
| assert emscripten_dir.exists() | ||
| return emscripten_dir | ||
|
|
||
|
|
||
| def get_all_xbuildenv_version_info(env: dict[str, str]) -> list[PyodideXBuildEnvInfo]: | ||
|
|
@@ -315,9 +314,9 @@ def setup_python( | |
| log.step( | ||
| f"Installing Emscripten {emscripten_version} and applying Pyodide-specific patches ..." | ||
| ) | ||
| emscripten_dir = install_emscripten(env, emscripten_version, xbuildenv_cache_path) | ||
|
|
||
| env["PATH"] = os.pathsep.join([str(emscripten_dir), env["PATH"]]) | ||
|
Member
There was a problem hiding this comment. I wonder if we should put this path before or after the original PATH. In pyodide-build, if a system emscripten is installed, it dominates. This is to ensure we don't shadow the user-installed emscripten. However, regarding the cibuildwheel would normally run in CI (I guess)? It is rare that people would want to install emscripten themselves, so this might make sense.
Member
Author
There was a problem hiding this comment. Good question. We have been doing That said, we can always change this behaviour and append to PATH if someone requests it at a later point in time, of course.
Contributor
There was a problem hiding this comment. I think for this one, I'm happy for cibuildwheel to control the environment. PATH can have all sorts of things on it that cause users issues, actually. I also wonder if Seems like it'd do what we want- $ PYODIDE_EMSCRIPTEN_DIR=/tmp/somedir pyodide config get emscripten_dir
/tmp/somedir |
||
|
|
||
| return env | ||
|
|
||
|
|
||
|
Member
Author
There was a problem hiding this comment. Git/GitHub shows this file as renamed and can't identify this – more accurately, the 3.12 constraints were deleted and followed by the creation of a new file for the 3.14 constraints.
Contributor
There was a problem hiding this comment. You can split them into two commits, and then we can rebase and merge instead of squash. |
|
agriyakhetarpal marked this conversation as resolved.
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -96,17 +96,19 @@ def __call__(self, build_id: str) -> bool: | |
| return False | ||
| if EnableGroup.GraalPy not in self.enable and fnmatch(build_id, "gp*"): | ||
| return False | ||
| # NOTE: Disable this when we don't have any Pyodide prereleases (e.g., 314.0.0a1+) | ||
| # When doing this, also: | ||
| # 1. update Pyodide tests in unit_test/build_selector_test.py and unit_test/options_test.py accordingly. | ||
| # 2. update Python versions for Pyodide identifiers in cibuildwheel/selector.py. | ||
| # 3. update constraints as necessary via bin/generate_pyodide_constraints.py and add/delete | ||
| # Pyodide constraints files in cibuildwheel/resources/constraints/ as necessary. | ||
| # When disabling, update the pattern to match the experimental Python version in case | ||
| # it is bumped to Python 3.15 (likely cp315-pyodide_* but could remain as 3.14 as well). | ||
| # This depends on the CPython version being used in the Pyodide runtime at the time. | ||
| if EnableGroup.PyodidePrerelease not in self.enable and fnmatch( | ||
| build_id, "cp314-pyodide_*" | ||
|
Member
There was a problem hiding this comment. I think it is slightly uncomfortable that we need to turn this on and off every time we make an alpha release. Would there be a way to remove this? As we standardized
Contributor
There was a problem hiding this comment. We still need a notion of alpha versus stable I'd say. We can change the platform after alpha releases so people should wait for a stable release to upload wheels.
Member
There was a problem hiding this comment.
Hmm, my idea was to freeze the ABI as of alpha release, and that is what I was going to say in the blog post (pyodide/pyodide-blog#66). I think we all agreed freezing the ABI earlier before making a stable release (pyodide/pyodide#5580), do you think we need another, say "beta" release for that then?
Member
Author
There was a problem hiding this comment. Yes. I agree with @hoodmane here. I think we should still make the difference between alpha and stable. This is because even for PyPI and wheels for conventional (desktop) platforms, it is advised to only test building wheels with an upcoming ABI – not upload them. Uploading to PyPI does not happen until there is a stable CPython release. Doing this also protects us in the event where may need to change the ABI after our proclamation of stabilising it, due to any bugs or regressions that get caught. We only have to do this process a few times a year, at most, so I would personally be fine with it flipping the switches on and off in this manner.
Member
There was a problem hiding this comment. I see. I thought PyPI allows uploading Python wheels when CPython alpha or beta release is out. If it doesn't, then it would make sense for us to do the same approach as well.
Member
Author
There was a problem hiding this comment. Yes, I think CPython does so starting with b1, but it is still discouraged, IIRC. @henryiii, I assume you'd be the right person to know about this topic!
Member
Author
There was a problem hiding this comment. Even then, freezing the ABI and making the announcement is still fine I think – the more eyes we receive on this, the better our adoption will be for PEP 783, as downstream package authors will be better prepared to make the jump.
Contributor
There was a problem hiding this comment. CPython is not guaranteed to be ABI stable till the last beta release (historically, everyone basically uses RC1 as the first "stable" release, and that's what cibuildwheel does, but officially it's last beta, beta 4, at least in the past). So you should not upload a wheel build with an early beta, but I don't think anyone stops it. In cibuildwheel you acknowledge it being a bad idea to upload with the IMO this should be removed whenever the ABI is stable. Doesn't have to match some special "final" release, but it has to be ensured to be stable.
Contributor
There was a problem hiding this comment. This line is required as long as the default_pyodide_version in build-platforms is set to a prerelease version. Once the first 'proper' pyodide release is out for 3.14, we'll announce it in the release notes, remove this line, our users will begin building for it, we'll never set that Then again, if the ABI will be frozen as of the first alpha, you could argue we don't need the |
||
| ): | ||
| return False | ||
|
|
||
| should_build = selector_matches(self.build_config, build_id) | ||
| should_skip = selector_matches(self.skip_config, build_id) | ||
|
|
||
|
Contributor
There was a problem hiding this comment. This is nice as a point-in-time capture of the current thinking! I'll just caution that these kinds of maintainer notes often go out-of-date or else require lots of maintenance as time goes on. Two ideas-
But, if you're happy to keep it updated @agriyakhetarpal, and it's helpful for you, this sounds good to me. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Maintaining Pyodide support | ||
|
|
||
| This page describes some short steps on how to update cibuildwheel's Pyodide platform code when the set of supported Pyodide Python versions changes – for example, when: | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| - a new Pyodide alpha release arrives with support for a new [Pyodide ABI](https://pyodide.org/en/latest/development/abi.html) (which may be tied to updates in Emscripten and CPython versions, compiler/linker flags, Rust toolchain support and so on), or | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
| - when that alpha graduates to a stable release. | ||
|
|
||
| ## Background | ||
|
|
||
| Pyodide ships/may ship two kinds of Python builds at any point in time (note that these numbers may not be up to date): | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| - **Stable** – the most recent full Pyodide release (e.g., `0.29.x` / cp313). This is enabled by default with no special `CIBW_ENABLE` flag needed. | ||
| - **Prerelease** – an alpha/beta/rc Pyodide release that uses the _next_ CPython version (e.g., `314.0.0a1` / cp314). Users must opt in with `CIBW_ENABLE: pyodide-prerelease` to build against this version. This may or may not be available at any given time, depending on the Pyodide release cycle. | ||
|
Member
There was a problem hiding this comment. As I mentioned in the previous comment, I think we should remove differentiating Pyodide alpha from stable at least in cibuildwheel-side of view. |
||
|
|
||
| The guards in `cibuildwheel/selector.py` enforce this distinction. The constraints files under `cibuildwheel/resources/` pin the exact tool versions that go with each build. | ||
|
|
||
| --- | ||
|
|
||
| ## When a new Pyodide prerelease becomes available | ||
|
|
||
| For example, consider a scenario when Pyodide ships a new `315.0.0a1` with cp315 support. | ||
|
|
||
| ### 1. Add the new Python configuration | ||
|
|
||
| In `cibuildwheel/resources/build-platforms.toml`, add an entry under `[pyodide]`: | ||
|
|
||
| ```toml | ||
| { identifier = "cp315-pyodide_wasm32", version = "3.15", default_pyodide_version = "315.0.0a1", node_version = "v24" }, | ||
| ``` | ||
|
|
||
| `version` is the CPython version string, `default_pyodide_version` is the Pyodide release to use when the user does not pin one explicitly (use the latest available alpha/beta for a prerelease entry), and `node_version` is the minimum Node.js major required by that Pyodide release — check the [pyodide-build FAQ](https://pyodide-build.readthedocs.io/en/latest/faq.html#what-node-js-version-do-i-need) for a rudimentary idea of what the correct value is. | ||
|
Contributor
There was a problem hiding this comment. The |
||
|
|
||
| ### 2. Update the prerelease guards in the selector | ||
|
|
||
| In `cibuildwheel/selector.py`, update the patterns in the `PyodidePrerelease` guards to match the new identifier: | ||
|
|
||
| ```python | ||
| if EnableGroup.PyodidePrerelease not in self.enable and fnmatch( | ||
| build_id, "cp315-pyodide_*" | ||
| ): | ||
| return False | ||
| ``` | ||
|
|
||
| Also, update the comment above it to reflect the new version numbers and the instructions for when this guard should eventually be disabled. | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### 3. Generate and pin a constraints file | ||
|
|
||
| The easiest way is to run the `update_constraints` `nox` session, which reads `build-platforms.toml` and regenerates all Pyodide constraints files automatically: | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| ```bash | ||
| nox -s update_constraints | ||
| ``` | ||
|
|
||
| Alternatively, run the generator script directly: | ||
|
|
||
| ```bash | ||
| python bin/generate_pyodide_constraints.py 315.0.0a1 \ | ||
| | uv pip compile - --python-version=3.15 \ | ||
| -o cibuildwheel/resources/constraints-pyodide315.txt | ||
| ``` | ||
|
|
||
| ### 4. Update tests and update CI configuration | ||
|
|
||
| - Update the unit tests so the new identifier is accepted by the selector with `PyodidePrerelease` enabled and rejected without it. Pyodide-specific integration tests may also need their hardcoded expected-wheel lists extended. | ||
|
|
||
| - Run the full test suite with `CIBW_PLATFORM=pyodide` and `CIBW_ENABLE=pyodide-prerelease` environment variables to make sure the new configuration is well exercised in CI. | ||
|
|
||
| - In GHA, make sure `CIBW_ENABLE: pyodide-prerelease` is set in the Pyodide CI job so that the prerelease identifier is exercised in CI. | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## When a Pyodide prerelease becomes stable | ||
|
|
||
| Pyodide uses a versioning scheme where the stable release for a given CPython version is named `[PythonMajorMinor].0.0`, so the first stable release shipping cp314 will be **`314.0.0`**. See [pyodide/pyodide#6084](https://github.com/pyodide/pyodide/issues/6084) for a rationale of this versioning scheme. | ||
|
|
||
| ### 1. Update the stable entries, and remove (or replace) the prerelease entry | ||
|
|
||
| In `build-platforms.toml`, update the former prerelease entry's `default_pyodide_version` to the new stable release (e.g. `314.0.0`) and remove the prerelease marker from the identifier if present. Remove previous prerelease entries if they are now obsolete, or update them to the next prerelease if one is available. Based on your discretion, you may choose to keep the old stable entry for a while if it is still supported by Pyodide, or remove it immediately if it is already retired. | ||
|
Contributor
There was a problem hiding this comment. Can we simplify this to say just whatever the most likely thing is? We're talking to ourselves here, if we later disagree with the advice we can deviate from it or change it.
Member
Author
There was a problem hiding this comment. I think we need to decide on this before I change this sentence, because we want to be aligned on a policy for how many Pyodide versions/ABIs we should support at a time in cibuildwheel. So far, it's always been two at a time (or even one, some time back). Please read my other comment: #2812 (comment)
Contributor
There was a problem hiding this comment.
|
||
|
|
||
| ### 2. Disable or update the prerelease guards | ||
|
|
||
| In `selector.py`: | ||
|
|
||
| - **If a new prerelease is available**: update the `fnmatch` pattern to the next identifier (e.g. `cp315-pyodide_*`) as described above. | ||
| - **If there is no new prerelease**L remove the `PyodidePrerelease` logic entirely. | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### 3. Update the constraints file | ||
|
|
||
| Run `nox -s update_constraints` to regenerate the constraints file for the newly stable version. If the entry was already in `build-platforms.toml` as a prerelease, its constraints file already exists, and this step just refreshes it against the stable Pyodide release and updates the dependencies' versions. | ||
|
|
||
| ### 4. Update tests and CI configuration | ||
|
|
||
| Update the unit tests so the newly stable identifier is accepted by the selector without needing `PyodidePrerelease` in the enable set. Pyodide-specific integration tests may also need their hardcoded expected-wheel lists extended. | ||
|
|
||
| - Run the full test suite with `CIBW_PLATFORM=pyodide` environment variables to make sure the new configuration is well exercised in CI. | ||
|
|
||
| - Remove `CIBW_ENABLE: pyodide-prerelease` from the GHA Pyodide CI job steps, as no prerelease identifiers should remain after the stable release ships. | ||
|
agriyakhetarpal marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## When an old Pyodide version is to be retired | ||
|
|
||
| ### 1. Remove the python configuration | ||
|
|
||
| Delete the entry from `build-platforms.toml`. | ||
|
|
||
| ### 2. Delete the constraints file | ||
|
|
||
| Remove `cibuildwheel/resources/constraints-pyodideXYZ.txt`. | ||
|
|
||
| ### 3. Update tests and CI configuration | ||
|
|
||
| Remove references to the old identifier from the unit tests, integration tests, and drop any expected-wheel entries for it from the test helper. Also, ensure that no CI job is still trying to build it. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need force?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need it to ensure that we always have a fresh copy of the Emscripten toolchain installed and patched. One might argue that cibuildwheel runs in CI on fresh runners and therefore this wouldn't be necessary, but I would suggest there are two reasons: