Allow the use of arbitrary Pyodide versions

[agriyakhetarpal]

Description

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table

┌────────────┬────────────┬────────────┬───────────────────────────┬────────────┐
│ Version    │ Python     │ Emscripten │ pyodide-build             │ Compatible │
├────────────┼────────────┼────────────┼───────────────────────────┼────────────┤
│ 0.27.0a2   │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.4     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.3     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.2     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.1     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.0     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
└────────────┴────────────┴────────────┴───────────────────────────┴────────────┘

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).

---

Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given pyodide-build version. This has been implemented through an environment variable CIBW_PYODIDE_VERSION and an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).

The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful to not tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

---

Suggested CHANGELOG entry

Since I didn't find a way to add an entry without the pre-commit hook removing previous entries, I've added a few lines here based on the current state of this PR. Please feel free to suggest changes or modify these lines directly.

- 🛠 Provide [Pyodide version 0.26.4](https://github.com/pyodide/pyodide/releases/tag/0.26.4) with `cp312-pyodide_wasm32` (#2002)
- ✨ Allow the use of a custom Pyodide version to target by the use of the `CIBW_PYODIDE_VERSION` environment variable. This
is an experimental option and users are advised to look at the [compatible Pyodide versions](https://github.com/pyodide/pyodide/blob/main/pyodide-cross-build-environments.json) according to the [`pyodide-build`](https://github.com/pyodide/pyodide-build) version.

[agriyakhetarpal]

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

[agriyakhetarpal]

Generally looks good to me. I'd like to see:

1. pyodide_version override handled like all the other options
2. an override for pyodide_build_version

I'll check 1) in a bit, but for 2), we can do this with the current structure and use a newer pyodide-build version via the CIBW_DEPENDENCY_VERSIONS option (or the more specific CIBW_DEPENDENCY_VERSIONS_PYODIDE) by supplying a custom constraints file. Is this what you intended?

[agriyakhetarpal]

Also, I think I should add a test workflow (or another job through a matrix) that builds Pyodide wheels with the new CIBW_PYODIDE_VERSION variable set so that the functionality remains intact.

[henryiii]

This file gets generated, I don't see the bin/generate_schema.py being updated?

[agriyakhetarpal]

🤔 Just tried nox -s generate_schema and the changes went away. I'll check how to update the schema properly.

[henryiii]

Doesn't cibuildwheel/options.py also need updating? I don't see a .get("pyodide-versions") being read there.

[henryiii]

Wild idea (probably), but what about allowing the version to be part of the platform? So --platform pyodide would be the default version, but --platform pyodide-0.27.0a2 would be allowed, too. This would tie the pyodide-specific setting to the pyodide platform only. But this would not allow you to require a specific version inside pyproject.toml, I guess. Thoughts?

[henryiii]

RE: an override for pyodide_build_version

Can't this be done like overriding any other constrained package? You could set the versions to latest, or even pip install a specific version in before-build?

[agriyakhetarpal]

Wild idea (probably), but what about allowing the version to be part of the platform? So --platform pyodide would be the default version, but --platform pyodide-0.27.0a2 would be allowed, too. This would tie the pyodide-specific setting to the pyodide platform only. But this would not allow you to require a specific version inside pyproject.toml, I guess. Thoughts?

I would be fine with implementing this, but in principle (and to be slightly pedantic), "pyodide" would generally be the platform and "0.27.0a2" would be the version, so mixing the platform name and the version into one string sounds a bit odd to me – given that we don't have a "linux-cp313" of sorts in https://cibuildwheel.pypa.io/en/stable/options/#platform, right?

RE: an override for pyodide_build_version

Can't this be done like overriding any other constrained package? You could set the versions to latest, or even pip install a specific version in before-build?

Yes, I had the same suggestion in #2002 (comment), but TIL that before-build can also override, that is probably neater for those who would like to explore newer versions without pinning other constraints.

[henryiii]

If linux-cp313 was valid then pyodide-cp313 would be valid too, which is the same thing or helpful; I don't think that's a correct comparison. Maybe linux-manylinux2014 would be a better comparison. (Which we don't allow)

Unless someone really likes the idea, let's keep going with pyodide-version. My only thought is that it's a general config setting only applicable to a specific platform. Though we do have a few of those (the manylinux settings, for example).

[joerick]

To correct my above comment - I was confused about the version of pyodide-build versus pyodide itself - it seems that the option CIBW_PYODIDE_VERSION is still necessary, but that the ability to configure the package pyodide-build might also be useful, though is better done through CIBW_DEPENDENCY_VERSIONS.

[joerick]

It looks like the next step here is to make CIBW_PYODIDE_VERSION a proper option, with documentation. Would you like assistance with that @agriyakhetarpal ? I can certainly help with the options spec/parsing bit, though I think you might be better placed to do the documentation bit, there might be nuance/guidance I'm unaware of.

[joerick]

I've pushed a change to make the option read properly through options.py, and it removes some hardcoding of pyodide-build and emscripten - pyodide-build is already spec'd in dependency-version constraints, and an emscripten version can be read from pyodide-build's output at runtime.

Still to do -

• options TOML schema update
• tests
• docs for new option pyodide-version.