Add out-of-tree Pyodide builds in CI for numcodecs

[agriyakhetarpal]

This PR adds a CI job (emscripten-ci.yaml) that builds WASM wheels for numcodecs for use in Pyodide-based environments using the Emscripten toolchain. This was discussed in one of the community meetings for Zarr where it was said that this change would be welcome as a pull request, and follow-up work after this is complete is planned:

1. A job that publishes nighty WASM wheels for numcodecs to a PyPI-like distribution channel (not sure where, this might require requesting access on the https://github.com/scientific-python/upload-nightly-action repository (which corresponds to https://anaconda.org/scientific-python-nightly-wheels/)
2. A corresponding job for out-of-tree testing for Zarr that builds up on this job (and subsequently its nightly pushes)
3. Interactive documentation improvements with Sphinx that make use of these nightly wheels in some form (longer-term goal, soon but not very soon I guess)

TODO:

• Unit tests and/or doctests in docstrings
• Tests pass locally
• Docstrings and API docs for any new/modified user-facing classes and functions
• Changes documented in docs/release.rst
• Docs build locally
• GitHub Actions CI passes
• Test coverage to 100% (Codecov passes)

Additional context

Some xrefs for reviewers and project maintainers:

• NumPy's out-of-tree job: https://github.com/numpy/numpy/blob/main/.github/workflows/emscripten.yml and corresponding PR: CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy numpy/numpy#25894
• The same for PyWavelets (optional dependency for scikit-image): Build and test PyWavelets Pyodide wheels in CI PyWavelets/pywt#701 and Upload dev wheels to Anaconda.org + revamp wheels publishing workflow PyWavelets/pywt#714
• For the pandas repository BLD, TST: Build and test Pyodide wheels for pandas in CI pandas-dev/pandas#57896
• Add out-of-tree Pyodide builds in CI for scikit-image scikit-image/scikit-image#7350
• and so on

[agriyakhetarpal]

The workflow is currently going to fail because the file system access in Emscripten has a bug with saving .npy files – I had started on this PR roughly two weeks back, and I have now rebased on top of the changes from the main branch. Here is the related issue that I opened on the Pyodide repository a few hours ago: pyodide/pyodide#4779

@rgommers and I had fixed a similar error in PyWavelets/pywt#701 by converting all of the .npy files present to .npz files. If this is a suitable workaround at this time and does not render other PRs un-mergeable, please let me know and I would be happy to apply this fix here (it works/worked for loading .npz files, it should most likely work for saving them as well).

[joshmoore]

Thanks, @agriyakhetarpal. I've launched the workflows.

[agriyakhetarpal]

Ah, I think the miniconda action needs updating. I'll push a commit to fix that in a moment.

[agriyakhetarpal]

Never mind, looks like #528 is already on it – I can wait and rebase later after that gets merged. For the relevant jobs here, the Pyodide wheel build succeeds while the tests fail because of the reasons described above, and the rest of the test suite passes.

[agriyakhetarpal]

Marking this PR as a draft till there is a fix for this on the Pyodide and Emscripten ends. I will put my focus on an accompanying CI job and associated fixes for the Zarr repository.

[agriyakhetarpal]

It's been a while since this PR has been opened, could we get the ball rolling by temporarily skipping the previously failing test for now? I feel that numcodecs's CI can benefit from adding this workflow for the rest of the test suite, at least. I've revisited the merge conflicts in the previous few commits and updated the workflow file in 764f34e, I'll see how the builds go.

[agriyakhetarpal]

I've created pyodide/pyodide#5169 and pyodide/pyodide#5172 (please see above) for adding some missing dependencies to Pyodide. As a temporary measure, I've skipped the sole test that requires an updated version of the crc32c package: numcodecs/tests/test_checksum32.py::test_crc32c_checksum in 53f2096. I'll mark this PR as ready for review for now. @joshmoore, could you please trigger the workflows for me again?

[agriyakhetarpal]

https://github.com/agriyakhetarpal/numcodecs/actions/runs/11746955955 via my fork is the first passing workflow run, with a few caveats:

1. A (one) crc32c test is temporarily disabled (see above, will be available in the next Pyodide release)
2. zfpy is not available yet (see above, will be available in the next Pyodide release)
3. Zarr v3 tests are temporarily disabled until we upgrade to Zarr v3 in Pyodide, hopefully by the next release
4. I had to skip a doctest that was using threads in a messy manner, but it should work.

The rest of the skips/xfails are the usual WASM limitations where the multiprocessing/threading modules are not available, and we have a pretty significant portion of the test suite passing without issues. I've compiled crc32c and zfpy with Emscripten locally in the time being and triggered the test suite for numcodecs with them, which didn't reveal anything problematic. I'm open to including them in the CI job temporarily if needed – they don't take too long to compile.

[agriyakhetarpal]

Update from my previous comment above: I've cleaned up and merged the latest changes from the main branch, and agriyakhetarpal/numcodecs/actions/runs/12572995519/job/35045552245 via my fork passes all tests. We released https://github.com/pyodide/pyodide/releases/tag/0.27.0 yesterday, and I've bumped to it here.

• crc32c was upgraded to version 2.7.1 in Bump crc32c to version 2.7.1 pyodide/pyodide#5169 to fulfill the requirement, and is now available as a part of Pyodide 0.27.
• I did add zfpy in Add zfp and zfpy pyodide/pyodide#5172, but it appears that an upper bound on NumPy has been added here because it doesn't support it yet (Numpy 2.0 LLNL/zfp#210). Though, FWIW, via Update to NumPy v2 and bump versions for several dependent packages pyodide/pyodide#4925, we bumped to NumPy v2 and did not find any problems with building and/or testing zfpy against it in specific. Unfortunately, we don't have support for multiple versions per package yet in the Pyodide distribution since it contains the distribution and all the WASM wheels together, so the zfpy-specific test(s) will have to remain disabled. Work is in progress to unvendor the packages from the Pyodide runtime and is one of our goals with Pyodide 0.28 later in the year.

Overall, this is the result from the test suite: 786 passed, 49 skipped, 36 xfailed in 23.22s, which does seem quite promising.

Gentle ping, @joshmoore!

[agriyakhetarpal]

I am also happy to upstream this patch (and the one below) to Blosc if it is suggested to do so. :)

[agriyakhetarpal]

zfpy has been built in Pyodide, but against NumPy v2 (as described above). There is a chance that the tests should work with NumPy >=2, too if I were to install it separately instead of using the [zfpy] extra. I'm open to test that out if asked.

[agriyakhetarpal]

Suggested change

	on:
	# TODO: refine after this is ready to merge
	[push, pull_request, workflow_dispatch]
	on: [push, pull_request]
	concurrency:
	group: ${{ github.workflow }}-${{ github.ref }}
	cancel-in-progress: true