Proof of concept for depending on external blosc python package

[dstansby]

This is a proof of concept for depending on the blosc Python package for blosc compression, instead of using the sources we build ourselves. The goal here is to proove it's possible in as few as lines as possible.

If we go ahead with this we would need to first deprecate the following functions in numcodecs.blosc before the replacement could be done:

[
'destroy',
'init',
'compname_to_compcode',
'cbuffer_sizes',
'cbuffer_metainfo',
'err_bad_cname',
'decompress_partial'
]

TODO:

• Unit tests and/or doctests in docstrings
• Tests pass locally
• Docstrings and API docs for any new/modified user-facing classes and functions
• Changes documented in docs/release.rst
• Docs build locally
• GitHub Actions CI passes
• Test coverage to 100% (Codecov passes)

[codecov]

Codecov Report

Attention: Patch coverage is 91.02564% with 7 lines in your changes missing coverage. Please review.

Project coverage is 98.44%. Comparing base (6c0ea0f) to head (f83a176).

Files with missing lines	Patch %	Lines
numcodecs/blosc.py	90.14%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #571      +/-   ##
==========================================
- Coverage   99.92%   98.44%   -1.48%     
==========================================
  Files          62       63       +1     
  Lines        2752     2770      +18     
==========================================
- Hits         2750     2727      -23     
- Misses          2       43      +41     

Files with missing lines	Coverage Δ
numcodecs/__init__.py	100.00% <ø> (ø)
numcodecs/tests/test_blosc.py	100.00% <100.00%> (ø)
numcodecs/blosc.py	90.14% <90.14%> (ø)

... and 1 file with indirect coverage changes

[d-v-b]

looking back at the commit history, it seems like blosc has been vendored in this repo for a very long time. @alimanfoo and / or @jakirkham what do you think about this PR?

[normanrz]

Since we are using the zstd and lz4 code from within the c-blosc repo, how would we deliver those codecs when using the blosc wheels?

[alimanfoo]

Hi @dstansby, this was a while ago now but IIRC part of the original motivation for vendoring was to be able to minimise memory copies. So maybe worth a check on whether using the Python blosc package would introduce any extra memory copies.

[normanrz]

Hey @dstansby, can you remind me why it is desirable to use the external blosc package instead of the submodule? I can see a number of downsides but I am not clear about the upsides.

[dstansby]

My idea here was to do a minimal proof of concept, to see if the tests still pass (ie, data is still encoded the same way) if we depend on the blosc Python package.

For me, the advantages would be:

• Remove the need to compile c-blosc every time numcodecs is installed from source
• Remove a bunch of wrapper code from numcodecs, reducing maintenance burden

I'm not really sure if this is desirable or not overall though, and from reading various PRs and issues I think there's still an open question about the future of numcodecs. I certainly don't intend to merge this any time soon and without wider discusssion (which unfortunately I don't have the time to make happen). Hopefully this shows that it's technically possible to switch to the Python blosc package though.