Include zstd license in distributions

[Rogdham]

Fixes #42.

[Rogdham]

🔗 Link to GitHub Actions run building the wheels as artifacts

Example with pyzstd-0.17.0-cp313-cp313-win_arm64.whl:

.
├── pyzstd
│   ├── _c
│   │   ├── __init__.py
│   │   └── _zstd.cp313-win_arm64.pyd
│   ├── __init__.py
│   ├── __init__.pyi
│   ├── __main__.py
│   ├── py.typed
│   ├── _seekable_zstdfile.py
│   └── _zstdfile.py
└── pyzstd-0.17.0.dist-info
    ├── LICENSE
    ├── LICENSE_zstd   <─────────────────────────── this contains the zstd library license
    ├── METADATA
    ├── RECORD
    ├── top_level.txt
    └── WHEEL

[markdryan]

LGTM, although you only seem to be including one of ztsd's license. Is this intentional?

[Rogdham]

Thanks for checking out!

My understanding is that I have the right to choose one of the two licenses of zstd library, and so I choose the one which is the same as pyzstd's license.

This avoids shenanigans on the value of the license mentioned in several metadata of the sdist and wheels.

[markdryan]

My understanding is that I have the right to choose one of the two licenses of zstd library, and so I choose the one which is the same as pyzstd's license.

Fair enough. Thanks for the fix!

[MattSturgeon]

In NixOS/nixpkgs#449557 I'm seeing:

Backend subprocess exited when trying to invoke get_requires_for_build_wheel
FileNotFoundError: No such file or directory: /build/source/zstd/LICENSE

Full trace

Running phase: buildPhase
Executing pypaBuildPhase
Creating a wheel...
pypa build flags: --no-isolation --outdir dist/ --wheel --config-setting=--global-option=--dynamic-link-zstd
* Getting build dependencies for wheel...
Traceback (most recent call last):
  File "/nix/store/5x6qv3l05ykslysl3k94hz4vxf88hnfs-python3.13-pyproject-hooks-1.2.0/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
    ~~~~^^
  File "/nix/store/5x6qv3l05ykslysl3k94hz4vxf88hnfs-python3.13-pyproject-hooks-1.2.0/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/5x6qv3l05ykslysl3k94hz4vxf88hnfs-python3.13-pyproject-hooks-1.2.0/lib/python3.13/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/build/source/build_script/pyzstd_pep517.py", line 10, in get_requires_for_build_wheel
    return _orig.get_requires_for_build_wheel(config_settings) + requires
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/nix/store/yibmks4i48rxjd2z44d679881gw7p8f2-python3.13-setuptools-80.9.0/lib/python3.13/site-packages/setuptools/build_meta.py", line 331, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/yibmks4i48rxjd2z44d679881gw7p8f2-python3.13-setuptools-80.9.0/lib/python3.13/site-packages/setuptools/build_meta.py", line 301, in _get_build_requires
    self.run_setup()
    ~~~~~~~~~~~~~~^^
  File "/nix/store/yibmks4i48rxjd2z44d679881gw7p8f2-python3.13-setuptools-80.9.0/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup
    exec(code, locals())
    ~~~~^^^^^^^^^^^^^^^^
  File "<string>", line 256, in <module>
  File "<string>", line 123, in do_setup
  File "<string>", line 27, in read_stuff
OSError: [Errno 40] Too many levels of symbolic links: '/build/source/zstd/LICENSE'

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

Because there is no zstd/LICENSE file.

I assume because we package zstd and pyzstd separately, that we don't need to embed the upstream license? (--config-setting=--global-option=--dynamic-link-zstd)

Either way, it seems like a bug that the build script would fail with a FileNotFoundError here.

We can apply patches in nixpkgs (or fetch patches from upstream PRs/commits), so as soon as I know the correct approach we can deal with it.

[Rogdham]

Hello @MattSturgeon, thank you for reporting the issue.

After giving it some thoughts, I think that this is a side-effect of this PR indeed. More precisely, that your build script was working before but needs to be changed slightly due to it.

I think that what happens is that when the git clone is performed, the git submodules are not initialized and cloned. As a result, the zstd source code is not available in the zstd directory, and my build script that copies license information from it fails. Could you try to initialize and clone the submodules as well? If it were a git clone command, you could use the --recurse-submodules parameter. I leave it to you to find how to do that in a .nix file 😉

[MattSturgeon]

Thanks for the explanation. That makes sense.

We can simply pass fetchSubmodules = true to fetchFromGitHub, which I can confirm does fix the build.

However, seeing as we're building pyzstd with --dynamic-link-zstd, I assume:

1. The zstd submodule is only used for its LICENSE file.
2. We don't actually need to bundle the zstd license with pyzstd, because pyzstd isn't vendoring zstd.

Therefore I'd prefer to avoid fetching the submodules unnecessarily.

I can solve (1) by doing touch zstd/LICENSE, which fixes the build, but (2) is making me wonder why we're going down this code-path in the first place...

I would expect --dynamic-link-zstd (bound to DYNAMIC_LINK in do_setup) to completely skip any attempt to read the zstd submodule, including its license. This is kinda hinted at by:

Depending on how it is build, this package may distribute the zstd library, partially or in its integrality, in source or binary form.

Unless you're telling me we still need the zstd license in this scenario, I'll try and come up with a PR to exclude the zstd license when built with a dynamically linked zstd. (PR: #51)

[Rogdham]

I assume: 1- The zstd submodule is only used for its LICENSE file.

I think this is correct

2- We don't actually need to bundle the zstd license with pyzstd, because pyzstd isn't vendoring zstd.

Even with --dynamic-link-zstd, you will be building with the headers of libzstd and shipping the resulting artifact.

I am not a layer, and I have no clue if this is enough to require the license to be included next to the artifact. The fuzzy wording on top of the generated LICENSE_zstd reflects what for me is a legal uncertainty as much as the uncertainty of how the package is built downstream.

I'll try and come up with a PR to exclude the zstd license when built with a dynamically linked zstd.

Thank you for the PR. As mentioned there, I will need to think this some more.

In any case, I will not make a release in the coming week due to other priorities, so if you want a faster resolution, I suggest to go with alternatives like one of those you mentioned. Another possibility would be to get the source distribution from PyPi instead of cloning from github (you would not have the issue there because the sdist ships the necessary files from zstd).