Support specifying variants for wheel builds

[mgorny]

Accept a variant-name config setting that specifies variant metadata for the built wheel. When provided, the variants will be written into wheel metadata, and the variant hash will be appended to the filename.

Example usage:

python -m build -w -Cvariant-name=x86_64::baseline::v3 -Cvariant-name=blas::impl::mkl

[rgommers]

Nice! For variant-name that's probably it indeed.

For -Cvariant I think the next part is:

• passing on the variant keys/values, if given, to the actual build invocation (meson setup). Should be in the Project._configure method I believe.
• passing on cflags/cxxflags/ldflags to meson setup as well

-Cvariant-name=x86_64::baseline::v3

This looks a bit different from what I used in wheelnext/pep_xxx_wheel_variants#9. I'd use = instead of ::, that's more idiomatic. And baseline is a numpy-specific thing, so I don't think that quite fits here. The variant variable (key) is psabi and the value is x86-64-v3. Which composes to -Cvariant-name="psabi=x86-64-v3" (quotes are optional) in the docs I wrote.

[mgorny]

Nice! For variant-name that's probably it indeed.

For `-Cvariant I think the next part is:

* passing on the variant keys/values, if given, to the actual build invocation (`meson setup`). Should be in the `Project._configure` method I believe.

So, that would assume that projects using meson-python would have to include a variant option in their meson_options.txt, is that correct?

* passing on cflags/cxxflags/ldflags to `meson setup` as well

This part I'm not sure about. We currently don't have a single consistent behavior across different PEP 517 backends, regarding their default flags. In particular, we have two possible scenarios:

1. Build environment does not define flags. In that case, build backend uses default flags.
2. Build environment defines flags. In that case, most of backends (but not all) override default flags with user-specified flags.

If the plugin was implicitly setting flags, we'd effectively be shifting from 1. to 2.

-Cvariant-name=x86_64::baseline::v3

This looks a bit different from what I used in wheelnext/pep_xxx_wheel_variants#9. I'd use = instead of ::, that's more idiomatic. And baseline is a numpy-specific thing, so I don't think that quite fits here. The variant variable (key) is psabi and the value is x86-64-v3. Which composes to -Cvariant-name="psabi=x86-64-v3" (quotes are optional) in the docs I wrote.

I've used the exact values as end up in project metadata. Note that it needs to include a "provider" (i.e. plugin name), "key" and "value". I went for what seemed like a logical plugin split here: an "x86_64" plugin that includes different features related to x86_64 CPUs (or perhaps a more generic "x86" plugin would be better?), and a "blas" plugin specifically focused on BLAS implementations.

[rgommers]

So, that would assume that projects using meson-python would have to include a variant option in their meson_options.txt, is that correct?

Yes, correct. variant-name is only within meson-python, just getting the wheel filename and metadata right. variant is "pass it on to the package itself to do the right thing", which means they have to at least accept -Dvariant=xxx.

If the plugin was implicitly setting flags, we'd effectively be shifting from 1. to 2.

It should definitely be "append" behavior rather than "overwrite" in my opinion. The latter is poor in general, and it gets progressively worse when one has more sources where flags come from (environment variables, CLI args, flags in config files, flags in build scripts). Meson has a sane hierarchy to these, and I think CMake does too? CMake is important here; setuptools behavior is bad/limited, let's ignore that for now. For CMake, what should happen if CFLAGS isn't empty and a plugin also provides cflags (scikit-build-core can pass them on with -DCMAKE_C_FLAGS I believe), then they get combined.

I've used the exact values as end up in project metadata.

Okay, that's reasonable too - let's leave that as is here then. It's just syntax after all, we can decide on it later.

[mgorny]

I've added -Dvariant now. To be honest, I don't like it. It feels like a bad design, having two options with slightly different behavior, and implicitly passing stuff to meson when we can pass it explicitly instead — but I'm not going to insist.

[rgommers]

I don't like it. It feels like a bad design, having two options with slightly different behavior, and implicitly passing stuff to meson when we can pass it explicitly instead

I'm not quite sure what you mean - what is the alternative here that's explicit passing that isn't "don't have any auto-cflags behavior at all"?

[mgorny]

Well, I don't like "auto-cflags" either. If I were to design this, I would make it independent — the frontend takes a list of variant tags, the backend takes specific configuration on how to perform the build.

[rgommers]

Well, I don't like "auto-cflags" either. If I were to design this, I would make it independent — the frontend takes a list of variant tags, the backend takes specific configuration on how to perform the build.

Okay, I think Eli and Jonathan both liked it though (e.g., wheelnext/pep_xxx_wheel_variants#9 (comment)). Also archspec did this. It seems reasonable to me to have this feature, since without any common flags most packages where maintainers don't have a lot of expertise are going to get this wrong (or at least have to spend more time to get it right) and we'll see more variations/issues.

Without it, the spelling to build it is also very verbose, like -Csetup-args=-Dc_args=${{ matrix.x86_64[1] }} -Csetup-args=-Dcpp_args=${{ matrix.x86_64[1] }} as part of the build command in wheelnext/numpy#1. I know that a lot of maintainers who aren't fluent with config-settings struggle with such syntax.

[mgorny]

I get your point and I don't disagree. However, that won't change the fact that I don't like it, and I have a gut feeling that we may be missing something significant. But well, one way to find out (that is, by implementing it and seeing how it works out).

[rgommers]

Fair enough! I agree there's potential that we are missing something, so let's keep it in mind as we go.

[mgorny]

Now with wheelnext/variantlib#12, we get labeled wheels, e.g.:

$ python -m build -nw -Cvariant-name=blas::variant::mkl -Cvariant-name=x86_64::baseline::v3
[…]
Successfully built meson_python-0.18.0.dev0-py3-none-any-60f00cf2+mkl+x86_64_v3.whl
$ python -m build -nw -Cvariant-name=blas::variant::openblas
[…]
Successfully built meson_python-0.18.0.dev0-py3-none-any-b785a480+openblas.whl

[mgorny]

Here's the logic for appending labels:

meson-python/mesonpy/__init__.py

Lines 363 to 368 in 8c752d5

	for numlabels in range(len(labels), 0, -1):
	long_name = "+".join((name, *labels[:numlabels]))
	# if labels would give us filename that's longer than 128
	# characters (124 + .whl), strip them
	if len(long_name) < 124:
	return long_name

It tries to use all labels returned by the plugins, but if the filename turns out too long, it removes them one by one, until it is short enough. As a last resort, it outputs a filename with no labels.

[rgommers]

Does this take into account that we're running this code before auditwheel is going to make it quite a bit longer?

[mgorny]

Well, it obviously can't predict what will happen once meson-python is done with it. I guess the only way to handle that is to adjust the limit — and I've put a totally arbitrary number here. I suppose we could even make it configurable via config_settings, so people can adjust when their workflows alter the tags afterwards.

[rgommers]

Makes sense. We can always tweak the number later. My gut feel is that 128 is on the generous side, but it doesn't matter too much for now.

[rgommers]

Great! The code looks pretty clean and the resulting filenames are nice. I haven't tested locally, but I think this is pretty close to what we want/need.

[mgorny]

This is now meson-python branch on this fork.