Automatic find using pyproject.toml broken?

[penguinpee]

Hi,

TL;DR

How does one configure setuptools in pyproject.toml using automatic discovery, that honors the described exclude list? What has changed making the given pyproject.toml work on newer Python tools (with deprecation warning), but silently fail in older versions of the tool?

I'm a Fedora packager and one of the packages I maintain has recently changed to using pyproject.toml for setuptools. The've dropped setup.cfg completely. This lead to a number of issues trying to build in the Fedora build environment, which builds a wheel from the sources, installs it and runs tests.

There's a couple of issues I saw. Let me describe them one by one.

Setuptools find exclusion of directories (flat layout) not working

The upstream pyproject.toml uses:

[tool.setuptools.packages.find]
exclude = ["*.test"]

That lead to all directories being included in the module. That is docs/, scripts/ and test are included, even though according to the documentation they should be filtered out.

So, I added an explicit include = ["palettable"] to make this work. Upstream merged that into master. However, it only partially solved the issue.

Deprecation warning and build failure

Running the build with the patched pyproject.toml succeeded in F38 and rawhide (Fedora development branch), but failed in F37. In F38 and rawhide a deprecation warning was shown:

/usr/lib/python3.11/site-packages/setuptools/command/build_py.py:202: SetuptoolsDeprecationWarning:     Installing 'palettable.cartocolors' as data is deprecated, please list it in `packages`.
      !!
      ############################
      # Package would be ignored #
      ############################
      Python recognizes 'palettable.cartocolors' as an importable package,
      but it is not listed in the `packages` configuration of setuptools.
      'palettable.cartocolors' has been automatically added to the distribution only
      because it may contain data files, but this behavior is likely to change
      in future versions of setuptools (and therefore is considered deprecated).
      Please make sure that 'palettable.cartocolors' is included as a package by using
      the `packages` configuration field or the proper discovery methods
      (for example by using `find_namespace_packages(...)`/`find_namespace:`
      instead of `find_packages(...)`/`find:`).
      You can read more about "package discovery" and "data files" on setuptools
      documentation page.
  !!
    check.warn(importable)

This repeats for every submodule. But they are being added to the wheel in the end.

In F37 all the submodules are left out without any warning or error:

  copying palettable/utils.py -> build/lib/palettable
  copying palettable/palette.py -> build/lib/palettable
  copying palettable/__init__.py -> build/lib/palettable

The versions of the involved packages only differ in minor / patch versions between F37 and F38, if at all. With python-setuptools-wheel being the only package with a different major version.

python3-devel 3.11.3-2.fc37 3.11.3-2.fc38
python3-pytest 7.1.3-2.fc37 7.2.2-1.fc38
pyproject-rpm-macros 1.8.0-1.fc37 1.8.0-1.fc38
python-rpm-macros 3.11-5.fc37 3.11-10.fc38
python-pip-wheel 22.2.2-3.fc37 22.3.1-2.fc38
python-setuptools-wheel 62.6.0-3.fc37 65.5.1-2.fc38

[abravalheri]

Hi @penguinpee, please find my comments about the 2 questions below:

• Setuptools find exclusion of directories (flat layout) not working

  Please note that the automatic discovery (which automatic filters out some of the directories) will only take place if you don't provide a configuration for the packages field. This is a small excerpt of what we have in the docs:

  Automatic discovery will only be enabled if you don’t provide any configuration for packages and py_modules. If at least one of them is explicitly set, automatic discovery will not take place...

  It seems that your example does not fit in this category (since you are setting tool.setuptools.packages.find). Instead your project seems to fit into the category we call Custom Discovery in our docs, which means that you are completely in control of what gets inside the package and have to explicitly use the include and exclude directives. This is a small excerpt of what we have in the docs:

  Please have in mind that find_namespace: (setup.cfg), find_namespace_packages() (setup.py) and find (pyproject.toml) will scan all folders that you have in your project directory ... make sure to properly configure the include and/or exclude accordingly.

  Most of the times include = ["name_of_your_package*"] does the trick (note the * glob pattern at the end, to match sub-packages), but sometimes exclude might also be necessary depending on how you decided to organise your source code. In your case it would seem that you need both: include = ["palettable*"], exclude = ["*.test"]

• Deprecation warning and build failure

  What is likely to be happening is that the warning was introduced between the 2 versions, prompting the users to make sure that all sub-packages are listed in the packages configuration. There might also have been a few bug fixes between the 2 versions.

  If you are moving your configuration to pyproject.toml, it is very likely that once you get it right the warning will be solved (tool.setuptools.pacakges.find will automatically consider namespaces)¹.

  If you have any further trouble with this after having figured out the correct tool.setuptools.pacakges.find configuration, please provide a minimal reproducer using stock versions of setuptools from PyPI (unpatched for specific OSs). I recommend using a docker container for reproducibility.

Footnotes

1. Packages using setup.cfg may have to change find: to find_namespace: and packages using setup.py maybe have to replace find_packages() with find_namespace_packages(). ↩

[penguinpee]

Thanks for your explanation.

I tried without any setuptools table at all. That gave me the following:

setuptools.errors.PackageDiscoveryError: Multiple top-level packages discovered in a flat-layout: ['demo', 'palettable'].

To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.

If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:

1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names

To find more information, look for "package discovery" on setuptools docs.

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist

Somehow the automatic discovery determines demo/ to be a module. But why? There's no __init__.py in that directory. It only contains two files:

demo/
├── Cubehelix Demo.ipynb
└── Palettable Demo.ipynb

Okay. So I add the following to pyproject.toml:

[tool.setuptools.packages.find]
exclude = ["demo*"]

Now the package builds, but, of course, previously excluded stuff is once again included. I don't think that approach works (well). I gave an exclude statement, in order to exclude a certain directory. What I get is a hidden include that adds stuff, that shouldn't be added and wouldn't be added if automatic discovery was working.

If this were my package, I'd go for a src layout. Much cleaner. Just keep everything that's not module related outside src/. I may suggest that to upstream. But I feel they won't be liking it.

Anyway, for the time being I will have to deal with the current layout. The documentation mentions 'New in 61.0.0'. On F37 we are currently having 62.6.0. If there's a behavioral difference between 62.6.0 and some newer version, shouldn't that be mentioned in the docs?

I meanwhile switched to using:

[tool.setuptools]
packages = ["palettable"]

I have dropped the tool.setuptools.packages.find table. That works locally and includes palettable and all its submodules using python3 -m build. Yet, it still fails in the F37 build environment. Of course, that's something we have to figure out in Fedora.

However, I find the current behavior of setuptools with pyproject.toml frustrating. To repeat, I think it's a bad decision to entirely drop the default exclude filter as soon as an include or exclude statement is added to fine tune the behavior. At the very least this should be configurable. Something like a use_default_exclude bool perhaps. Let's hope this gets improved before the setuptools + pyproject.toml leaves beta. I'd be happy to file an RFE.

PS: I will be afk for some time. I will read and reply to any follow up when I get a chance. But I don't know when that will be exactly.

[abravalheri]

Hi @penguinpee, my suggestion in the previous post for your case my suggestion would be:

[tool.setuptools.packages.find]
include = ["palettable*"]
exclude = ["*.test"]

This is probably the most robust method. In theory, just listing packages = ["palettable"] should exclude all subpackages. It might be working for you due to a build cache (?), I don't know for sure. We would need a minimal reproducer to investigate further.

Somehow the automatic discovery determines demo/ to be a module. But why? There's no init.py in that directory. It only contains two files

In Python any folder is a valid package. If you start a Python REPL in the project root and try to run import demo it will work just fine. Effectively, it is a valid import package (ref).

However, I find the current behavior of setuptools with pyproject.toml frustrating. To repeat, I think it's a bad decision to entirely drop the default exclude filter as soon as an include or exclude statement is added to fine tune the behavior. At the very least this should be configurable. Something like a use_default_exclude bool perhaps. Let's hope this gets improved before the setuptools + pyproject.toml leaves beta. I'd be happy to file an RFE.

If users want to do something custom, it is only fair that they have complete control over the include/exclude list. Some projects may use names that are excluded by default, so keeping the default exclude list creates a different problem. My personal opinion is that setuptools configuration is already too complex and difficult to maintain/explain/understand. Therefore my position is that we should avoid exploding the number of configuration options.

[penguinpee]

First of all, my apologies for getting back to you only now. I didn't find the time to pick this up again after my vacation until now. Let's hope I can still put together all the pieces...

Hi @penguinpee, my suggestion in the previous post for your case my suggestion would be:

[tool.setuptools.packages.find]
include = ["palettable*"]
exclude = ["*.test"]

This is probably the most robust method.

I'm using that now. And it seems to work. I forgot to use the asterisk previously. It's a bit obnoxious. Why would I want to exclude submodules? If I do, I could achieve that using an explicit exclude. But I probably won't hurt myself on the same stone again...

In theory, just listing packages = ["palettable"] should exclude all subpackages. It might be working for you due to a build cache (?), I don't know for sure. We would need a minimal reproducer to investigate further.

Not sure I can provide you with a minimal reproducer. But the builds I ran and their output is still available. Build 6017925 is the build that failed in F37, but succeeded with deprecation warnings in F38 and rawhide (the Fedora development / next release branch).

In Python any folder is a valid package. If you start a Python REPL in the project root and try to run import demo it will work just fine. Effectively, it is a valid import package (ref).

Yeah. Meanwhile I learned that even empty directories are considered (sub)modules wrt to package discovery.

If users want to do something custom, it is only fair that they have complete control over the include/exclude list.

My suggestion would improve on that, I think. As I explained before, it doesn't make much sense defining an exclude list, just to discover that it entails an include of previously automagically excluded modules. So, adding a bool enhances the level of control users have, instead of discovering setuptools quirks the hard way, like I had to.

My personal opinion is that setuptools configuration is already too complex and difficult to maintain/explain/understand.

👍
I absolutely agree with that statement. 😉

Therefore my position is that we should avoid exploding the number of configuration options.

How about reducing the number of configuration options? I know this will potentially break packages. But the current situation doesn't sound very maintainable either. Maybe something for the long run. Just my $0.02.