The parent of setup.py is not in sys.path when using pyproject.toml

[hexagonrecursion]

I have a setup.py file that is too big. Parts of it were split into separate helper files. The files are imported like this:

import helper
from setuptools import setup
setup()

To simplify the example helper.py is empty

This works ok with pip until I try adding a pyproject.toml:

[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

Now I get:

pip install -e .
Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///home/user/example
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpgfsvhwzx
       cwd: /home/user/example
  Complete output (16 lines):
  Traceback (most recent call last):
    File "/usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 280, in <module>
      main()
    File "/usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 263, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py", line 114, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-0i4zo9yo/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 177, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-0i4zo9yo/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 159, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-0i4zo9yo/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 1, in <module>
      import helper
  ModuleNotFoundError: No module named 'helper'
  ----------------------------------------
WARNING: Discarding file:///home/user/example. Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpgfsvhwzx Check the logs for full command output.
ERROR: Command errored out with exit status 1: /usr/bin/python3 /usr/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpgfsvhwzx Check the logs for full command output.

It looks like the parent directory of setup.py is no longer in sys.path by default. Adding it manually does solve the issue:

import os
import sys

sys.path += [os.path.dirname(__file__)]
    
import helper
from setuptools import setup
setup()

Is this a bug or a feature? Is there a better way to split out parts of setup.py?

[abravalheri]

The default backend (setuptools.build_meta) will not add the project folder to the path on purpose.
That is only supported in setuptools.build_meta:__legacy__.

If you want to avoid modifying sys.path you can also try to use importlib

[hexagonrecursion]

will not add the project folder to the path on purpose

i.e. this is a feature. Good to know.

you can also try to use importlib

A quick look at the page you linked suggests that using importlib requires significantly more code than patching sys.path. Is there any benefit to going the importlib route?

If I want to split setup.py into several files, what is the recommended approach?

1. Patching sys.path?
2. Using setuptools.build_meta:__legacy__?
3. Relative imports? If so, could you please show me how or post a link to doc or tutorial?

[abravalheri]

Currently there is no recommended way, so please feel free to pick the one you like the most.

If you are going for the sys.path approach, I would personally make sure I don't have any Python file or folder in the project root that can overshadow modules of the standard library or any of the dependencies used during the build (direct or indirect/transient). For example, I would make sure to not have files like build.py, setuptools.py, pip.py, pep517.py etc...

[hexagonrecursion]

In case anyone else is experiencing the same issues: I have found the "docs" for setuptools.build_meta:__legacy__: #1652. TLDR: the difference from setuptools.build_meta is that sys.path and sys.argv[0] are populated in a way that avoids breaking "a non-trivial number of [PyPi] packages."

[abravalheri]

Thank you @hexagonrecursion for the link!

This link might also be relevant: pypa/pip#6163 (comment).