What is the purpose of -nspkg.pth?

[tjni]

Hi everyone,

The legacy namespaces code path produces this file, which it seems like the modern PEP 420 code path does not. I also saw some attempts in the commit history of enabling it only for old Python versions that were backed out. The motivation behind my question is that there is a subtle bug (?) in nixpkgs that seems related to this file, and I'm wondering if it's safe for us to delete it.

Details of the Bug (?)

In nixpkgs, a package is installed into a path that looks like /nix/store/...-{package}-{version}/lib/python310/site-packages. Unlike in a virtualenv, these paths are disjoint from each other. When we create a binary for a Python application, we take these paths of the dependencies of the application and one-by-one call site.addsitedir on them to add them to sys.path and run the .pth files.

Let's say our namespace package is a.b. When *-nspkg.pth executes, it might create a module spec for a if one does not exist and add a path pointing to /nix/store/...-a-{version}/lib/python310/site-packages/a. On the other hand, if there is a namespace package a.c that does not use the legacy path and does not come with a .pth file, its path /nix/store/...-c-{version}/lib/python310/site-packages/a is not added.

In some cases (I think when _last_parent_path of _NamespacePath is cached), it becomes impossible to import a.c.

Reproduction

1. Create a virtualenv called env1 and install sphinxcontrib-applehelp==1.0.4 in it (this does not use the legacy path).
2. Create a virtualenv called env2 and install sphinxcontrib-serializinghtml==1.1.5 in it (this does use the legacy path).
3. Create an empty virtualenv called env3.
4. Start the interpreter from env3: env3/bin/python.
5. Run:

>>> import site
>>> p = site.addsitedir('env1/lib/python3.10/site-packages', site._init_pathinfo())
>>> p = site.addsitedir('env2/lib/python3.10/site-packages', p)
>>> import sphinxcontrib.applehelp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'sphinxcontrib.applehelp'

>>> import importlib
>>> importlib.invalidate_caches()
>>> import sphinxcontrib.applehelp
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/tni/code/sphinx-test/env1/lib/python3.10/site-packages/sphinxcontrib/applehelp/__init__.py", line 19, in <module>
    import sphinx
ModuleNotFoundError: No module named 'sphinx'

[abravalheri]

I believe that the -nspkg.pth file is an enabler for the pkg_resources.declare_namespace API, and I suspect that without it it will not work when multiple packages share the legacy namespace.

This PR has an updated compatibility table about mixing legacy and new-style namespaces (all installed in the same virtual env): pypa/sample-namespace-packages#22.

I am not sure about the consequences of nixpkgs approach.