Kernels dies immediately due to library name collisions

[sdenton4]

Hello! I am experiencing a very strange issue where kernels are dying immediately due to a failure to import asyncio in zmq.

[I 2023-09-20 23:08:05.213 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel_launcher.py", line 15, in <module>
    from ipykernel import kernelapp as app
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel/__init__.py", line 5, in <module>
    from .connect import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel/connect.py", line 11, in <module>
    import jupyter_client
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/jupyter_client/__init__.py", line 3, in <module>
    from .asynchronous import AsyncKernelClient
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/jupyter_client/asynchronous/__init__.py", line 1, in <module>
    from .client import AsyncKernelClient  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/jupyter_client/asynchronous/client.py", line 5, in <module>
    import zmq.asyncio
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/zmq/asyncio.py", line 9, in <module>
    import asyncio
  File "/usr/lib/python3.11/asyncio/__init__.py", line 26, in <module>
    coroutines.__all__ +
    ^^^^^^^^^^
NameError: name 'coroutines' is not defined. Did you mean: 'iscoroutine'?

This occurs when using the poetry-installed dependencies for our project here:
https://github.com/google-research/perch

And the following is the output from jupyter --version:

Selected Jupyter core packages...
IPython          : 8.15.0
ipykernel        : 6.25.2
ipywidgets       : 8.1.1
jupyter_client   : 8.3.1
jupyter_core     : 5.3.1
jupyter_server   : 2.7.3
jupyterlab       : 4.0.6
nbclient         : 0.8.0
nbconvert        : 7.8.0
nbformat         : 5.9.2
notebook         : 7.0.4
qtconsole        : 5.4.4
traitlets        : 5.10.0

Observations:

• Opening an ipython session (inside the virtual environment), I can run import asyncio and from ipykernel import kernelapp as app without producing the error, suggesting that the problem is somehow related to the frozen_runpy environment?
• The problem has now been observed on my own machine, and on the machine of a colleague in a completely unrelated institution.
• My colleague started seeing the issue on August 29th, after successfully running our notebooks previously.
• For whatever reason, in asyncio.__init__, base_events.__all__ does not trigger an error, but coroutines does.

Various things tried so far:

• Upgraded project from to Python 3.10 to 3.11. Doesn't help.
• Downgraded Jupyter-client from 1.3.1 to 1.3.0 (released in June). Doesn't help.
• Downgraded pyzmq from 25.1.1 to 24.0.0. Doesn't help.

I am completely flummoxed.

[sdenton4]

Removing almost all dependencies from the pyproject.toml file (leaving only python, notebook, and numpy) yields this error when starting a new kernel in an empty notebook:

[I 2023-09-20 23:43:13.634 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (5/5), new random ports
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel_launcher.py", line 15, in <module>
    from ipykernel import kernelapp as app
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel/__init__.py", line 5, in <module>
    from .connect import *  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/tomdenton/.cache/pypoetry/virtualenvs/chirp-K3KuLXgo-py3.11/lib/python3.11/site-packages/ipykernel/connect.py", line 8, in <module>
    from subprocess import PIPE, Popen
  File "/usr/lib/python3.11/subprocess.py", line 49, in <module>
    import signal
  File "/home/tomdenton/source/perch/chirp/signal.py", line 22, in <module>
    import jax
ModuleNotFoundError: No module named 'jax'

It's a bit surprising that the subprocess import signal is trying to load code from a user module. (not surprising that jax isn't found, since I removed jax from the virtual env for the experiment.)

[minrk]

Can you show the output of ls "/home/tomdenton/source/perch/chirp"? Try running your script in an empty directory, without any .py files.

Running python scripts adds the 'script directory' (and PYTHONPATH, if defined) to sys.path at the highest priority (i.e. before the standard library!). For interactive Python sessions like IPython, that means the current directory (/home/tomdenton/source/perch/chirp/ in your case, I think). This means that any .py file or folder whose name collides with another Python module, including in the standard library. The consequences are clear, seeing your signal.py being imported instead of the standard library signal module. I'm guessing a similar collision is the cause of the asyncio import failure as well, even though the error is delayed somewhat.

Interactive Python sessions add the current working directory as the first entry in sys.path, meaning that any file in the current directory (e.g. signal.py) will be selected before the standard library. With later Python versions, this can happen with directories, as well. IPython tries to do this at lower priority than Python's own script_dir behavior, to avoid exactly this issue.

You can try starting with the environment variable PYTHONVERBOSE=1, which will show you all the things Python imports and from where.

[sdenton4]

OK! That's progress!

I added an 'empty_dir' and running jupyter notebook from there lets me compute 2+2 without any kernel death. So it is most likely an obscure collisions problem.

My base path looks like this:

(chirp-py3.11) tomdenton@tomdenton1:~/source/perch$ ls -lh
total 436K
-rw-r-----  1 tomdenton primarygroup   11 Sep 20 14:38 AUTHORS
drwxr-x--- 16 tomdenton primarygroup 4.0K Sep 20 23:43 chirp
-rw-r-----  1 tomdenton primarygroup 1.1K Sep 20 14:38 CONTRIBUTING.md
-rw-r-----  1 tomdenton primarygroup  12K Sep 20 14:38 LICENSE
-rw-r-----  1 tomdenton primarygroup 398K Sep 20 23:52 poetry.lock
-rw-r-----  1 tomdenton primarygroup 1.1K Sep 20 23:50 pyproject.toml
-rw-r-----  1 tomdenton primarygroup 1.3K Sep 20 14:38 README.md

So that means we've got a solid understanding of the cause, and need to come up with an appropriate fix. We can't just always run the notebook from an empty directory because:

a) The chirp directory contains lots of code and libraries which we'll need to use in our notebooks.

b) Running jupyter notebook from the empty directory means we can't navigate to the python notebooks in the Jupyter UX, unfortunately.

Thanks for your help so far! Very happy to hear any additional ideas for mitigation.

[sdenton4]

Alright, further meditation has helped.

The problem is that the notebooks are inside the 'chirp' directory alongside some files whose names conflict with some deep core modules. (like, for example, 'signal'.)

When we start a new notebook in the root directory, there are no conflicts, because all of the files/modules will be of the form 'chirp.signal'. But if I start the notebook from inside the chirp directory, signal conflicts because it isn't under our clever namespace.

The solution then is to move our notebooks to a non-conflicting place (liek the top-level directory) and add a chirp. to the start of all of our import statements.

[minrk]

The solution then is to move our notebooks to a non-conflicting place (liek the top-level directory) and add a chirp. to the start of all of our import statements.

👍 it's a lot easier to pick one name that avoids conflicts than checking every single filename.

You should also be able to avoid the collision problem with c.InteractiveShellApp.ignore_cwd = True in your ipython_config.py. However, this would also make your local directory imports fail, if you want to do them at all.

I'll close this as there isn't really anything for jupyter-client (or more precisely IPython) to do about it beyond the one option above. It's a general Python issue.