Connection failed, kernel does not exist #6435
Comments
|
@adamjstewart I see the same thing with my installation from July 12 :( |
|
Given the cross-post to I would recommend adding You might also try running the notebook via an application like papermill (e.g., |
|
Thanks for the tip! Debug didn't reveal anything (at least to my naïve eyes), but here is the log: Will try papermill now. |
|
Output from papermill: $ papermill Untitled.ipynb Untitled-out.ipynb
Input Notebook: Untitled.ipynb
Output Notebook: Untitled-out.ipynb
Executing: 0%| | 0/1 [00:00<?, ?cell/s]Error occurred while starting new kernel client for kernel 3b09e6e4-0b86-4186-b645-f3416958cfaf: Kernel died before replying to kernel_info
Executing: 0%| | 0/1 [00:01<?, ?cell/s]
Traceback (most recent call last):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/bin/papermill", line 8, in <module>
sys.exit(papermill())
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-mgslf22ooaphw5p6f4fzmpf2yck6v576/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/cli.py", line 250, in papermill
execute_notebook(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/execute.py", line 113, in execute_notebook
nb = papermill_engines.execute_notebook_with_engine(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 367, in execute_notebook
cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
return PapermillNotebookClient(nb_man, **final_kwargs).execute()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-jpp2hrr5u57wgt2xqwv4tx62uafpsgxs/lib/python3.10/site-packages/papermill/clientwrap.py", line 43, in execute
with self.setup_kernel(**kwargs):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-n24ngvr7wkodkzdrghql5gtqbwk3cv76/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-iq72qbinrhcmmpuisb2o6mi2xdq3wr4g/lib/python3.10/site-packages/nbclient/client.py", line 598, in setup_kernel
self.start_new_kernel_client()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.1.0-zl3znlsenizlfwmu5xcexw6pig5kicpm/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 160, in wrapped
return loop.run_until_complete(inner)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-n24ngvr7wkodkzdrghql5gtqbwk3cv76/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-iq72qbinrhcmmpuisb2o6mi2xdq3wr4g/lib/python3.10/site-packages/nbclient/client.py", line 559, in async_start_new_kernel_client
await ensure_async(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.1.0-zl3znlsenizlfwmu5xcexw6pig5kicpm/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 172, in ensure_async
result = await obj
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-client-8.1.0-tqn5lvwmp7uy3rvekrvrliuzzavgdyq5/lib/python3.10/site-packages/jupyter_client/client.py", line 199, in _async_wait_for_ready
raise RuntimeError(msg)
RuntimeError: Kernel died before replying to kernel_infoNotebooks (renamed with .txt suffix to allow upload): |
|
Thanks for the additional information and for trying out papermill. You're right, this didn't reveal a whole lot. On the bright side, both scenarios are producing identical results. The initial "handshake" message of I don't know spack (or python package managers in general for that matter) but am wondering if you're able to set up a conda or pip env and eliminate spack from the equation. Although there were M1-related issues when it came about initially, I haven't heard of M1/arm64 continuing to be problematic and, therefore, a tad suspicious of spack (but mostly out of my naïveté). |
|
Completely reasonable, Spack has a couple differences from other package managers. First, like Conda, each package has to be updated manually, so it could be that one of the packages that had M1 issues hasn't been updated. If you know off the top of your head which packages had issues that would help a lot. Second, like Nix and Brew, Spack installs each package to its own separate installation prefix and (optionally) later symlinks them to a common shared environment. I believe we had issues with Jupyter plugins in the past that assumed everything was in the same directory, but I think we solved those via env vars. @rchoudhary uses pip and assures me that pip-installed Jupyter works for him, so I suspect you're right that this issue is Spack-specific and a newer version of a package may help. Just not sure which packages to try to update. |
|
Looking at the |
|
Updated ipykernel and all of its direct dependencies to the latest and greatest versions but I'm still seeing the same issue. Will try recursively updating the rest of papermill's dependencies and see if it works. |
|
Ok. I don't expect the papermill-dependency exercise to yield anything since they are not common to those dependencies in the notebook/server code other than those same dependencies in ipykernel. That said, a datapoint is a datapoint. It seems like trimming down the kernel side of things would be most informative, but I'm not familiar with those kinds of applications. You might try running |
|
Yep, tried upgrading all dependencies (including zmq), still no luck.
|
|
Thanks @adamjstewart. This (IMHO) implies the issue is probably at the sockets layer - which has some merit given that's a low-level area of the stack. I'm pinging @minrk in case he has some ideas. I think one more thing worth trying is to enable debug logging on {
"argv": [
"python",
"-m",
"ipykernel_launcher",
"--debug",
"-f",
"{connection_file}"
],
"display_name": "Python 3 (ipykernel)",
"language": "python",
"metadata": {
"debugger": true
}
}You do not need to restart the notebook server but will need to open the notebook so that a new kernel is launched. It should dump quite a lot of information into the console. I've included the output up to the reception of the ipykernel debug output...
|
|
This may or may not explain things, but I don't have a kernel.json file: $ jupyter kernelspec list
Available kernels:
python3 /Users/Adam/spack/var/spack/environments/system/.spack-env/._view/7carnllnc4vuxozpkgg34llszt3okbr7/lib/python3.10/site-packages/ipykernel/resources
$ ls /Users/Adam/spack/var/spack/environments/system/.spack-env/._view/7carnllnc4vuxozpkgg34llszt3okbr7/lib/python3.10/site-packages/ipykernel/resources
logo-32x32.png logo-64x64.png logo-svg.svg |
|
It's strange that Is there a Can you list what is in the parent directory regardless? I suspect this is indeed the package installation location. Also, what happens when you run |
Nope.
Fun times on macOS with SIP enabled. If I sudo mkdir, chown, and run again, it successfully installs a kernel: It does show up now. I added the debug flag to this kernel and ran papermill again: $ papermill -k my-test-kernel Untitled.ipynb Untitled-out.ipynb
Input Notebook: Untitled.ipynb
Output Notebook: Untitled-out.ipynb
Executing: 0%| | 0/1 [00:00<?, ?cell/s][IPKernelApp] Using existing profile dir: '/Users/Adam/.ipython/profile_default'
[IPKernelApp] Searching path ['/Users/Adam/.ipython/profile_default', '/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython', '/usr/local/etc/ipython', '/etc/ipython'] for config files
[IPKernelApp] Attempting to load config file: ipython_config.py
[IPKernelApp] Looking for ipython_config in /etc/ipython
[IPKernelApp] Looking for ipython_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_config in /Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython
[IPKernelApp] Looking for ipython_config in /Users/Adam/.ipython/profile_default
[IPKernelApp] Attempting to load config file: ipython_kernel_config.py
[IPKernelApp] Looking for ipython_kernel_config in /etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /usr/local/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/etc/ipython
[IPKernelApp] Looking for ipython_kernel_config in /Users/Adam/.ipython/profile_default
[IPKernelApp] Loading connection file /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] Starting the kernel at pid: 9626
[IPKernelApp] shell ROUTER Channel on port: 64546
[IPKernelApp] stdin ROUTER Channel on port: 64548
[IPKernelApp] control ROUTER Channel on port: 64550
[IPKernelApp] iopub PUB Channel on port: 64547
[IPKernelApp] Heartbeat REP Channel on port: 64549
[IPKernelApp] Writing connection file: /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] To connect another client to this kernel, use:
[IPKernelApp] --existing /private/var/folders/jv/cgkfvslj6nq1l7cw0c8c_8gm0000gn/T/tmpw2zay9ml.json
[IPKernelApp] Seeing logger to stderr, rerouting to raw filedescriptor.
Error occurred while starting new kernel client for kernel fd44057d-34db-421f-9baa-4f602f210dba: Kernel died before replying to kernel_info
Executing: 0%| | 0/1 [00:01<?, ?cell/s]
Traceback (most recent call last):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/bin/papermill", line 8, in <module>
sys.exit(papermill())
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-click-8.1.3-phvm6ngokygjyb77gs5jxr3ueeyjt2jl/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/cli.py", line 250, in papermill
execute_notebook(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/execute.py", line 113, in execute_notebook
nb = papermill_engines.execute_notebook_with_engine(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 49, in execute_notebook_with_engine
return self.get_engine(engine_name).execute_notebook(nb, kernel_name, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 367, in execute_notebook
cls.execute_managed_notebook(nb_man, kernel_name, log_output=log_output, **kwargs)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/engines.py", line 436, in execute_managed_notebook
return PapermillNotebookClient(nb_man, **final_kwargs).execute()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-papermill-2.4.0-t7e6joq5jbsx4jtvtctq6d7zcohqyben/lib/python3.10/site-packages/papermill/clientwrap.py", line 43, in execute
with self.setup_kernel(**kwargs):
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-nqxvgunmul2o32f53yaoowwkpu2ej3ia/lib/python3.10/site-packages/nbclient/client.py", line 598, in setup_kernel
self.start_new_kernel_client()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.3.0-edicz5adwjeqyakfhzbgxvyokby4t4o3/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 166, in wrapped
return loop.run_until_complete(inner)
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/python-3.10.10-yadxcbmd2o7afi3pto7do5fd5aqjl6oo/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-nbclient-0.7.2-nqxvgunmul2o32f53yaoowwkpu2ej3ia/lib/python3.10/site-packages/nbclient/client.py", line 559, in async_start_new_kernel_client
await ensure_async(
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-core-5.3.0-edicz5adwjeqyakfhzbgxvyokby4t4o3/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line 182, in ensure_async
result = await obj
File "/Users/Adam/spack/opt/spack/darwin-ventura-m2/apple-clang-14.0.0/py-jupyter-client-8.1.0-eh43c6atq5rxccxjlcgc4latl6xdvelz/lib/python3.10/site-packages/jupyter_client/client.py", line 199, in _async_wait_for_ready
raise RuntimeError(msg)
RuntimeError: Kernel died before replying to kernel_infoSee anything useful there? |
|
Thanks for the response (and your patience)! This is helpful, although I still cannot determine the cause of the kernel's abrupt termination. I believe this issue should be investigated from the Based on the ipykernel debug output, we can determine that, in all likelihood, the kernel has completed the Barring other ideas, I think it would be interesting to instrument additional logging in @blink1073 - does it make sense to transfer this issue to ipykernel for now? I suppose it could be an issue in Also, do you happen to have any ideas here? |
|
I'd suggest trying to install |
Same issue.
Papermill is sufficient to reproduce the issue, and it doesn't depend on ipykernel in any way, so it can't be the cause of the problem. |
How did you come to the conclusion that ipykernel is not the issue? It is that process that is not starting completely enough to respond to I suspect the problem is environmental in nature and it's probably most direct to continue using debug logging to determine where the ipykernel process is (silently!) falling on its face. |
Papermill depends on nbclient, but nbclient doesn't depend on ipykernel: https://github.com/jupyter/nbclient/blob/main/pyproject.toml I do happen to have ipykernel installed (6.16.0), but if it's required to use nbclient, then the dependency is missing from its pyproject.toml. |
|
ipykernel is not a direct dependency in the Python sense. nbclient doesn't call ipykernel methods directly. It issues messages to configured ZMQ sockets (that the two processes know about via the shared connection information). This decoupling enables the ability to have different kernel implementations for various languages, etc., because this communication goes over "generic" sockets. However, in this case, nbclient is not able to start issuing cell contents as messages because the initial handshaking that occurs during a kernel's startup has not been completed successfully. |
|
This seems to be working now with the latest versions of all packages! |
|
i have the same issue running on a fresh mamba env with python 3.11.4 i have the exact same behaviour, with 5 restarts then kernel is said to have died. can't even execute a cell in a jupyter notebook :( |
|
Similar issue here. And jupyter notebook cells hang forever |
Describe the bug
I'm experiencing connection/kernel issues with my local copy of Jupyter. I'm not sure if this is the right place to report this, but I encounter the issue with both
jupyter labandjupyter notebookso it may be an issue with a common core/server dependency.To Reproduce
jupyter laborjupyter notebookprint('hello world')lineExpected behavior
I would Jupyter to find and use the kernel and run my code properly.
Screenshots
With
jupyter labI see the following log:With
jupyter notebookI see the following log:I also see a popup in the browser:
Desktop (please complete the following information):
Additional context
All Jupyter libraries are built from source using the Spack package manager. Here are all versions:
Let me know if there are any additional steps I can take to debug this or at least figure out which of these many dependencies has an issue. Also let me know if any of these dependencies should be updated.
The text was updated successfully, but these errors were encountered: