Share running kernels between processes #811
Comments
|
It's not possible like that, because both kernel managers don't share their state. You can see it if you list the kernels in the second process: print(mkm.list_kernel_ids())
# shows []You would need to connect to the same kernel using a new set of ZMQ sockets. This means that if you start the kernel in the first process, the second process should access the connection file that was created in the first process. |
|
@davidbrochart thank you for your reply! How to get information about the connection file? Is there any from jupyter_client import MultiKernelManager
mkm = MultiKernelManager()
km = mkm.get_kernel("some-kernel-id")
# how to get connection file information?
connection_file = km.get_connection_file() |
|
You should get the connection file from the process that started the kernel: from jupyter_client import MultiKernelManager
mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id")
km.connection_file
# 'kernel-some-kernel-id.json'The connection file was written in the current directory with the name |
|
EDIT: Actually, I just realized that the I also realized that David's comment about the file's location is correct, although I think that might be a bug. If we don't write the connection file to well-known location, then its nearly impossible for other applications to use the same information. (Perhaps I'm missing something here as well.) |
|
@davidbrochart @kevin-bates thank you for responses! I run David's code and the connection file is not created. I've run I've created two files # process1.py
import time
import json
from jupyter_client import MultiKernelManager
mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id") # the kernel connection file is not created
print(km.connection_file)
print(km.get_connection_info())
# manually save connection info
with open("my-kernel.json", "w") as fout:
info = km.get_connection_info()
info["key"] = info["key"].decode("utf-8")
print(info)
fout.write(json.dumps(info))
time.sleep(60) # sleep so second process can connect# process2.py
import json
from jupyter_client import KernelClient
info = json.load(open("my-kernel.json"))
kc = KernelClient()
kc.load_connection_info(info)
print(kc)
# how to check if KernelClient is connected?
# hot to get access to KernelManager?
print(kc.history()) # it throws error |
|
In the second process you need either a blocking or an asynchronous kernel client. I don't think you need a kernel manager in the second process as the kernel is already managed in the first process. # process2.py
import json
from jupyter_client import BlockingKernelClient
info = json.load(open("kernel-some-kernel-id.json"))
kc = BlockingKernelClient()
kc.load_connection_info(info)
kc.execute("with open('abc', 'w') as f: f.write('def')")
# file abc is created |
|
@davidbrochart thank you! It works. # process1.py
import time
import json
from jupyter_client import MultiKernelManager
mkm = MultiKernelManager()
mkm.start_kernel(kernel_name="python3", kernel_id="some-kernel-id")
km = mkm.get_kernel("some-kernel-id")
print(km.connection_file)
print(km.get_connection_info())
with open("my-kernel.json", "w") as fout:
info = km.get_connection_info()
info["key"] = info["key"].decode("utf-8")
print(info)
fout.write(json.dumps(info))
kc = km.blocking_client()
kc.execute_interactive("a = 13") # define variable `a`
time.sleep(60)# process2.py
import json
from jupyter_client import BlockingKernelClient
info = json.load(open("my-kernel.json"))
kc = BlockingKernelClient()
kc.load_connection_info(info)
kc.execute_interactive("print(a)") # prints variable `a` from the process1.pyI first run Anyway, the kernel connection info is not created automatically. I write it manually. Is it a bug? |
🎉
Could be, it would be great if you could investigate and maybe come up with a PR! |
|
I would be super happy to help. Any tips where to start investigation? Which files to look? Maybe there are some tests that I can check/create? |
|
Thanks for looking into this and I apologize for my confusing comments. The provisioner returns the connection information to the kernel manager here where it should be written to the file. |
|
@kevin-bates @davidbrochart I found that the kernel information is saved but in different directory. I was looking for kernel files in Thank you very much for your help!!! |
|
@pplonski - for completeness, the "Notebook Server" (i.e., notebook or jupyter_server) will set the Since it sounds like you want some coordination to occur between processes, you might want to look at setting |
|
Thank you @kevin-bates for help! |
|
I have a few more questions:
|
|
One of the things built into provisioners is the ability to get and load information about the provisioner and the connection to the kernel it is provisioning. The idea is that a This is primarily applicable to remote kernels where the node on which jupyter_client was running has gone down, yet the kernel, being remote, is still running, but it could be used for processes locally (and assuming the first process doesn't bring the kernel down with it via signal handling, etc.) |
|
Thank you @davidbrochart and @kevin-bates for responses! I've created two files based on your responses:
|
It is currently considered alive if it launched the kernel. |
|
@davidbrochart thank you! I've checked and it is impossible to have two It might be an option to create some mock Provisioner that will take process ID of running kernel. It might be possible if Provisioner process is only needed to start and kill a kernel process. |
Is it possible to run a kernel in one process and have access to it in the second process?
Example code:
first process (file
start.py):second process (file
getkernel.py):Is it possible to run processes in the terminal:
and have access to kernel
some-kernel-idingetkernel.pyprocess?The text was updated successfully, but these errors were encountered: