Hi 馃憢
I've been running into deadlocks when using parallel scene-loading in Mitsuba.
I believe I have narrowed it down to a lock inversion where:
- Thread A: Holds the GIL and is waiting to acquire the Dr.Jit state lock
- Thread B: Holds the Dr.Jit state lock and is waiting to acquire the GIL
The most relevant snippet of code is https://github.com/mitsuba-renderer/drjit/blob/master/src/python/log.cpp#L53-L70. Funnily enough, the documentation of log_callback says that it should never acquire the GIL but it does so almost immediately馃槗 .
Effectively, almost all jitc_log calls in drjit-core will acquire the GIL and will most likely already be holding the main Dr.Jit state lock. This is an issue because any other thread could already be holding the GIL and is waiting to acquire the state lock (that's the usual order of lock acquisition for something as simple as a + b in Pyhton).
Here's a reproducer I put together. If the logs are disabled, the file runs just fine. When the logs are enabled, the process should hang and attaching a debugger should show you the deadlock I described above.
import drjit as dr
import sys
import threading
import faulthandler
faulthandler.dump_traceback_later(10)
disable_logs = False
if len(sys.argv) > 1 and sys.argv[1] == "disable_logs":
disable_logs = True
dr.set_log_level(
dr.LogLevel.Disable
if disable_logs else
dr.LogLevel.Info
)
ready = threading.Event()
def evaluate() -> None:
# Some arbitrary arithmetic to create a dummy kernel
value = dr.arange(dr.llvm.Float, 1 << 16)
for offset in range(32):
value = dr.sin(value + offset)
ready.set()
# Hold Dr.Jit's state lock and try to get the GIL to log the kernel launch (reminder: interally `dr.eval` immediately releases the GIL)
dr.eval(value)
worker = threading.Thread(target=evaluate)
worker.start()
ready.wait()
# Try to retain the GIL while trying to take Dr.Jit's state lock.
while worker.is_alive():
dr.llvm.Float(1)
worker.join()
Hi 馃憢
I've been running into deadlocks when using parallel scene-loading in Mitsuba.
I believe I have narrowed it down to a lock inversion where:
The most relevant snippet of code is https://github.com/mitsuba-renderer/drjit/blob/master/src/python/log.cpp#L53-L70. Funnily enough, the documentation of
log_callbacksays that it should never acquire the GIL but it does so almost immediately馃槗 .Effectively, almost all
jitc_logcalls indrjit-corewill acquire the GIL and will most likely already be holding the main Dr.Jit state lock. This is an issue because any other thread could already be holding the GIL and is waiting to acquire the state lock (that's the usual order of lock acquisition for something as simple asa + bin Pyhton).Here's a reproducer I put together. If the logs are disabled, the file runs just fine. When the logs are enabled, the process should hang and attaching a debugger should show you the deadlock I described above.