DeprecationWarning when using multiprocessing with Python 3.12

[kanti92]

From time ago, I get this warning:

DeprecationWarning: This process (pid=9288) is multi-threaded, use of fork() may lead to deadlocks in the child.

when I use VIP routines with nproc>1.

[VChristiaens]

Thanks for reporting this warning @kanti92 .

I hadn't seen the warning in my usual Python 3.10 VIP environment. I suspect Python 3.12 may be more strict about it.
By default the fork multiprocessing starting method (faster although less safe) is chosen when available on the machine attempting to run VIP in multiprocessing.

After some thoughts, it may be better to adapt the priority order regarding the method to adopt. I'll do some more research about it.

That being said, there seems to be a significant increase in computing time in your multiprocessing attempt. Just to be sure: did you set the following environment variables before running your script to prevent numpy from multithreading?

export OMP_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export MKL_NUM_THREADS=1

If not helping with computer efficiency, does setting the above variables and running your script in a Python 3.10 environment help? I don't get the warning message nor a drop in efficiency when multiprocessing a big cube with pca_annular (but the opposite as expected) on my side.

[GillesOrban]

Hi ! After investigation, here are a few findings:
1/ I can duplicate this DeprecationWarning with Python 3.12 on Linux, but not on Mac OS Python 3.12
2/ I found out that VIP is forcing to use fork via multiprocessing.set_start_method("fork", force=True) in utils_conf.py, but fork is deprecated. This error does not appear on Mac because I think the 'fork' method is not available on Mac, and it uses probably 'spawn'.

A quick fix is to replace 'fork' by 'forkserver' line 481 of utils_conf.py.

Side note, I also see that utils_conf.py is first filtering error (line 479) and then resetting warning (line 488). This has the negative effect that a user cannot filter warnings / errors according to his/her preferences.

[VChristiaens]

Thanks for investigating @GillesOrban!
Yeah I was also thinking to change the priority order such that the 'forkserver' starting method would be preferred when available on the machine running VIP. Probably safer to avoid 'fork' altogether, and just fall back on the 'spawn' method in for machines where 'forkserver' is not available.