Python sub-interpreter support

[jabraham17]

#26597 added support for Python sub-interpreters in the Python interop module, but nightly testing found a number of bugs in it. I attribute this to race conditions when creating/using/destroying sub interpreters. #26614 turns off some of this testing, but this issue is to capture the failures I saw, as well as a desire to fix this.

Summary of failures

I was looking mostly at test/library/packages/Python/correctness/compareIterationPatterns.chpl and test/library/packages/Python/correctness/subInterp.chpl.

On my M1 Mac with 8 cores and Python 3.13.1, all tests ran fine. Switching to Python 3.12.7, I found subInterp.chpl would segfault on teardown of the sub-interpreter.

On a linux64 system with 48 cores and Python 3.13.0, subInterp.chpl would sometimes segfault and sometimes work. compareIterationPatterns.chpl would hang on sub interpreter creation. Notably, lowering the number of tasks with CHPL_RT_NUM_THREADS_PER_LOCALE would allow the test to continue. CHPL_RT_NUM_THREADS_PER_LOCALE=8 was fine, CHPL_RT_NUM_THREADS_PER_LOCALE=16 was not.

[bradcray]

Are sub-interpreters a new-ish feature in Python? Given the closeness of the version numbers you quote, is it likely that the bugs are still being worked out of them on the Python side?

Is this something that could be reproduced in a pure Python program, and if so, is there a forum where we could report it / ask about it?

[jabraham17]

They've been around for a bit, but they have changed a lot in the Python 3.12/3.13 timeframe. So it could be a Python problem (this was my initial assumption in the differences between the 3.13.1 on my laptop and 3.13.0 on the nightly machine). However, I find it equally as likely that the Python module makes a bad assumption about sub-interpreters and is using them wrong.