Support for platforms where multiprocessing does not work #173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Check if multiprocessing is supported and else set multiprocessing=False. When later creating list of tasks to run we then choose to either run pool.apply_async(function), or to just eval(function). Import test for multiprocessing is taken from https://github.com/pipxproject/pipx/blob/4870ba6bd0164448c0f6b34dd663cba33b396287/src/pipx/commands/list_packages.py#L12. The full error message from trying to run pythontex on android looks like: ``` This is PythonTeX 0.17 Traceback (most recent call last): File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module> from _multiprocessing import SemLock, sem_unlink ImportError: cannot import name 'SemLock' from '_multiprocessing' (/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/data/data/com.termux/files/usr/bin/texlive/pythontex", line 62, in <module> pythontex.main() File "/data/data/com.termux/files/usr/share/texlive/texmf-dist/scripts/pythontex/pythontex3.py", line 2811, in main do_multiprocessing(data, temp_data, old_data, engine_dict) File "/data/data/com.termux/files/usr/share/texlive/texmf-dist/scripts/pythontex/pythontex3.py", line 1269, in do_multiprocessing pool = multiprocessing.Pool(jobs) File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 119, in Pool return Pool(processes, initializer, initargs, maxtasksperchild, File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 191, in __init__ self._setup_queues() File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/pool.py", line 343, in _setup_queues self._inqueue = self._ctx.SimpleQueue() File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue return SimpleQueue(ctx=self.get_context()) File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__ self._rlock = ctx.Lock() File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/context.py", line 67, in Lock from .synchronize import Lock File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 30, in <module> raise ImportError("This platform lacks a functioning sem_open" + ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. ```
Grimler91
force-pushed
the
no-multiprocessing
branch
from
190b048 to
7ff120b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi,
Currently trying to run pythontex on android gives an error due to multiprocessing not working. There's been plenty of discussion on how to deal with this in other python projects, see for example pypa/pip#8161.
With this PR we check if multiprocessing is supported and else set
multiprocessing=False. When later creating list of tasks to run we
then choose to either run pool.apply_async(function), or to just
eval(function).
Error message when currently running pythontex
I have not updated pythontex2.py, mainly due to that 99 % of all users running into this problem will be using the terminal emulator termux (as I am) where python3 is already the default python version.
Let me know if you are not happy with the code format, associated comments or if I should fix something else in the commits!