Process mode does not respect the number of threads

[bcbrock]

Hi Ran,

Thank you for fixing the issues I mentioned a few days ago. I noticed another issue recently: When using processes, you can not limit the number of running processes, because _run_via_pool() is using the threading semaphore, and the task has already started in a new process when it acquires the semaphore. (This is important to me because I'll run out of memory if all of the processes run simultaneously).

If you'd like I could suggest a fix via a pull request, but I think all that's needed is to import both types of Semaphore and make sure the correct one is used in createPool().

Thanks,

Bishop Brock

[ranaroussi]

Hey!

I would love it if you can suggest a fix via a pull request :)

Thanks,
Ran

[ranaroussi]

?

[bcbrock]

Sorry Ran, this fell through the cracks. I'm travelling now but will propose a fix when I return in a week or 2.

[ranaroussi]

?

[bcbrock]

Hi Ran,

I hope this is better late than never. Unfortunately, I do not have permission from my employer to contribute to Open-Source projects. However, if you find the following helpful you are welcome to do with it as you see fit.

Best Wishes,

Bishop Brock

diff --git a/multitasking/__init__.py b/multitasking/__init__.py
index 1d83583..041d3c9 100644
--- a/multitasking/__init__.py
+++ b/multitasking/__init__.py
@@ -24,8 +24,10 @@ __version__ = "0.0.9"
 from sys import exit as sysexit
 from os import _exit as osexit
 
-from threading import Thread, Semaphore
-from multiprocessing import Process, cpu_count
+from threading import Thread, Semaphore as tSemaphore
+from multiprocessing import Process, cpu_count, Semaphore as pSemaphore
+
+semaphore = {"process": pSemaphore, "thread": tSemaphore}
 
 config = {
     "CPU_CORES": cpu_count(),
@@ -84,7 +86,7 @@ def createPool(name="main", threads=None, engine=None):
     config["ENGINE"] = engine
 
     config["POOLS"][config["POOL_NAME"]] = {
-        "pool": Semaphore(threads) if threads > 0 else None,
+        "pool": semaphore[engine](threads) if threads > 0 else None,
         "engine": Process if "process" in engine.lower() else Thread,
         "name": name,
         "threads": threads