concurrent.futures backend

[RealOrangeOne]

Python already has a form of concurrent tasks queue built in: concurrent.futures. Perhaps a simple backend based on this would be useful? It would likely be limiting, but for incredibly small deployments it would provide an intermediary between ImmediateBackend and DatabaseBackend.

[adamchainz]

I would rather we don’t offer such an option, since it will not provide any reliability in the case of process crashes. Let’s make the database backend the preferred/promoted path.

[RealOrangeOne]

I generally agree, the database backend is significantly more useful. I mostly opened this to collect the idea, and wonder whether it's a useful concept for any kind of workload (The answer could absolutely be "no").

[stuartmaxwell]

I'm not sure of the technical merits of this, but I'd love the idea of a "ThreadBackend". My use-case is the simple (and common) case of sending an email in the background. I don't need to wait for the result, and sending emails is inherently unreliable anyway. Changing my app to need an additional worker process and database tables, feels like unnecessary complexity.

I did manage to achieve what I wanted by using an async view that calls an async function that sends the email using httpx and a REST API. But this also introduces complexities with needing to change the webserver and potentially breaking whitenoise and other stuff.

[stuartmaxwell]

In my searching for a solution, I came across this: https://github.com/morlandi/django-task

That appeared to offer what I wanted, but hasn't been updated in a while, lacked documentation, and I couldn't get it working.

I'm not qualified to comment on the technical merits of these solutions. I'm just after a way to send an email in the background without having to run separate worker processes.

[cantin-f]

You can check that :
https://github.com/peter-wangxu/persist-queue

[stuartmaxwell]

That project seems to be about storing tasks to disk for later processing. The discussion here is about executing tasks immediately without storing them, but doing so in a different thread to avoid blocking.

[cantin-f]

Sorry I forgot the important part 👎
I used https://github.com/peter-wangxu/persist-queue to store queue without have to use Celery or other message broker, and tasks were executed with https://github.com/aleksve/ischedule.
No broker, no worker processes.