Async workers

[hille721]

I have a tasks where I need to do a sleep. As I saw that async tasks are supported I created one and using asyncio.sleep. With that I thought this will then not block the queue.
But in fact when I just tested, it turns out it will. If one task is running, no other tasks will be started. I checked now that the worker itself (the db_worker) is sync and calling the tasks with call.
Wouldn't it be possible to have an async worker which is then calling the tasks with acall?

[RealOrangeOne]

This would be great, but I don't think it's especially simple. For now, keeping sync and async function execution the same makes things more dependable. Similarly, it potentially changes the resource requirements for a worker, as it may now be running multiple tasks concurrently.

[emuwakwe]

hi guys - i saw acall functionality in the Task object but the db_worker uses .call() for all objects.

What's the best way to impement an async worker? right now I have a task that takes 10 minutes to complete and it looks like db_worker executes synchronously through the queue so my tasks are just waiting on a response for a long time when they could also start the other tasks in the queue

[RealOrangeOne]

It currently isn't possible. The DB worker process will only execute a single task at once. async support is more about being able to use async frameworks and tools than it is about increasing concurrency of tasks.

It'd be great to have this eventually, but it needs some serious thought.

[emuwakwe]

Thanks for the comment, very helpful.

My issue right now is each task is just an I/O operation (API call to my GCP cloud function) that takes 15m to run so in your opinion, the only way to get around the massive cue that forms is to spin up more worker processes?

[hille721]

@emuwakwe you could also put it together in one task, as inside the task you can perform async operations

[emuwakwe]

Yeah that's what I did actually. I created a separate queue and added several tasks then another endpoint cron job to run every 30secs that batches em up. Thanks for your help!

…

On Sat, Jan 18, 2025 at 1:13 AM Christoph Hille ***@***.***> wrote: @emuwakwe <https://github.com/emuwakwe> you could also put it together in one task, as inside the task you can perform async operations — Reply to this email directly, view it on GitHub <#117 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AK7R44OFCHST46L5HWOLKKD2LILMRAVCNFSM6AAAAABRGF6GMOVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBXGQYTEMA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>