TRFC-2: Multiqueue support

[s3rius]

Abstract

Currently our brokers only leverage one queue and all tasks are sent in a single queue. The idea is to make it possible to assign different queues to tasks and being able to send tasks and read from multiple queues.

Solution

Currently there's a PR #418 that adds ability to add new methods to broker.task. We can make use of this feature in order to achieve the following API:

@broker.task.queue(...)
async def task(): ...

The question is how to specify queues, because queue API is different to all brokers.

One of possible solutions is to add function extra_queue method to broker, that returns a generic ExtraQueue object which can be used by the same broker to change task's destination.

Here's a usage example:

second_queue = broker.extra_queue(...)

# And then we can use these objects to assign tasks to these queues.

@broker.task.queue(second_queue)
async def task() :..

Questions

• Define ExtraQueue class. (or return strings)
• How to declare queues that brokers need to read from?
• Shall we split extra_queue to extra_read_queue and extra_write_queue