[Questions] Rabbit HA Cluster: shall classic durable queues be avoided?

[YuryHrytsuk]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.1.2

Erlang version used

27.3.x

Operating system (distribution) used

Ubuntu, Docker Swarm (StackS)

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

...

Logs from node 1 (with sensitive values edited out)

...

Logs from node 2 (if applicable, with sensitive values edited out)

...

Logs from node 3 (if applicable, with sensitive values edited out)

...

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-configuring-mechanism
cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.1 = rabbit@rabbit-node01_rabbit01
cluster_formation.classic_config.nodes.2 = rabbit@rabbit-node02_rabbit02
cluster_formation.classic_config.nodes.3 = rabbit@rabbit-node03_rabbit03

# explicitly set cluster name so that all nodes share the same name
# see https://github.com/rabbitmq/rabbitmq-server/discussions/11355
# and https://groups.google.com/g/rabbitmq-users/c/A8v8XSmnCnA/m/4ZpoTtEWAQAJ
# important for grafana dashboard to work properly
cluster_name=osparc.local

## Sets the initial quorum queue replica count for newly declared quorum queues.
## This value can be overridden using the 'x-quorum-initial-group-size' queue argument
## at declaration time.
# https://www.rabbitmq.com/docs/quorum-queues#quorum-requirements
quorum_queue.initial_cluster_size = 3

# Extract proper client ip when behind a proxy (e.g. haproxy)
# https://www.rabbitmq.com/docs/networking#proxy-protocol
# WARNING: this forces clients to use a proxy (direct access to nodes does not work)
proxy_protocol = true

default_queue_type = quorum

Steps to deploy RabbitMQ cluster

1. Clone https://github.com/ITISFoundation/osparc-ops-environments/
2. Go to ./services/rabbit
3. Add env file
4. Run make start-cluster

Steps to reproduce the behavior in question

1. Create a 3-node rabbitmq cluster
2. Start an application that uses and relies on durable classic queue
3. Stop RabbitMQ Node which hosts durable classic queue mentioned above
4. See application going down because queue is not accessible

advanced.config

No response

Application code

No response

Kubernetes deployment file

No response

What problem are you trying to solve?

I want to have a highly available RabbitMQ setup which is robust to 1 node going down (we have 3 in total). By robust I mean that my applications that rely on rabbitmq and communicate over it, keep running even when 1 rabbitmq node goes down.

However, I noticed that my application gets stuck if 1 RabbitMQ node goes down. I presume the culprits are classic durable queues that are stuck to that node.

Shall one avoid durable classic queues if high availability is the goal (aka being robust to 1 node going down in 3-nodes cluster setup)?

[MirahImage]

If your goal is high availability, you should be using quorum queues (or streams). Those are our highly available types.

[YuryHrytsuk]

If data-safety is not priority, it is fine to have transient classic queue and lose data when rabbitmq node goes down (but at least queue is available if there are other rabbitmq nodes running)

The problem I want to highlight and verify I understand it correctly is:

Using durable classic queues has a fundamental flaw: if rabbitmq node goes down, the durable classic queue is no longer available (even though other rabbitmq nodes of the cluster are running happily) and all workloads relying on it are stuck until rabbitmq node is back. This can be solved by using quorum queues or making classic queue transient (non-durable)

[mkuratczyk]

If data safety is not important but availability is (a common RPC scenario for example) then using exclusive queues is often a good option. Quite likely, you will also like the local-random-exchange. Take a look at the scenario described here: https://www.rabbitmq.com/docs/local-random-exchange

[YuryHrytsuk]

Here is the answer I came to:

Classical durable queues in HA setup should be avoided (starting from 4.0 version).

Reason: if node goes down, all classical durable queues that lived on the node are not available anymore and your rabbitmq clients are stuck until node is back (this is relevant from version 4.0 where classical queues cannot be mirrored anymore)

[michaelklishin]

@YuryHrytsuk I mean, we've only had the Classic Mirrored Queues doc guide flashing explicit warnings in red for a few years.

Not only they "should be avoided" in 4.x where the feature does not exist at all (the CQ mirroring part, classic queues do exist as a non-replicated queue type), they should be avoided, period. Quorum queues have been around since 2018.

In most scenarios classic mirrored queues work exactly as expected. The problem is that in some scenarios they don't, and their distributed algorithm is unfixable for those cases. But your explanation is a significant oversimplification.