How to remove messages from target cluster in blue/green deployment when message is ack in source cluster?

[prajyot7878]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

26.1.x

Operating system (distribution) used

RPM/DEB

How is RabbitMQ deployed?

Generic binary package

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

# PASTE OUTPUT HERE, BETWEEN BACKTICKS

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

Donwload tar.gz on each node and run and form cluster

Steps to reproduce the behavior in question

enable rabbitmq_federation plugin, publish and ack message

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

I am currently running a blue/green deployment with RabbitMQ, consisting of 3 nodes in each cluster. I have set up federation between both clusters using the rabbitmq_federation plugin.

My requirement is to remove a message from the target cluster once it has been acknowledged in the source cluster. I realize that the rabbitmq_federation plugin is not designed to handle this, as it creates a separate copy of the message and sends it to the target cluster, resulting in two independent messages.

I am aware that setting up a message TTL on the target cluster can control this, but I am specifically looking for a solution that allows the removal of messages from the target cluster once they are acknowledged in the source cluster, without relying on message TTL.

Is there any alternative approach or plugin that can help achieve this? Or any recommended best practices to fulfill this requirement?

Thank you for your help!

[MirahImage]

The plugin you're looking for is shovel https://www.rabbitmq.com/docs/shovel
The specific usage in the case of blue/green deployment is covered in the blue/green deployment guide https://www.rabbitmq.com/docs/blue-green-upgrade#drain-messages

[prajyot7878]

This will remove message from Cluster a once publish to Cluster B in that case my application consumer will not have message to process.

I am looking like If on Cluster A application consumer gets the message, It ACKs it on Cluster A which will remove message from Cluster A and the same time I want to remove it from Cluster B so that Cluster B should not have old unwanted backlog and eventually application consumer will not process duplicate message.

Here's the desired behavior I am looking for:

1. Application consumer retrieves and processes a message from Cluster A.
2. Application consumer acknowledges the message in Cluster A, which removes the message from Cluster A.
3. Simultaneously, the same message is removed from Cluster B, preventing any buildup and avoiding duplicate processing

[michaelklishin]

Queue federation will gradually move messages as the Blue-Green Deployment doc guide plus a blog post on the topic from 2025 explain.

You can use shovels and create them programmatically, then delete them programmatically (or decommission the entire original cluster) but Blue-Green Deployments usually rely on queue federation.

[prajyot7878]

Here's the desired behavior I am looking for:

1. Application consumer retrieves and processes a message from Cluster A.
2. Application consumer acknowledges the message in Cluster A, which removes the message from Cluster A.
3. Simultaneously, the same message is removed from Cluster B, preventing any buildup and avoiding duplicate processing

So are you suggesting that, we have to mange duplicate message handling on application side so it will not process duplicate message?

[michaelklishin]

@prajyot7878 there is no such thing as cross-cluster acknowledgements, including at the protocol level (you can only acknowledge a delivery on your current connection/session/channel), and I don't see why they would be necessary given what queue federation links already do.

Read the Blue-Green deployment guide, it's a practice that's been widely used since 2018 or so.

[michaelklishin]

Queue federation moves messages from the upstream cluster to the local one, it does not replicate messages (like exchange federation does).

If you follow the Blue-Green deployment guide and do not deploy publishers to both clusters in parallel, there should be no duplicates but as a general rule, your consumers should be ready to handle them even then.