Khepri - Auto delete queues after a network partition are stuck / down

[luos]

Describe the bug

Hi,

We were testing out Khepri and noticed that some auto-delete durable queues can become unavailable if there is a network partition.

Reproduction steps

1. Create a 3 node cluster, 4.1.3, with Khepri enabled
2. Create an auto-delete, (transient or durable) queue with a consumer on NODE-1
3. Create a fully partitioned NODE-1
4. Observe the queue going to the ‘down’ on NODE-2 and NODE-3
5. Consumer gets disconnected from NODE-1
6. Restore the network for NODE-1
7. Queue shows “running” state but cannot function. It can be deleted manually.
8. On re-declaration, the operation fails with a not_found exception:

2025-09-11 08:40:24.537518+00:00 [error] <0.2066.0> Channel error on connection <0.2057.0> (172.31.31.98:18500 -> 172.31.22.178:5672, vhost: '/', user: 'testuser'), channel 1:
2025-09-11 08:40:24.537518+00:00 [error] <0.2066.0> operation queue.declare caused a channel exception not_found: failed to perform operation on queue 'cq-1' in vhost '/' due to timeout
2025-09-11 08:40:24.869547+00:00 [warning] <0.2057.0> closing AMQP connection <0.2057.0> (172.31.31.98:18500 -> 172.31.22.178:5672 - perf-test-configuration-0, vhost: '/', user: 'testuser', duration: '1M, 2s'):

Expected behavior

Auto delete queue is deleted on majority partition because it lost the consumer.

Additional context

[michaelklishin]

@luos I don't have any logs to work with but very likely this comes down to the fact that

1. Node 1 is in the minority, so it won't be able to perform any schema database updates
2. Queue recovery entirely depends on the data it gets from the schema database

So this is not an issue with the implementation, this is a fundamental incompatibility of non-replicated durable queues and Khepri's fundamental assumptions that stem from Raft (a node in a minority cannot perform any writes).

You are welcome to investigate this further to have a more detailed description of what's going on.

[michaelklishin]

@luos @dumbbell suggests that we have seen this before with exclusive CQs, and it is pending an investigation but currently there are higher priority Khepri improvements.

[dumbbell]

The Khepri improvements being worked on are to address the issue with exclusive queues. I don't know if this will fix anything with auto-delete (unlikely).

[kjnilsson]

I would expect the auto delete queue to stay running until it is able to perform the meta data store update to delete itself.

[kjnilsson]

Quorum queues have a similar issue when a delete happens on a khepri minority

rabbitmq-server/deps/rabbit/src/rabbit_quorum_queue.erl

Lines 885 to 930 in 403e5aa

	case ra:delete_cluster(Servers, Timeout) of
	{ok, {_, LeaderNode} = Leader} ->
	MRef = erlang:monitor(process, Leader),
	receive
	{'DOWN', MRef, process, _, _} ->
	%% leader is down,
	%% force delete remaining members
	ok = force_delete_queue(lists:delete(Leader, Servers)),
	ok
	after Timeout ->
	erlang:demonitor(MRef, [flush]),
	ok = force_delete_queue(Servers)
	end,
	notify_decorators(QName, shutdown),
	case delete_queue_data(Q, ActingUser) of
	ok ->
	_ = erpc_call(LeaderNode, rabbit_core_metrics, queue_deleted, [QName],
	?RPC_TIMEOUT),
	{ok, ReadyMsgs};
	{error, timeout} = Err ->
	Err
	end;
	{error, {no_more_servers_to_try, Errs}} ->
	case lists:all(fun({{error, noproc}, _}) -> true;
	(_) -> false
	end, Errs) of
	true ->
	%% If all ra nodes were already down, the delete
	%% has succeed
	ok;
	false ->
	%% attempt forced deletion of all servers
	?LOG_WARNING(
	"Could not delete quorum '~ts', not enough nodes "
	" online to reach a quorum: ~255p."
	" Attempting force delete.",
	[rabbit_misc:rs(QName), Errs]),
	ok = force_delete_queue(Servers),
	notify_decorators(QName, shutdown)
	end,
	case delete_queue_data(Q, ActingUser) of
	ok ->
	{ok, ReadyMsgs};
	{error, timeout} = Err ->
	Err
	end

[kjnilsson]

We could perhaps reduce some of these by making the meta data store update for deletes having two phase:

1. Update queue record with deleting status
2. Perform actual delete
3. Remove queue record

If 1 fails we don't proceed with the rest or enter a retry loop. Ofc partitions can happen between 1 and 3 but it is much less likely to occur.

[luos]

Hi,

Thanks for reviewing this.

I think this may be a problem in case let's say the node hosting the AD queue simply crashes as well - in which cases the live process never comes back so it can not delete itself. (I haven't tested this, maybe there is some cleanup on restart). I'd also "guess" that it's relatively common to have a partition and a node crash at the same time, which means the process is no longer alive.

I am not sure what would be a good solution, I agree it's similar but different to the exclusive queue case.

Maybe something when the node reconnects to the cluster could clean up ad/transient queues which are no longer running?

Probably it's fine to leave the queue record during the partition as it could be still be running on the other side, so somehow the reconnecting node could be responsible for the cleanup.