How can I configure stream replication connections to use IPv6 in an IPv6-only environment?

[adippl]

Describe the bug

I'm deploying kolla-ansible 2025.2 openstack in ipv6 only lab.
As a part of this deployment kolla ansible deploys rabbitmq in single node or cluster configuration.
Single node deployment works perfectly, clustered configuration doesn't work.

I don't have a lot of experience with rabbitmq but I see a lot of refused connection between nodes:
When I try to check on which ports rabbitmq listens I see this:

tcp   LISTEN 0      0                          0.0.0.0:6114       0.0.0.0:*    users:(("beam.smp",pid=25,fd=365))
tcp   LISTEN 0      0                          0.0.0.0:6054       0.0.0.0:*    users:(("beam.smp",pid=25,fd=58)) 
tcp   LISTEN 0      0                          0.0.0.0:6044       0.0.0.0:*    users:(("beam.smp",pid=25,fd=379))
tcp   LISTEN 0      0                          0.0.0.0:6045       0.0.0.0:*    users:(("beam.smp",pid=25,fd=315))
tcp   LISTEN 0      0                          0.0.0.0:6350       0.0.0.0:*    users:(("beam.smp",pid=25,fd=86)) 
tcp   LISTEN 0      0                          0.0.0.0:6266       0.0.0.0:*    users:(("beam.smp",pid=25,fd=403))
tcp   LISTEN 0      0                          0.0.0.0:6268       0.0.0.0:*    users:(("beam.smp",pid=25,fd=247))
tcp   LISTEN 0      4096                     127.0.0.1:4369       0.0.0.0:*    users:(("epmd",pid=108,fd=3))     
tcp   LISTEN 0      1024   [fd00:0:0:1097::1001]:15672         [::]:*    users:(("beam.smp",pid=25,fd=185))
tcp   LISTEN 0      4096   [fd00:0:0:1097::1001]:4369          [::]:*    users:(("epmd",pid=108,fd=5))     
tcp   LISTEN 0      128    [fd00:0:0:1097::1001]:5672          [::]:*    users:(("beam.smp",pid=25,fd=186))
tcp   LISTEN 0      4096                         [::1]:4369          [::]:*    users:(("epmd",pid=108,fd=4))     
tcp   LISTEN 0      128    [fd00:0:0:1097::1001]:25672         [::]:*    users:(("beam.smp",pid=25,fd=18)) 

If I understand this correctly, replication cannot take place since rabbitmq opens ipv4 ports on hosts without any ipv4 addresses. Listening ipv4 sockets are not reachable over ipv6.

Aside from that, rabbitmq cluster looks ok, nodes online are detected
Normal, non replication, connections to rabbitmq work, DNS names are resolvable.

docker exec -it rabbitmq rabbitmqctl cluster_status
Cluster status of node rabbit@os-node-b ...
Basics

Cluster name: rabbit@os-node-b
Total CPU cores available cluster-wide: 12

Cluster Tags

(none)

Disk Nodes

rabbit@os-node-a
rabbit@os-node-b
rabbit@os-node-c

Running Nodes

rabbit@os-node-a
rabbit@os-node-b
rabbit@os-node-c

Versions

rabbit@os-node-b: RabbitMQ 4.1.5 on Erlang 27.3.4.5
rabbit@os-node-a: RabbitMQ 4.1.5 on Erlang 27.3.4.5
rabbit@os-node-c: RabbitMQ 4.1.5 on Erlang 27.3.4.5

Config files:
rabbitmq.conf

# NOTE(yoctozepto): rabbitmq uses the raw format (e.g. fd::) of IPv6 address;
# despite specifying port via colon, the url format (e.g. [fd::]) is not accepted
listeners.tcp.1 = fd00:0:0:1097::1001:5672
cluster_partition_handling = pause_minority

management.listener.ip = fdc2:fd30:e110:1097::1001
management.listener.port = 15672
management.load_definitions = /etc/rabbitmq/definitions.json

cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
cluster_formation.classic_config.nodes.0 = rabbit@os-node-a
cluster_formation.classic_config.nodes.1 = rabbit@os-node-b
cluster_formation.classic_config.nodes.2 = rabbit@os-node-c

rabbitmq-env.conf

RABBITMQ_NODENAME=rabbit@os-node-a
RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq
RABBITMQ_DIST_PORT=25672
RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp "

export ERL_EPMD_ADDRESS=fd00:0:0:1097::1001
export ERL_EPMD_PORT=4369
export ERL_INETRC=/etc/rabbitmq/erl_inetrc

erl_inetrc

{inet6,true}.

I've read rabbitmq networking guide and it's ipv6 section. If I understood it correctly this config should be enough to get rabbitmq to working over ipv6.

I've also tried to get enviroment variables configured at the container lever instead of rabbitmq-env.conf files but there was no difference.

I would appreciate any feedback on this issue.

Reproduction steps

1. Install rabbitmq cluster on ipv6-only nodes

Expected behavior

Rabbitmq shouldn't bind to ipv4 while operating in ipv6 mode.

Additional context

No response

[adippl]

Example errors of rabbitmq failing to create inter-node connections

2025-11-23 21:27:09.021111+00:00 [warning] <0.3146.0> rabbit_stream_coordinator: Error while starting replica for __scheduler_fanout_1763933228204814727 on node rabbit@os-node-c in 1 : {{shutdown,{connection_refused,{child,undefined,#Ref<14727.1515118949.2499280897.96548>,{...},...}}},{gen_server,call,[<14727.2765.0>,await,infinity]}}
2025-11-23 21:27:09.022311+00:00 [warning] <0.3151.0> __scheduler_fanout_1763933228204814727 [osiris_replica_reader:init/1] could not connect replica reader to replica at [{127,0,0,1},"os-node-a"] port 6434, Reason: connection_refused
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>   crasher:
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     initial call: osiris_replica_reader:init/1
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     pid: <0.3151.0>
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     registered_name: []
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     exception exit: connection_refused
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>       in function  gen_server:init_it/6 (gen_server.erl, line 2210)
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     ancestors: [osiris_replica_reader_sup,osiris_sup,<0.131.0>]
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     message_queue_len: 0
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     messages: []
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     links: [<0.136.0>]
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     dictionary: []
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     trap_exit: true
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     status: running
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     heap_size: 2586
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     stack_size: 29
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>     reductions: 3975
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0>   neighbours:
2025-11-23 21:27:09.022431+00:00 [error] <0.3151.0> 
2025-11-23 21:27:09.023263+00:00 [warning] <0.3148.0> rabbit_stream_coordinator: Error while starting replica for __scheduler_fanout_1763933228204814727 on node rabbit@os-node-a in 1 : {{shutdown,{connection_refused,{child,undefined,#Ref<14726.2527381616.2231107585.146061>,{...},...}}},{gen_server,call,[<14726.3857.0>,await,infinity]}}
2025-11-23 21:27:09.034103+00:00 [warning] <0.3156.0> __scheduler_fanout_1763933228204814727 [osiris_replica_reader:init/1] could not connect replica reader to replica at [{127,0,0,1},"os-node-a"] port 6484, Reason: connection_refused
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>   crasher:
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     initial call: osiris_replica_reader:init/1
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     pid: <0.3156.0>
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     registered_name: []
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     exception exit: connection_refused
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>       in function  gen_server:init_it/6 (gen_server.erl, line 2210)
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     ancestors: [osiris_replica_reader_sup,osiris_sup,<0.131.0>]
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     message_queue_len: 0
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     messages: []
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     links: [<0.136.0>]
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     dictionary: []
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     trap_exit: true
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     status: running
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     heap_size: 2586
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     stack_size: 29
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>     reductions: 3975
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0>   neighbours:
2025-11-23 21:27:09.034226+00:00 [error] <0.3156.0> 

For some reason it tries to connect to remote nodes on ipv4 localhost

[michaelklishin]

@adippl IPv6 support for stream replication connections was introduced 9 months ago in rabbitmq/osiris#177 and has first shipped in RabbitMQ 4.1.0.

I'd need to see if we expose it in rabbitmq.conf but even if we don't, the setting can be controlled with advanced.config:

[
  {osiris, [
    {replica_ip_address_family, inet6}
  ]}
].

[michaelklishin]

There is no rabbitmq.conf equivalent yet but 4.1.0 release notes do mention the setting.

[adippl]

Thanks for the explanation.
Out of the box kolla deploys rabbitmq with advanced.config which looks like this:

[
  {kernel, [
    {inet_dist_use_interface, {16#fd00,16#0,16#0,16#1097,16#0,16#0,16#0,16#1001}},
    {inet_dist_listen_min, 25672},
    {inet_dist_listen_max, 25672}
  ]}
].

Replacing it with your example resolves the issue and allows rabbitmq to work correctly.

I've modified original config to add osiris section and it also works:

[
  {kernel, [
    {inet_dist_use_interface, {16#fd00,16#0,16#0,16#1097,16#0,16#0,16#0,16#1001}},
    {inet_dist_listen_min, 25672},
    {inet_dist_listen_max, 25672}
  ]},
  {osiris, [
    {replica_ip_address_family, inet6}
  ]}
].

Is there documentation for this syntax outside of release notes?
It's not in https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/docs/advanced.config.example

[michaelklishin]

@adippl the advanced.config file is an Erlang term file.

So you need to know the application name (in this case, osiris, a core of the streams implementation) and the key path. Inspecting effective node configuration will reveal a lot of configurable things that, however, for the most part either can be configured via rabbitmq.conf or do not need to be configured by almost no one.

[michaelklishin]

And yes, you have combined the osiris.replica_ip_address_family correctly with the existing configuration, it should be on the top level next to kernel (or rabbit).

So, you should be ready to go for now. We will expose this setting to rabbitmq.conf very soon.

[adippl]

Got my rabbitmq cluter working.
Thank you for the explanation.

[michaelklishin]

#15011 exposes the above setting to rabbitmq.conf.

Thank you for bringing this topic up, @adippl.