set up ClickHouse keeper cross AKS cluster

[chaos827]

I followed this article(https://www.farukcevik.com.tr/blog/2024/02/07/clickhousedisasterrecovery/) to set up ClickHouse in two data centers, for the ClickHouse instance, I used external Load Balancer ip to set up remote servers and it works well, so I followed the same way to set up ClickHouse Keeper, but seems the only 3 prod keeper instances are active, the bcp instances are not health.

this may config for the primary
<keeper_server>
<server_id>1</server_id>
<raft_configuration>
1hostname19444
2hostname29444
3hostname39444
4external load balancer ip for bcp19444
5external load balancer ip for bcp29444
6external load balancer ip for bcp39444
</raft_configuration>
</keeper_server>

for the bcp:
<keeper_server>
<server_id>1</server_id>
<raft_configuration>
1external load balancer ip for prod19444
2external load balancer ip for prod29444
3external load balancer ip for prod39444
4hostname19444
5hostname29444
6hostname39444
</raft_configuration>
</keeper_server>

I used clickhouse-keeper-client to show the config info, the prod keeper, shows there are 6 servers, and in the BCP, the clickhouse-keeper-client throws error:
Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: 10.200.0.10:2181
Code: 999. Coordination::Exception: Keeper server rejected the connection during the handshake. Possibly it's overloaded, doesn't see leader or stale: while receiving handshake from ZooKeeper. (KEEPER_EXCEPTION) (version 24.5.1.1763 (official build)), 10.200.0.10:2181

I do not figure out why the clickhkouse-keeper-client throws this error, and if I set the bcp clickhouse to only connect to the bcp keeper, the bcp clickhouse data is uncoordinated, does someone know how to set up keeper in two AKS?
Many thanks.

[alex-zaitsev]

Setting up Keeper and ZooKeeper across regions is not recommended due to latency that will affect inserts. We recommend to keep the quorum in one datacenter and have observers in another one.