Kafka cluster

[ebilbil]

Hi guys,
we plan to move from Petrosian to Spasky shortly and I wonder about Kafka architecture.
We currently have a distributed setup (2x UI/webservice, 2x Providers inventory and an external replicated database server)
Is it possible with Miq appliance to have a kafka cluster with multiple brokers in order to avoid having a spof?

[lMysticl]

Yes. Kafka HA is supported as long as every ManageIQ appliance in the region connects to the same logical Kafka cluster. The limit is one cluster per region, not one broker. Pointing appliances in the same region at different clusters can split event producers and consumers.

Multiple broker addresses can be configured in config/messaging.yml:

base: &base
  hosts:
    - kafka1
    - kafka2
  port: 9092

manageiq-messaging converts hosts into Kafka's bootstrap.servers. If the brokers use different ports, omit port and include it in each host entry instead.

The remaining limitation is appliance_console, which currently prompts for a single Message Server Hostname; the underlying client configuration already supports multiple hosts.

References: the maintainer clarification, manageiq-messaging client, and appliance console configuration.

[agrare]

The limitation should be 1 logical kafka cluster per region, not 1 broker/server. You absolutely can configure kafka HA, what you can't do is have two worker appliances in the same region pointing to two different kafka clusters because that will wreak havoc with e.g. event processing (worker 1 listening to kafka cluster 1, worker 2 pushing events to kafka cluster 2).

The manageiq-messaging client does accept a hosts array, and you can configure the config/messaging.yml with a hosts key instead of a singular host. Appliance console

Just as a quick test I threw this together in my devel config/messaging.yml

---
base: &base
  hosts:
  - kafka1
  - kafka2
  port: 9092
  protocol: Kafka
  encoding: json
  username: admin
  password: smartvm

(NOTE if the port is not the same for all hosts you are expected to omit the port option and provide the port in each of the hosts entries like kafak1:9092, kafak2:9093)

And you can see it is being used by bootstrap.servers:

vmdb(dev)> MiqQueue.messaging_client("test")
=> 
#<ManageIQ::Messaging::Kafka::Client:0x00007f069cbad2f0
 @encoding="json",
 @kafka_client=
  #<Rdkafka::Config:0x00007f069cbacb70
   @config_hash=
    {:"bootstrap.servers"=>"kafka1:9092,kafka2:9092",
     :"client.id"=>"test",
     :"sasl.mechanism"=>"PLAIN",
     :"sasl.username"=>"admin",
     :"sasl.password"=>"smartvm",
     :"security.protocol"=>"PLAINTEXT"},
   @consumer_poll_set=true,
   @consumer_rebalance_listener=nil>>

We should definitely fix the appliance_console prompt to allow for multiple hosts.

If you run into any issues while testing this out please open bugs (https://github.com/ManageIQ/manageiq/issues/new) as Kafka HA is expected to work.

[lMysticl]

Thanks for the correction. I read “one Kafka broker per region” too literally; the actual constraint is one logical Kafka cluster per region. The hosts example confirms the supported HA configuration. I’ve corrected my answer.

[agrare]

We definitely need to improve the documentation in this area so thank you for pointing this out!

[Fryguy]

Merged ManageIQ/manageiq-documentation#1911 to clarify it's 1 cluster not 1 broker.