Cluster rebalancing improvements - UI issue

[joekent123]

Background

We want to make improvements to the Console for indicating the 'balance' state of a cluster as well as showing the before/after impact of kafkaRebalance CRs via Cruise Control.

Phases

1. Improve the current Kafka nodes page and the Rebalance tab on that page. Add charts showing the state of the cluster to indicate imbalance.
2. Add a new page for kafkaRebalance CRs which show a before/after impact of the proposal in the CR.

Design

Link to prototype: https://www.figma.com/proto/eLMaxO8o4hzYossIBq8fhe/Cluster-rebalancing?node-id=451-9572&p=f&viewport=-2687%2C326%2C0.38&t=6wzkAY6zh1JocnpW-1&scaling=min-zoom&content-scaling=fixed&starting-point-node-id=451%3A9572&page-id=451%3A9571
Password: uncut-comma-kindle-pasta

Link to design page in Figma: https://www.figma.com/design/eLMaxO8o4hzYossIBq8fhe/Cluster-rebalancing?node-id=451-9571

[MikeEdgar]

@joekent123 @tomncooper @kyguy , after reviewing a test rebalance scenario in my cluster, I have a few concerns about data availability for the proposal and progress information.

1. Strimzi does not request from Cruise Control the verbose executor state information [1], so the ConfigMap will never contain partition information as shown in the design.
   

2. The executor state JSON is only set in the ConfigMap when CC reports that more than zero bytes have already been moved [2]. In cases where partitions will be relocated, this value never increases, so the JSON is never available. As noted above, it would not contain partition information regardless.

3. The proposal information in Strimzi does not seem to be refreshing consistently. I have not debugged it whatsoever yet, but only after deleting the ConfigMap does the information seem to be reloaded. Multiple attempts to annotate the CR with refresh do not seem to update the proposal information.

I have ideas on how we can propose improvements in Strimzi to enhance this, which will take some time to be accepted and released, of course. We should discuss further, but given what I have seen, I suggest we limit the initial change in the Console to be the Node listing and the clean-up on the Rebalance listing. My view at the moment is that the CC proposal and execution status published by Strimzi needs to be more robust before we build something on top of it.

[1] https://github.com/strimzi/strimzi-kafka-operator/blob/2dc770410d518ce163ca1410bbabe6c50986d37b/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaRebalanceConfigMapUtils.java#L128
[2] https://github.com/strimzi/strimzi-kafka-operator/blob/2dc770410d518ce163ca1410bbabe6c50986d37b/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaRebalanceConfigMapUtils.java#L133