Per-queue Raft metrics

[paguos]

Is your feature request related to a problem? Please describe.

I would like to have queue base metrics to see the raft status of each Quorum Queue.

Describe the solution you'd like

Add additional metrics to monitor the status of the quorum queues. The main need for this is to be able to create alerts when a queue's quorum is under a given limit.

Describe alternatives you've considered

I try to get that information from existing metrics:

curl localhost:15692/metrics | grep raft
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 98950  100 98950    0     0   160k      0 --:--:-- --:--:-- --:--:--  160k
# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
rabbitmq_raft_term_total 1808
# TYPE rabbitmq_raft_log_snapshot_index gauge
# HELP rabbitmq_raft_log_snapshot_index Raft log snapshot index
rabbitmq_raft_log_snapshot_index 57764232
# TYPE rabbitmq_raft_log_last_applied_index gauge
# HELP rabbitmq_raft_log_last_applied_index Raft log last applied index
rabbitmq_raft_log_last_applied_index 57780704
# TYPE rabbitmq_raft_log_commit_index gauge
# HELP rabbitmq_raft_log_commit_index Raft log commit index
rabbitmq_raft_log_commit_index 57780704
# TYPE rabbitmq_raft_log_last_written_index gauge
# HELP rabbitmq_raft_log_last_written_index Raft log last written index
rabbitmq_raft_log_last_written_index 57780704
# TYPE rabbitmq_raft_entry_commit_latency_seconds gauge
# HELP rabbitmq_raft_entry_commit_latency_seconds Time taken for a log entry to be committed
rabbitmq_raft_entry_commit_latency_seconds 0.0023636363636363638

But none of them have any label that can be used for getting the quorum of an individual label.

Additional context

No response

[michaelklishin]

rabbitmq-queues quorum_status

already provides what can be provided per-queue.

There are already over 60 or so metrics per queue, even after a few of them were removed in 4.0.x.
This can and in practice will create significant cardinality issues when visualizing them, which is
why for years now the Prometheus plugin has had two operating modes and scraping endpoints. You are using the aggregated one.

[paguos]

That explains a lot. Thank you 👍🏼

[michaelklishin]

@paguos this should have been a discussion. You are likely unaware of this section of the Prometheus plugin documentation.

Plus there are #12148 and #13107, so, evidence of new Raft metrics being added over time for QQs and other Raft-based components such as Khepri.

[paguos]

Thanks for the feedback. Agree this should be a discussion. I was not aware of that feature. Thanks for the help!

[paguos]

@michaelklishin could you help pointing out the metrics I need to use to have the following information in a grafana chart?

I'm interested in the number of members of a quorum for each queue.

[paguos]

Same, is very odd. I see that metric from the other instances, but without the queue/vhost label in it.

[mkuratczyk]

This metric with no labels is returned if you don't use per-object metrics:

> curl -s localhost:15692/metrics/per-object | rg ^rabbitmq_raft_log_last_applied_index
rabbitmq_raft_log_last_applied_index{vhost="/",queue="qq"} 54

> curl -s localhost:15694/metrics/ | rg ^rabbitmq_raft_log_last_applied_index
rabbitmq_raft_log_last_applied_index 96

This metric with no labels is pretty much useless by the way. It's only there because we have some higher level aggregation performed from per-object metrics to aggregated metrics, so it ends up as an aggregated metric (sum). Probably what you are seeing.

We'll be working on improving the RAFT metrics very soon. This pointless aggregation will disappear as part of that work.

[paguos]

I checked the metric for every pod:

Server 1:

> curl http://localhost:30001/metrics/per-object | grep rabbitmq_raft_log_last_applied_index
...
rabbitmq_raft_log_last_applied_index{vhost="dev",queue="transcribe"} 173
rabbitmq_raft_log_last_applied_index{vhost="test",queue="analysis"} 9826
rabbitmq_raft_log_last_applied_index{raft_cluster="rabbitmq_metadata"} 28

Server 2 & 3:

> curl http://localhost:30002/metrics/per-object | grep rabbitmq_raft_log_last_applied_index
...
rabbitmq_raft_log_last_applied_index{raft_cluster="transcribe"} 12155
rabbitmq_raft_log_last_applied_index{raft_cluster="analysis"} 1873
rabbitmq_raft_log_last_applied_index{raft_cluster="rabbitmq_metadata"} 32

Any idea how are we getting this different metrics? We are running v4.0.3

[mkuratczyk]

somehow this check must be failing and the raft_cluster fallback is used: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_prometheus/src/collectors/prometheus_rabbitmq_core_metrics_collector.erl#L740-L752

But I don't see why this could happen.

The upcoming changes I mentioned will get rid of this code block altogether, since the Ra metrics will come from a different place (we currently have to methods of tracking Ra metrics and this is the old one which we'll stop using). While this looks like a bug, I can't reproduce it right now and need to focus on other things.

[paguos]

Thanks for the help @mkuratczyk. Please let us know when you are done with the ra metrics 👍🏼