Add time-based consumer group lag and indication of lag trend

[joekent123]

Adding time-based (rather than offset-based) consumer group lag was one of the most common priority items coming out of the workshop on making the Console more suitable for app developers. Time based lag (e.g. this consumer group is 30 seconds behind the most recent offset in this partition) is a common metrics we see in competitor products.

In addition to adding time-based lag we should also consider adding a chart or indication if the lag is growing, shrinking or maintaining over time as this is a useful indicator to an app developer of consumer group health.

---

Below is a work item summary generated in Bob:

1 — Consumer Lag Trending & Time-Based Lag Display

Priority: 🔴 High | Scope: M

Workshop stories addressed

• "An app developer needs to be able to see how far behind in time their app is — CG lag metrics should show time lag as well as absolute message lag."
• "An app developer can see consumer groups that are lagging behind."
• "An app developer can see that a consumer is lagging in either time or number of messages."
• "An app developer can identify trends before they become a problem."
• "An app developer can view performance trends in their Kafka cluster so that they can identify potential issues."

Current state

The console shows point-in-time consumer group lag (message count and current/end offsets). There is no lag history, no trend chart, and no time-based lag calculation (i.e. how many seconds/minutes behind the latest message the consumer currently is).

What needs building

1. Time-lag column alongside message-lag in the consumer groups list and detail view. Convert message offset delta to a time estimate using the topic's message.timestamp.type and last-seen message timestamps.
2. Lag trend chart on the consumer group detail page — a time-series chart (backed by Prometheus) showing lag over the last N minutes/hours, with a configurable time window selector.
3. Lag history in the topic consumer-groups sub-tab — summary sparklines or trend indicators (up/stable/down) alongside current lag numbers.

Competitive context

• Confluent Control Center — lag trending is a core feature; considered table stakes.
• Conduktor — rich consumer lag charts with time-range selection.
• AKHQ — basic lag display; no native trending.
• Redpanda Console — consumer group lag visible; trending limited.

Gap severity: Critical. Lag is the primary health signal for app developers. Without trend data, developers cannot distinguish a recovering lag from a worsening one at a glance.

[tomncooper]

Thanks for highlighting this @joekent123. I have been thinking about it since the workshop you ran. First, I want to provide a bit of context on what Kafka stores with regard to temporal offset information and then suggest some approaches.

What Kafka stores

Each record carries a single timestamp, either set by the producer (CREATE_TIME, the default) or overwritten by the broker at append time (LOG_APPEND_TIME). You can't have both today, though (the under discussion since 2022) KIP-870 proposes dual-timestamp tracking.

Within a log segment, Kafka maintains a sparse time index (.timeindex) that maps timestamps to offsets. This backs the offsetsForTimes() API, which is the only native way to go from a point in time to a position in the log.

When a consumer commits an offset, the broker records a commitTimestampMs internally, but this is never exposed back to consumers. The OffsetFetch API returns only the offset, leader epoch, and metadata string.

So currently there is no native API to get the timestamp of the record at a committed offset. You have to fetch the record yourself.

Possible approaches to temporal lag

• Approach 1: read the record at the committed offset and compare its timestamp to now (or to the record at log-end)
  • This tells you how stale the consumer's view is
  • It is backward-looking metric: "how far behind is this consumer group right now?"
  • Useful for SLA monitoring
  • Requires the relevant permissions to read the topics and consumer groups you want to monitor
  • Introduces additional consumer load to the cluster
• Approach 2: sample the committed offset (for each partition) at two points in time to get the arrival rate ( λ), then apply Little's Law (W = L / λ) where L is the current offset lag
  • This tells you how long (on average) a record will sit in the backlog before being consumed
  • It is a forward-looking metric: "how long on average will new records wait to be consumed?"
  • This can be useful for scaling decisions
  • It doesn't require access to the underlying topics and consumer groups, just the information from the Admin API
  • However, this would just be an estimate. It makes strong assumptions about the arrival rate into partitions (that they remain uniform).

Where they diverge

• Bursty production (ten thousand records land in one second, then silence)
  • Approach 1 shows a small time gap since the records all have similar timestamps
  • Approach 2 shows a high wait time since the queue depth is large relative to consumption rate
• Consumer restart after hours of downtime
  • Approach 1 shows a large time gap
  • Approach 2 shows a small wait time since the consumer is catching up at maximum throughput
• Producer clock skew with CREATE_TIME (the producers are adding timestamps well off wall clock time)
  • Approach 1 can give misleading results since the timestamp at the committed offset could be ahead of the log end
  • Approach 2 is unaffected since it never reads record timestamps, only offset arithmetic

In steady state with uniform production, the two approaches converge. They split apart when production is bursty, consumption rate changes suddenly, or timestamps are unreliable. For approach 2, you would have a look-back window over which you would estimate the arrival rate, which would allow you to tune how much variation you capture in the calculation of the arrival rate, which is another dimension to manage.

Approach 1 is more invasive and complex to administer, approach 2 is an estimate but could be done by adding functionality to existing tools like the kafka_exporter. I actually created a PoC branch for adding approach 2 metrics to the kafka_exporter. However, that repo is not very well maintained and if we wanted this feature any time soon I think we would probably need to fork the project. Alternatively, we could create a new exporter (in Java - kafka_exporter is written in Go) project which contains one or both of these metrics and is better maintained.

[joekent123]

On the assumption that whatever route we go with in your write up @tomncooper results in a single 'seconds' value of lag I've put together the following screens as an example of what we could show:

Add a 'lag trend' mini-chart in the consumer group table row. This would be an indication to a user that the trend is growing, shrinking or maintaining. If possible would be a nice feature if the chart could indicate a problematic trend in some way (in the image it's just the colour, which is not accessible so I would need to think more about this).

New summary info tile at the top of an individual group with the name, state, lag and trend. Also two new charts representing lag (offsets) and lag (seconds) over time. All partitions show by default.

Chart option to split lag across all partitions within the topic the consumer group is consuming from.

Expanded CG table row showing details of lag for all partitions.

[MikeEdgar]

@joekent123 these screens look good. I am still not sure how we will get the metrics, but assuming they are existing, it should be doable. One comment/question is - would it be better to move "members" off to a separate tab and have an "overview" tab with just the name/state/lag/trend and the metrics?

[joekent123]

Nice idea @MikeEdgar here's a few screens with that layout: