Skip to content

[improve][broker] Add consumer_address label to per-consumer Prometheus metrics#26102

Open
Dream95 wants to merge 2 commits into
apache:masterfrom
Dream95:improve_prometheus-consumer-address-label
Open

[improve][broker] Add consumer_address label to per-consumer Prometheus metrics#26102
Dream95 wants to merge 2 commits into
apache:masterfrom
Dream95:improve_prometheus-consumer-address-label

Conversation

@Dream95

@Dream95 Dream95 commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Main Issue: #24322

Motivation

Per-consumer metrics only have consumer_name and consumer_id as labels.
Trouble is, consumer_id isn't globally unique — it's assigned per connection and gets reused.
If consumer_name is empty and consumer_id happens to collide, Prometheus can't tell consumers apart. This is probably the reason for the duplicate samples in #24322.

Modifications

  • Add a consumer_address label to per-consumer Prometheus metrics in TopicStats.writeConsumerMetric(), using the same client source address and port already exposed in consumer stats (consumer.getClientAddressAndPort()).

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is already covered by existing tests, such as (please describe tests).

  • PrometheusMetricsTest.testPerConsumerStats() — verifies consumer_address is present on consumer-level metrics and absent on subscription-level metrics, and that the value matches admin topic stats

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…us metrics

Signed-off-by: Dream95 <zhou_8621@163.com>
@void-ptr974

Copy link
Copy Markdown
Contributor

I’m concerned about adding consumer_address as a default Prometheus label. Prometheus creates a separate time series for every unique label combination. Since this value includes the client source address and port, every reconnect can produce a new label value and therefore a new time series. In large clusters or with frequent reconnects, this can quickly multiply the number of series and increase Prometheus memory/storage/query cost. I think we should avoid adding this label by default.

@Dream95

Dream95 commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

I’m concerned about adding consumer_address as a default Prometheus label. Prometheus creates a separate time series for every unique label combination. Since this value includes the client source address and port, every reconnect can produce a new label value and therefore a new time series. In large clusters or with frequent reconnects, this can quickly multiply the number of series and increase Prometheus memory/storage/query cost. I think we should avoid adding this label by default.

How about just exposing the IP? IPs don't usually change on reconnect. And in large clusters, duplicate IDs are way more likely, so an FQN really becomes a must.

…metheus label

Signed-off-by: Dream95 <zhou_8621@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants