Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feat][broker] Support configuring replicator rate limiter per cluster #36

Conversation

nodece
Copy link
Collaborator

@nodece nodece commented Mar 6, 2025

Motivation

This PR enhances Pulsar's replication capabilities by adding support for configuring replicator rate limiters on a per-cluster basis. Currently, Pulsar only supports a single global replicator dispatch rate that applies to all remote clusters. This limitation makes it difficult to optimize replication performance when different destination clusters have varying network conditions or resource constraints.

By allowing administrators to configure different dispatch rates for different destination clusters, this PR enables more granular control over replication traffic, which can help prevent network congestion and improve overall system stability.

Modifications

  • Added support for configuring replicator dispatch rates on a per-cluster basis at both namespace and topic levels.
  • Extended the Admin API to support cluster-specific replicator dispatch rate operations.
  • Modified the internal replicator implementation to use cluster-specific rate limiters when available.
  • Updated the CLI commands to support the new cluster parameter.
  • Maintained backward compatibility with existing configurations.
  • Added comprehensive test coverage for the new functionality.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@nodece nodece force-pushed the enhance-relicator-rate-limit-by-cluster branch from 9cfe603 to 49c9259 Compare March 10, 2025 06:31
@nodece nodece force-pushed the enhance-relicator-rate-limit-by-cluster branch from 49c9259 to 1d2177c Compare March 10, 2025 09:23
Signed-off-by: Zixuan Liu <[email protected]>
@nodece nodece merged commit 15935b3 into ascentstream:branch-as-2.10 Mar 11, 2025
26 of 27 checks passed
@nodece nodece deleted the enhance-relicator-rate-limit-by-cluster branch March 11, 2025 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants