Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

target-allocator collector watcher minUpdateInterval is not configurable #3666

Open
atoulme opened this issue Jan 25, 2025 · 5 comments · May be fixed by #3672
Open

target-allocator collector watcher minUpdateInterval is not configurable #3666

atoulme opened this issue Jan 25, 2025 · 5 comments · May be fixed by #3672
Labels
area:target-allocator Issues for target-allocator enhancement New feature or request

Comments

@atoulme
Copy link
Contributor

atoulme commented Jan 25, 2025

Component(s)

target allocator

Describe the issue you're reporting

The target-allocator collector watcher uses a minUpdateInterval setting to configure how often to run a watch action.

This setting is set to 5s, and is not overridable in configuration. This issue tracks making this value configurable.

@CodePrometheus
Copy link
Contributor

@atoulme My idea is to add an UpdateInterval field to the Config struct, which will also be added to the CLI flag, could I take over this?

@atoulme
Copy link
Contributor Author

atoulme commented Jan 25, 2025

This issue has not been triaged by the project, but yes, that's what I think should be done. Thanks!

@swiatekm swiatekm added enhancement New feature or request area:target-allocator Issues for target-allocator and removed needs triage labels Jan 27, 2025
@swiatekm
Copy link
Contributor

Is there any specific reason you'd like to change this value? It only exists for rate limiting changes to the set of collectors. Any such change triggers a target reallocation, which can be expensive. We rate limit changes to targets themselves in the same way, and the interval is also not configurable there.

I'm not opposed to making it configurable, but I'd like to understand why that's necessary first.

@atoulme
Copy link
Contributor Author

atoulme commented Jan 27, 2025

This minimum is too small and we are seeing cases where the operator runs aground of the throttling limits of the API server. In general, we want all the knobs available to raise those limits for large customers. This is just one such occurrence, more will come up as we investigate.

@swiatekm
Copy link
Contributor

This minimum is too small and we are seeing cases where the operator runs aground of the throttling limits of the API server.

I'm not sure I understand. This value doesn't have any effect on how target allocator talks to the API Server - in this case, there's a watch on collector Pods, and it receives all the updates. The only thing this value affects is how often targets are reallocated as a result of changes to the active collector set.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:target-allocator Issues for target-allocator enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants