feat: Adjust publisher options #1558
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Specifies the time to wait for new check execution's data for the tenant before removing the handler, which would cause any non delivered data to be lost. Adjust it to 65 minutes. In the worst case scenario, where the publisher is unable to push for 1h, setting it to 65 minutes will allow the longer running checks (current max=1h) to still buffer data for new executions.
When retries are exhausted, the data being pushed is not added again to the buffer queue, and instead it's discarded. With the previous configuration (20 retries, 30ms min backoff, 2s max backoff), if we consider that each push request takes ~200ms, the total time before exhausting retries and discarding data was ~35s. With the new configuration (50 retries, 50ms min backoff, 30s max backoff), if we consider that each push request takes ~200ms, the total time before exhausting retries and discarding data is ~20min. Notice that data might still be discarded earlier due to buffer size limits.
Specifies the max number of bytes to hold in the buffer per tenant in case of unsuccessfull retries before discarding data. In lack of better data for calculations.. running rough estimates we can consider each check execution to report ~15 metrics with 10 labels and one sample each. That will give us a rough estimate of ~5Kb per check execution. With the previous value of 128Kb, that would hold data for 25 check executions. In the best scenario where a tenant runs a single check in that agent, considering a check frequency of 1m, that would allow for 25 minutes before discarding data. In most cases this number will be much lower, as a tenant will be running more than one check in the same agent, resulting in a very short time window before data is discarded. Therefore, increase the buffer limit size to 1mb. Considering that that will only have an effect when data is unable to be pushed to particular Mimir/Loki cells, accept the memory increase in order to try to reduce discarded data. Autoscalers will scale agents if necessary based on the memory threshold.
Number of items limit seems difficul to reason about and setting a reasonable limit. Therefore remove this limit and apply only the size limit which fulfills the same purpose.
Adjust backoff for testing purposes.
ka3de
force-pushed
the
adjust-publisher-options
branch
from
58d9d75 to
a75a29e
Compare
nadiamoe
approved these changes
Oct 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adjusts the publisher options in order to account for new "slower running" checks (periodicity up to 1h), and reduce data discarded on a long (~15m) outage.
See particular commits for more details.
Updates grafana/synthetic-monitoring/issues/378.