Skip to content

Commit e64c326

Browse files
authored
Replace the internal reactivelimiter with a wrapped Failsafe-go limiter (#12773)
<!-- Thanks for sending a pull request! Before submitting: 1. Read our CONTRIBUTING.md guide 2. Rebase your PR if it gets out of sync with main --> #### What this PR does This PR replaces Mimir's internal reactive limiter with a wrapper around Failsafe-go's [adaptive limiter](https://failsafe-go.dev/adaptive-limiter/), which has moved ahead in terms of maturity. It also uses Failsafe-go's [Prioritizer](https://failsafe-go.dev/execution-prioritization/). #### Which issue(s) this PR fixes or relates to Fixes #<issue number> #### Checklist - [ ] Tests updated. - [x] Documentation added. - [x] `CHANGELOG.md` updated - the order of entries should be `[CHANGE]`, `[FEATURE]`, `[ENHANCEMENT]`, `[BUGFIX]`. If changelog entry is not needed, please add the `changelog-not-needed` label to the PR. - [ ] [`about-versioning.md`](https://github.com/grafana/mimir/blob/main/docs/sources/mimir/configure/about-versioning.md) updated with experimental features.
1 parent 9b47112 commit e64c326

36 files changed

+1434
-2211
lines changed

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
* [CHANGE] Cost attribution: Reduce the default maximum per-user cardinality of cost attribution labels to 2000. #12625
2323
* [CHANGE] Querier, query-frontend: Add `_total` suffix to `cortex_mimir_query_engine_common_subexpression_elimination_duplication_nodes_introduced`, `cortex_mimir_query_engine_common_subexpression_elimination_selectors_eliminated` and `cortex_mimir_query_engine_common_subexpression_elimination_selectors_inspected` metric names. #12636
2424
* [CHANGE] Alertmanager: Change the severity for InitialSyncFailed from 'critical' to 'warning'. #12824
25+
* [CHANGE] Ingester: Renamed experimental reactive limiter options. #12773
2526
* [FEATURE] Ingester: Expose the number of active series ingested via OTLP as `cortex_ingester_active_otlp_series`. #12678
2627
* [FEATURE] Distributor, ruler: Add experimental `-validation.name-validation-scheme` flag to specify the validation scheme for metric and label names. #12215
2728
* [FEATURE] Ruler: Add support to use a Prometheus-compatible HTTP endpoint for remote rule evaluation. See [remote evaluation mode](https://grafana.com/docs/mimir/latest/operators-guide/architecture/components/ruler/#remote-over-http-https) for more details. This feature can be used to federate data from multiple Mimir instances. #11415

cmd/mimir/config-descriptor.json

Lines changed: 88 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -4399,7 +4399,7 @@
43994399
"kind": "field",
44004400
"name": "enabled",
44014401
"required": false,
4402-
"desc": "Enable reactive limiting when making requests to ingesters",
4402+
"desc": "Enable reactive limiting when making requests to a service",
44034403
"fieldValue": null,
44044404
"fieldDefaultValue": false,
44054405
"fieldFlag": "ingester.push-reactive-limiter.enabled",
@@ -4408,101 +4408,101 @@
44084408
},
44094409
{
44104410
"kind": "field",
4411-
"name": "short_window_min_duration",
4411+
"name": "min_limit",
44124412
"required": false,
4413-
"desc": "Minimum duration of the window that is used to determine the recent, short-term load on the system",
4413+
"desc": "Minimum inflight requests limit",
44144414
"fieldValue": null,
4415-
"fieldDefaultValue": 1000000000,
4416-
"fieldFlag": "ingester.push-reactive-limiter.short-window-min-duration",
4417-
"fieldType": "duration",
4415+
"fieldDefaultValue": 2,
4416+
"fieldFlag": "ingester.push-reactive-limiter.min-limit",
4417+
"fieldType": "int",
44184418
"fieldCategory": "experimental"
44194419
},
44204420
{
44214421
"kind": "field",
4422-
"name": "short_window_max_duration",
4422+
"name": "max_limit",
44234423
"required": false,
4424-
"desc": "Maximum duration of the window that is used to determine the recent, short-term load on the system",
4424+
"desc": "Maximum inflight requests limit",
44254425
"fieldValue": null,
4426-
"fieldDefaultValue": 30000000000,
4427-
"fieldFlag": "ingester.push-reactive-limiter.short-window-max-duration",
4428-
"fieldType": "duration",
4426+
"fieldDefaultValue": 200,
4427+
"fieldFlag": "ingester.push-reactive-limiter.max-limit",
4428+
"fieldType": "int",
44294429
"fieldCategory": "experimental"
44304430
},
44314431
{
44324432
"kind": "field",
4433-
"name": "short_window_min_samples",
4433+
"name": "initial_limit",
44344434
"required": false,
4435-
"desc": "Minimum number of samples that must be recorded in the window",
4435+
"desc": "Initial inflight requests limit",
44364436
"fieldValue": null,
4437-
"fieldDefaultValue": 50,
4438-
"fieldFlag": "ingester.push-reactive-limiter.short-window-min-samples",
4437+
"fieldDefaultValue": 20,
4438+
"fieldFlag": "ingester.push-reactive-limiter.initial-limit",
44394439
"fieldType": "int",
44404440
"fieldCategory": "experimental"
44414441
},
44424442
{
44434443
"kind": "field",
4444-
"name": "long_window",
4444+
"name": "max_limit_factor",
44454445
"required": false,
4446-
"desc": "Short-term window measurements that are stored in an exponentially weighted moving average window, representing the long-term baseline inflight time",
4446+
"desc": "The maximum inflight limit as a multiple of current inflight requests",
44474447
"fieldValue": null,
4448-
"fieldDefaultValue": 60,
4449-
"fieldFlag": "ingester.push-reactive-limiter.long-window",
4450-
"fieldType": "int",
4448+
"fieldDefaultValue": 5,
4449+
"fieldFlag": "ingester.push-reactive-limiter.max-limit-factor",
4450+
"fieldType": "float",
44514451
"fieldCategory": "experimental"
44524452
},
44534453
{
44544454
"kind": "field",
4455-
"name": "sample_quantile",
4455+
"name": "recent_window_min_duration",
44564456
"required": false,
4457-
"desc": "The quantile of recorded response times to consider when adjusting the concurrency limit",
4457+
"desc": "Minimum duration of the window that is used to collect recent response time samples",
44584458
"fieldValue": null,
4459-
"fieldDefaultValue": 0.9,
4460-
"fieldFlag": "ingester.push-reactive-limiter.sample-quantile",
4461-
"fieldType": "float",
4459+
"fieldDefaultValue": 1000000000,
4460+
"fieldFlag": "ingester.push-reactive-limiter.recent-window-min-duration",
4461+
"fieldType": "duration",
44624462
"fieldCategory": "experimental"
44634463
},
44644464
{
44654465
"kind": "field",
4466-
"name": "min_inflight_limit",
4466+
"name": "recent_window_max_duration",
44674467
"required": false,
4468-
"desc": "Minimum inflight requests limit",
4468+
"desc": "Maximum duration of the window that is used to collect recent response time samples",
44694469
"fieldValue": null,
4470-
"fieldDefaultValue": 2,
4471-
"fieldFlag": "ingester.push-reactive-limiter.min-inflight-limit",
4472-
"fieldType": "int",
4470+
"fieldDefaultValue": 30000000000,
4471+
"fieldFlag": "ingester.push-reactive-limiter.recent-window-max-duration",
4472+
"fieldType": "duration",
44734473
"fieldCategory": "experimental"
44744474
},
44754475
{
44764476
"kind": "field",
4477-
"name": "max_inflight_limit",
4477+
"name": "recent_window_min_samples",
44784478
"required": false,
4479-
"desc": "Maximum inflight requests limit",
4479+
"desc": "Minimum number of samples that must be recorded in the recent window before updating the limit",
44804480
"fieldValue": null,
4481-
"fieldDefaultValue": 200,
4482-
"fieldFlag": "ingester.push-reactive-limiter.max-inflight-limit",
4481+
"fieldDefaultValue": 50,
4482+
"fieldFlag": "ingester.push-reactive-limiter.recent-window-min-samples",
44834483
"fieldType": "int",
44844484
"fieldCategory": "experimental"
44854485
},
44864486
{
44874487
"kind": "field",
4488-
"name": "initial_inflight_limit",
4488+
"name": "recent_quantile",
44894489
"required": false,
4490-
"desc": "Initial inflight requests limit",
4490+
"desc": "The quantile of recent recorded response times to consider when adjusting the concurrency limit",
44914491
"fieldValue": null,
4492-
"fieldDefaultValue": 20,
4493-
"fieldFlag": "ingester.push-reactive-limiter.initial-inflight-limit",
4494-
"fieldType": "int",
4492+
"fieldDefaultValue": 0.9,
4493+
"fieldFlag": "ingester.push-reactive-limiter.recent-quantile",
4494+
"fieldType": "float",
44954495
"fieldCategory": "experimental"
44964496
},
44974497
{
44984498
"kind": "field",
4499-
"name": "max_limit_factor",
4499+
"name": "baseline_window_age",
45004500
"required": false,
4501-
"desc": "The maximum limit as a multiple of current inflight requests",
4501+
"desc": "The average age of baseline samples aggregated recent samples are added to",
45024502
"fieldValue": null,
4503-
"fieldDefaultValue": 5,
4504-
"fieldFlag": "ingester.push-reactive-limiter.max-limit-factor",
4505-
"fieldType": "float",
4503+
"fieldDefaultValue": 10,
4504+
"fieldFlag": "ingester.push-reactive-limiter.baseline-window-age",
4505+
"fieldType": "int",
45064506
"fieldCategory": "experimental"
45074507
},
45084508
{
@@ -4552,7 +4552,7 @@
45524552
"kind": "field",
45534553
"name": "enabled",
45544554
"required": false,
4555-
"desc": "Enable reactive limiting when making requests to ingesters",
4555+
"desc": "Enable reactive limiting when making requests to a service",
45564556
"fieldValue": null,
45574557
"fieldDefaultValue": false,
45584558
"fieldFlag": "ingester.read-reactive-limiter.enabled",
@@ -4561,101 +4561,101 @@
45614561
},
45624562
{
45634563
"kind": "field",
4564-
"name": "short_window_min_duration",
4564+
"name": "min_limit",
45654565
"required": false,
4566-
"desc": "Minimum duration of the window that is used to determine the recent, short-term load on the system",
4566+
"desc": "Minimum inflight requests limit",
45674567
"fieldValue": null,
4568-
"fieldDefaultValue": 1000000000,
4569-
"fieldFlag": "ingester.read-reactive-limiter.short-window-min-duration",
4570-
"fieldType": "duration",
4568+
"fieldDefaultValue": 2,
4569+
"fieldFlag": "ingester.read-reactive-limiter.min-limit",
4570+
"fieldType": "int",
45714571
"fieldCategory": "experimental"
45724572
},
45734573
{
45744574
"kind": "field",
4575-
"name": "short_window_max_duration",
4575+
"name": "max_limit",
45764576
"required": false,
4577-
"desc": "Maximum duration of the window that is used to determine the recent, short-term load on the system",
4577+
"desc": "Maximum inflight requests limit",
45784578
"fieldValue": null,
4579-
"fieldDefaultValue": 30000000000,
4580-
"fieldFlag": "ingester.read-reactive-limiter.short-window-max-duration",
4581-
"fieldType": "duration",
4579+
"fieldDefaultValue": 200,
4580+
"fieldFlag": "ingester.read-reactive-limiter.max-limit",
4581+
"fieldType": "int",
45824582
"fieldCategory": "experimental"
45834583
},
45844584
{
45854585
"kind": "field",
4586-
"name": "short_window_min_samples",
4586+
"name": "initial_limit",
45874587
"required": false,
4588-
"desc": "Minimum number of samples that must be recorded in the window",
4588+
"desc": "Initial inflight requests limit",
45894589
"fieldValue": null,
4590-
"fieldDefaultValue": 50,
4591-
"fieldFlag": "ingester.read-reactive-limiter.short-window-min-samples",
4590+
"fieldDefaultValue": 20,
4591+
"fieldFlag": "ingester.read-reactive-limiter.initial-limit",
45924592
"fieldType": "int",
45934593
"fieldCategory": "experimental"
45944594
},
45954595
{
45964596
"kind": "field",
4597-
"name": "long_window",
4597+
"name": "max_limit_factor",
45984598
"required": false,
4599-
"desc": "Short-term window measurements that are stored in an exponentially weighted moving average window, representing the long-term baseline inflight time",
4599+
"desc": "The maximum inflight limit as a multiple of current inflight requests",
46004600
"fieldValue": null,
4601-
"fieldDefaultValue": 60,
4602-
"fieldFlag": "ingester.read-reactive-limiter.long-window",
4603-
"fieldType": "int",
4601+
"fieldDefaultValue": 5,
4602+
"fieldFlag": "ingester.read-reactive-limiter.max-limit-factor",
4603+
"fieldType": "float",
46044604
"fieldCategory": "experimental"
46054605
},
46064606
{
46074607
"kind": "field",
4608-
"name": "sample_quantile",
4608+
"name": "recent_window_min_duration",
46094609
"required": false,
4610-
"desc": "The quantile of recorded response times to consider when adjusting the concurrency limit",
4610+
"desc": "Minimum duration of the window that is used to collect recent response time samples",
46114611
"fieldValue": null,
4612-
"fieldDefaultValue": 0.9,
4613-
"fieldFlag": "ingester.read-reactive-limiter.sample-quantile",
4614-
"fieldType": "float",
4612+
"fieldDefaultValue": 1000000000,
4613+
"fieldFlag": "ingester.read-reactive-limiter.recent-window-min-duration",
4614+
"fieldType": "duration",
46154615
"fieldCategory": "experimental"
46164616
},
46174617
{
46184618
"kind": "field",
4619-
"name": "min_inflight_limit",
4619+
"name": "recent_window_max_duration",
46204620
"required": false,
4621-
"desc": "Minimum inflight requests limit",
4621+
"desc": "Maximum duration of the window that is used to collect recent response time samples",
46224622
"fieldValue": null,
4623-
"fieldDefaultValue": 2,
4624-
"fieldFlag": "ingester.read-reactive-limiter.min-inflight-limit",
4625-
"fieldType": "int",
4623+
"fieldDefaultValue": 30000000000,
4624+
"fieldFlag": "ingester.read-reactive-limiter.recent-window-max-duration",
4625+
"fieldType": "duration",
46264626
"fieldCategory": "experimental"
46274627
},
46284628
{
46294629
"kind": "field",
4630-
"name": "max_inflight_limit",
4630+
"name": "recent_window_min_samples",
46314631
"required": false,
4632-
"desc": "Maximum inflight requests limit",
4632+
"desc": "Minimum number of samples that must be recorded in the recent window before updating the limit",
46334633
"fieldValue": null,
4634-
"fieldDefaultValue": 200,
4635-
"fieldFlag": "ingester.read-reactive-limiter.max-inflight-limit",
4634+
"fieldDefaultValue": 50,
4635+
"fieldFlag": "ingester.read-reactive-limiter.recent-window-min-samples",
46364636
"fieldType": "int",
46374637
"fieldCategory": "experimental"
46384638
},
46394639
{
46404640
"kind": "field",
4641-
"name": "initial_inflight_limit",
4641+
"name": "recent_quantile",
46424642
"required": false,
4643-
"desc": "Initial inflight requests limit",
4643+
"desc": "The quantile of recent recorded response times to consider when adjusting the concurrency limit",
46444644
"fieldValue": null,
4645-
"fieldDefaultValue": 20,
4646-
"fieldFlag": "ingester.read-reactive-limiter.initial-inflight-limit",
4647-
"fieldType": "int",
4645+
"fieldDefaultValue": 0.9,
4646+
"fieldFlag": "ingester.read-reactive-limiter.recent-quantile",
4647+
"fieldType": "float",
46484648
"fieldCategory": "experimental"
46494649
},
46504650
{
46514651
"kind": "field",
4652-
"name": "max_limit_factor",
4652+
"name": "baseline_window_age",
46534653
"required": false,
4654-
"desc": "The maximum limit as a multiple of current inflight requests",
4654+
"desc": "The average age of baseline samples aggregated recent samples are added to",
46554655
"fieldValue": null,
4656-
"fieldDefaultValue": 5,
4657-
"fieldFlag": "ingester.read-reactive-limiter.max-limit-factor",
4658-
"fieldType": "float",
4656+
"fieldDefaultValue": 10,
4657+
"fieldFlag": "ingester.read-reactive-limiter.baseline-window-age",
4658+
"fieldType": "int",
46594659
"fieldCategory": "experimental"
46604660
},
46614661
{

0 commit comments

Comments
 (0)