Prerequisites
Code of Conduct
Feature Summary
Let health-events-analyzer process events concurrently, partitioned by node. It currently consumes its change stream in a single serial loop, so its throughput is fixed at one event per handling time regardless of resources, and one replica cannot be scaled or supplemented.
Problem/Use Case
Measured throughput. The component's own histogram reports 17.67 ms per event over 104,090 events (health_event_analyzer_event_handling_duration_seconds, sum 1,838.81 s). Wall time held between 17.5 and 19.4 ms across offered rates from 1.8 to 55 events/s, so it is flat rather than degrading. That fixes the ceiling at roughly 57 events/s.
Why more resources do not help. store-client/pkg/client/event_processor.go consumes the stream in one for/select loop calling handleSingleEvent synchronously; nothing in health-events-analyzer/pkg/ starts a goroutine, an errgroup, or a WaitGroup. Each event runs one aggregation per rule (reconciler.go:243 ranges over the rule set, :449 calls Aggregate), and most of the 17.67 ms is spent waiting on MongoDB rather than computing: CPU is about 5.8 ms of it, so at saturation the container draws 0.2 cores against a 2-core limit with zero throttled CFS periods. Raising CPU or memory changes nothing.
Why replicas do not help either. There is no leader election and no partitioning; replicaCount is 1 and all replicas would share a single resume token keyed by clientName: health-events-analyzer. A second replica would re-process every event and the two would overwrite each other's checkpoint.
Scale implication. At an ambient rate of 0.1 events per node per second, one replica saturates at roughly 570 nodes. A 100,000-node fleet on that assumption offers 10,000 events/s against a 57/s ceiling.
This is not a regression. #1676 improved this by roughly 10x -- the microbenchmark in #1523 measured v1.16.0 at 186 ms/event and ~4.6 events/s, against 17.67 ms and ~57/s on v1.22.0. The remaining gap is structural rather than a matter of further query tuning.
Proposed Solution
Partition by node and process partitions concurrently. #1676 already made this safe: it added an enforced same-node constraint as the mandatory first filter of every rule pipeline, so events for different nodes cannot influence each other's evaluation. Events for the same node must stay ordered relative to each other; events for different nodes need not be.
Concretely, a worker pool keyed by a hash of healthevent.nodename, with one in-flight event per node at a time. With the measured 17.67 ms per event, N workers give roughly N x 57 events/s, so 16 workers reach ~900/s and 176 reach the 10,000/s a 100,000-node fleet implies.
Checkpointing is the part that needs design. The resume token may only advance past an event once every earlier event has been handled, so out-of-order completion requires tracking a low-water mark rather than writing the token of whichever event finished last. The current code takes the simple path because completion is strictly ordered.
Two smaller changes worth considering alongside it, both suggested by the measurement rather than by the architecture:
- Each event runs 8 aggregations, one per rule set, and most of the 17.67 ms is MongoDB round-trip. Rules sharing a match shape could be evaluated in one aggregation, cutting the per-event cost directly rather than parallelising it.
- If partitioning by node is adopted, sharding across replicas becomes possible too -- separate resume tokens per shard, each covering a disjoint node range -- which would also remove the single point of failure.
Two things to confirm before accepting this design. Not every shipped rule has been verified as genuinely node-scoped -- MultipleRemediations in particular -- and a rule that correlates across nodes would break node-partitioning. And the 0.1 events per node per second above is a scale-test assumption, not a measurement from a production fleet; the real rate should be established before sizing a worker pool against it.
Prerequisites
Code of Conduct
Feature Summary
Let health-events-analyzer process events concurrently, partitioned by node. It currently consumes its change stream in a single serial loop, so its throughput is fixed at one event per handling time regardless of resources, and one replica cannot be scaled or supplemented.
Problem/Use Case
Measured throughput. The component's own histogram reports 17.67 ms per event over 104,090 events (
health_event_analyzer_event_handling_duration_seconds, sum 1,838.81 s). Wall time held between 17.5 and 19.4 ms across offered rates from 1.8 to 55 events/s, so it is flat rather than degrading. That fixes the ceiling at roughly 57 events/s.Why more resources do not help.
store-client/pkg/client/event_processor.goconsumes the stream in onefor/selectloop callinghandleSingleEventsynchronously; nothing inhealth-events-analyzer/pkg/starts a goroutine, anerrgroup, or aWaitGroup. Each event runs one aggregation per rule (reconciler.go:243ranges over the rule set,:449callsAggregate), and most of the 17.67 ms is spent waiting on MongoDB rather than computing: CPU is about 5.8 ms of it, so at saturation the container draws 0.2 cores against a 2-core limit with zero throttled CFS periods. Raising CPU or memory changes nothing.Why replicas do not help either. There is no leader election and no partitioning;
replicaCountis 1 and all replicas would share a single resume token keyed byclientName: health-events-analyzer. A second replica would re-process every event and the two would overwrite each other's checkpoint.Scale implication. At an ambient rate of 0.1 events per node per second, one replica saturates at roughly 570 nodes. A 100,000-node fleet on that assumption offers 10,000 events/s against a 57/s ceiling.
This is not a regression. #1676 improved this by roughly 10x -- the microbenchmark in #1523 measured v1.16.0 at 186 ms/event and ~4.6 events/s, against 17.67 ms and ~57/s on v1.22.0. The remaining gap is structural rather than a matter of further query tuning.
Proposed Solution
Partition by node and process partitions concurrently. #1676 already made this safe: it added an enforced same-node constraint as the mandatory first filter of every rule pipeline, so events for different nodes cannot influence each other's evaluation. Events for the same node must stay ordered relative to each other; events for different nodes need not be.
Concretely, a worker pool keyed by a hash of
healthevent.nodename, with one in-flight event per node at a time. With the measured 17.67 ms per event, N workers give roughly N x 57 events/s, so 16 workers reach ~900/s and 176 reach the 10,000/s a 100,000-node fleet implies.Checkpointing is the part that needs design. The resume token may only advance past an event once every earlier event has been handled, so out-of-order completion requires tracking a low-water mark rather than writing the token of whichever event finished last. The current code takes the simple path because completion is strictly ordered.
Two smaller changes worth considering alongside it, both suggested by the measurement rather than by the architecture:
Two things to confirm before accepting this design. Not every shipped rule has been verified as genuinely node-scoped --
MultipleRemediationsin particular -- and a rule that correlates across nodes would break node-partitioning. And the 0.1 events per node per second above is a scale-test assumption, not a measurement from a production fleet; the real rate should be established before sizing a worker pool against it.