Prerequisites
Code of Conduct
Feature Summary
Apply the lazy-fetch pattern from #1627 to fault-remediation's live change-stream path. That PR replaced the queued item with a documentID on the cold-start path only; the live path still retains the whole decoded event, costing 7.79 KB per queued event against 0.47 KB on the path that was fixed.
Problem/Use Case
The two paths build different queue items:
reconciler.go:2019 request := reconcileRequest{documentID: documentID} // cold start
reconciler.go:2065 request := reconcileRequest{event: &eventOut} // live streamEventWithToken.Event is a map[string]any of the full change-stream document, held for as long as the item sits in the workqueue.
Measured on an 11,000-node EKS cluster, same component, same cluster, same day:
| Backlog source |
Queued events |
Working set |
Per event |
| cold start |
200,668 |
41.5 -> 133.7 MB |
0.47 KB |
| live stream |
1,389,136 |
791 -> 9,463 MB |
7.79 KB |
For comparison, node-drainer holds 0.67-0.69 KB per queued event on either path, measured at 200k and 1M backlogs after #1121 made the same change there.
The practical consequence is the container limit. At 7.79 KB/event the shipped 256 Mi limit is exhausted at roughly 33,000 queued events, and 4 Gi at roughly 500,000. On the cold-start path the same limits hold 550,000 and 8.9 million.
This only bites when fault-remediation falls behind while running, since a backlog accumulated during a restart takes the cheap path. It normally dispatches in 0.08 s per node, faster than the drain stage feeding it, so the queue grows only under a burst large enough to outrun it -- which is precisely when an OOM is least welcome. That framing is an inference from the relative stage rates, not something measured by putting the component behind under production conditions.
Proposed Solution
Build the live-path item as reconcileRequest{documentID: ...} and fetch the event in the worker, exactly as the cold-start path does today. The lazy fetch already exists and is already measured: #1627 reports the added read as 0.285 ms per event at a 200k backlog and 0.558 ms at 800k.
Two details worth carrying over from #1121, which made the same change for node-drainer:
- Value-based
DocumentID fixes deduplication as a side effect. Two enqueues of the same logical event currently create distinct pointers and bypass workqueue dedup; with a value key the workqueue marks a re-enqueued in-flight event dirty instead of adding a duplicate entry.
- The public enqueue interface need not change.
The trade is a datastore read per queued event against holding the document in memory. #1121 measured that as an 87% memory reduction at 50k events, and #1627 as 7.8-9.7x at 200k-800k, with reconcile latency unchanged within noise.
Component
Fault Management
Prerequisites
Code of Conduct
Feature Summary
Apply the lazy-fetch pattern from #1627 to fault-remediation's live change-stream path. That PR replaced the queued item with a
documentIDon the cold-start path only; the live path still retains the whole decoded event, costing 7.79 KB per queued event against 0.47 KB on the path that was fixed.Problem/Use Case
The two paths build different queue items:
EventWithToken.Eventis amap[string]anyof the full change-stream document, held for as long as the item sits in the workqueue.Measured on an 11,000-node EKS cluster, same component, same cluster, same day:
For comparison, node-drainer holds 0.67-0.69 KB per queued event on either path, measured at 200k and 1M backlogs after #1121 made the same change there.
The practical consequence is the container limit. At 7.79 KB/event the shipped
256 Milimit is exhausted at roughly 33,000 queued events, and4 Giat roughly 500,000. On the cold-start path the same limits hold 550,000 and 8.9 million.This only bites when fault-remediation falls behind while running, since a backlog accumulated during a restart takes the cheap path. It normally dispatches in 0.08 s per node, faster than the drain stage feeding it, so the queue grows only under a burst large enough to outrun it -- which is precisely when an OOM is least welcome. That framing is an inference from the relative stage rates, not something measured by putting the component behind under production conditions.
Proposed Solution
Build the live-path item as
reconcileRequest{documentID: ...}and fetch the event in the worker, exactly as the cold-start path does today. The lazy fetch already exists and is already measured: #1627 reports the added read as 0.285 ms per event at a 200k backlog and 0.558 ms at 800k.Two details worth carrying over from #1121, which made the same change for node-drainer:
DocumentIDfixes deduplication as a side effect. Two enqueues of the same logical event currently create distinct pointers and bypass workqueue dedup; with a value key the workqueue marks a re-enqueued in-flight event dirty instead of adding a duplicate entry.The trade is a datastore read per queued event against holding the document in memory. #1121 measured that as an 87% memory reduction at 50k events, and #1627 as 7.8-9.7x at 200k-800k, with reconcile latency unchanged within noise.
Component
Fault Management