Provision SQS interruption queue for Karpenter - #18607
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @shreyaabaranwal. Thanks for your PR. I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
When Karpenter is enabled, kOps doesn't set up an SQS queue for it, so it never receives interruption events (spot interruptions, instance state changes, scheduled maintenance, rebalance recommendations). nodeTerminationHandler can't be enabled alongside Karpenter either, since validation blocks that combination. Net result: clusters running Karpenter get no interruption handling at all right now. This adds a KarpenterBuilder that sets up an SQS queue plus the EventBridge rules/targets for the four interruption event types, mirroring what we already do for NodeTerminationHandler's queue. The queue name is passed into the addon through a new KarpenterQueueName template function, used in both the Helm values (settings.interruptionQueue) and the static manifest (INTERRUPTION_QUEUE env var). Also adds the SQS permissions Karpenter needs to actually read from that queue (DeleteMessage, GetQueueUrl, GetQueueAttributes, ReceiveMessage). These are added unconditionally since SQS actions don't support resource-level conditions -- same reasoning already used for AddNodeTerminationHandlerSQSPermissions.
87102e7 to
ddaa2fe
Compare
|
"Hi @hakman @olemarkus, just checking in on this , happy to make any changes needed. |
|
Hi @hakman @olemarkus, following up again - happy to split this into smaller PRs if that'd make review easier, or address any specific concerns. Let me know what would help move this forward. |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
Summary
When Karpenter is enabled, kOps doesn't provision an SQS queue for it, so Karpenter never receives interruption events (spot interruptions, instance state changes, scheduled maintenance, rebalance recommendations).
nodeTerminationHandlercan't be enabled alongside Karpenter either, since validation blocks that combination, so clusters running Karpenter currently get no interruption handling at all.KarpenterBuilderthat provisions an SQS queue plus the EventBridge rules/targets for the four interruption event types, mirroring what's already done forNodeTerminationHandler's queue.KarpenterQueueNametemplate function, used in both the Helm values (settings.interruptionQueue) and the static manifest (INTERRUPTION_QUEUEenv var).DeleteMessage,GetQueueUrl,GetQueueAttributes,ReceiveMessage) toAddKarpenterPermissions, added unconditionally since SQS actions don't support resource-level conditions.Fixes #18016
Test plan
go build ./...gofmt -l .(clean aside from pre-existing vendor files)go vet ./pkg/model/... ./cmd/kops/... ./upup/pkg/fi/cloudup/...go test ./pkg/model/iam/... ./pkg/model/awsmodel/... ./pkg/model/components/addonmanifests/karpenter/... ./upup/pkg/fi/cloudup/...go test ./cmd/kops/...(updatedTestKarpentergolden fixtures to include the new SQS queue, EventBridge rules/targets, and IAM policy changes)