OCPBUGS-98384: fix bastion cleanup KeyPair leak by capturing infraID/region eagerly#8982
OCPBUGS-98384: fix bastion cleanup KeyPair leak by capturing infraID/region eagerly#8982bryan-cox wants to merge 1 commit into
Conversation
…Pair leak The refactor in PR openshift#8309 changed bastion cleanup from defer to t.Cleanup(), which runs after the HostedCluster is already deleted. Since DestroyBastionOpts used Name/Namespace to look up the HC for infraID/region, the lookup fails silently and the bastion EC2 instance, security group, and KeyPair are leaked on every test run. Capture infraID and region at bastion creation time and pass them directly to DestroyBastionOpts. Also use context.Background() in the cleanup closure to avoid context cancellation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
openshift-ci-robot
added
the
jira/valid-reference
|
@bryan-cox: This pull request references Jira Issue OCPBUGS-98384, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-bug
📝 WalkthroughWalkthroughThe 🚥 Pre-merge checks | ✅ 10 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (10 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
openshift-ci
Bot
added
area/testing
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: bryan-cox The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
@bryan-cox: This pull request references Jira Issue OCPBUGS-98384, which is valid. 3 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/test e2e-aws |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/util/dump/journals.go`:
- Around line 166-175: Bound the bastion cleanup context instead of passing
context.Background() directly: create a child context with a fixed timeout
inside the t.Cleanup callback, defer its cancellation, and pass it to
destroyBastion.Run. Preserve independence from the parent test context while
ensuring cleanup cannot retry indefinitely.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 2a700cb1-f11c-4537-aab8-40bf7d50bf98
📒 Files selected for processing (1)
test/e2e/util/dump/journals.go
| t.Cleanup(func() { | ||
| destroyBastion := bastionaws.DestroyBastionOpts{ | ||
| Namespace: hc.Namespace, | ||
| Name: hc.Name, | ||
| InfraID: infraID, | ||
| Region: region, | ||
| AWSCredentialsFile: awsCreds, | ||
| } | ||
| if err := destroyBastion.Run(ctx, zapr.NewLoggerWithOptions(destroyLogger)); err != nil { | ||
| if err := destroyBastion.Run(context.Background(), zapr.NewLoggerWithOptions(destroyLogger)); err != nil { | ||
| t.Logf("error destroying bastion: %v", err) | ||
| } | ||
| }) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Unbounded context.Background() risks hanging cleanup indefinitely.
DestroyBastionOpts.Run uses the context passed here for wait.PollUntilContextCancel(ctx, 5*time.Second, true, ...), retrying on any retryable AWS error. With a bare context.Background() there is no deadline, so a persistently-retryable AWS error (throttling, transient network issue) will retry forever and can hang the test process instead of just this one cleanup step. Consider bounding cleanup with a fixed timeout instead of an unbounded background context — this still protects cleanup from the parent test context being canceled, but caps worst-case blocking time.
♻️ Proposed bounded cleanup
t.Cleanup(func() {
+ destroyCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
+ defer cancel()
destroyBastion := bastionaws.DestroyBastionOpts{
InfraID: infraID,
Region: region,
AWSCredentialsFile: awsCreds,
}
- if err := destroyBastion.Run(context.Background(), zapr.NewLoggerWithOptions(destroyLogger)); err != nil {
+ if err := destroyBastion.Run(destroyCtx, zapr.NewLoggerWithOptions(destroyLogger)); err != nil {
t.Logf("error destroying bastion: %v", err)
}
})📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| t.Cleanup(func() { | |
| destroyBastion := bastionaws.DestroyBastionOpts{ | |
| Namespace: hc.Namespace, | |
| Name: hc.Name, | |
| InfraID: infraID, | |
| Region: region, | |
| AWSCredentialsFile: awsCreds, | |
| } | |
| if err := destroyBastion.Run(ctx, zapr.NewLoggerWithOptions(destroyLogger)); err != nil { | |
| if err := destroyBastion.Run(context.Background(), zapr.NewLoggerWithOptions(destroyLogger)); err != nil { | |
| t.Logf("error destroying bastion: %v", err) | |
| } | |
| }) | |
| t.Cleanup(func() { | |
| destroyCtx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) | |
| defer cancel() | |
| destroyBastion := bastionaws.DestroyBastionOpts{ | |
| InfraID: infraID, | |
| Region: region, | |
| AWSCredentialsFile: awsCreds, | |
| } | |
| if err := destroyBastion.Run(destroyCtx, zapr.NewLoggerWithOptions(destroyLogger)); err != nil { | |
| t.Logf("error destroying bastion: %v", err) | |
| } | |
| }) |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/e2e/util/dump/journals.go` around lines 166 - 175, Bound the bastion
cleanup context instead of passing context.Background() directly: create a child
context with a fixed timeout inside the t.Cleanup callback, defer its
cancellation, and pass it to destroyBastion.Run. Preserve independence from the
parent test context while ensuring cleanup cannot retry indefinitely.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8982 +/- ##
=======================================
Coverage 43.67% 43.67%
=======================================
Files 771 771
Lines 95840 95840
=======================================
Hits 41862 41862
Misses 51067 51067
Partials 2911 2911
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
@bryan-cox: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
Test Resultse2e-aws
|
|
Now I can see the full picture clearly. The build process:
The build-log.txt interleaves output, so the actual error message was lost between the build output and the next INFO line. But the builds.json This is a CI infrastructure issue — a transient 503 from the image registry proxy, not related to the PR's code changes at all. The PR only modifies bastion cleanup logic (KeyPair leak), and the failure happened during the image build phase before any test code ran. Now I have all the evidence I need. Let me produce the final report: Test Failure Analysis CompleteJob Information
Test Failure AnalysisErrorSummaryThe Root CauseThe root cause is a transient 503 Service Unavailable error from The build process pre-pulled both base images successfully (the builder golang image and the base-rhel9 runtime image). However, when the actual Key observations confirming this is infrastructure-related:
Recommendations
Evidence
|
3 participants
What this PR does / why we need it:
PR #8309 (merged May 11, 2026) refactored
test/e2e/util/dump/journals.goto reduce cyclomatic complexity, changing bastion cleanup fromdefertot.Cleanup(). This changed the execution order so that bastion destroy now runs after the HostedCluster is already deleted from Kubernetes.Since
DestroyBastionOptswas constructed withName/Namespace, the destroy code tried to look up the already-deleted HC to resolveinfraIDandregion, failed with a "not found" error, and the error was silently swallowed byt.Logf. This left EC2 bastion instances, security groups, and KeyPairs orphaned in AWS on every e2e test run. The AWS KeyPair limit (5000) was hit on June 17, 2026.Fix: Capture
hc.Spec.InfraIDandhc.Spec.Platform.AWS.Regionat bastion creation time and pass them directly toDestroyBastionOpts, bypassing the HC lookup path entirely. Also usecontext.Background()in the cleanup closure to avoid the parent context being cancelled before cleanup runs.Which issue(s) this PR fixes:
Fixes OCPBUGS-98384
Special notes for your reviewer:
DestroyBastionOpts.Run()(cmd/bastion/aws/destroy.go:87-112) supports two ways to resolve infraID/region: by looking up the HC viaName/Namespace, or by acceptingInfraID/Regiondirectly. The old test code used the lookup path, which fails when the HC is already deleted. This fix switches to passingInfraID/Regiondirectly so no Kubernetes call is needed during cleanup.Reported by Dan Mace.
Checklist: