OCPBUGS-91650: disable consolidation in karpenter upgrade test

[maxcao13]

What this PR does / why we need it:

We disable consolidation as a hack to prevent flakiness in the karpenter_control_plane_upgrade suite and test. Erroenous consolidation can cause the test to fail where the new Node is consolidated by Karpenter due to Empty or Underutilized before the old node's pods get scheduled to it.

We should discuss the actual problem/fix separately in the upstream. Ref: https://redhat.atlassian.net/browse/OCPBUGS-91966. For now this should prevent flaky test failures blocking PRs from merging.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-91650

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
  • Updated an end-to-end upgrade test to reduce flakiness during control plane upgrade validation.
  • Adjusted the test’s node pool behavior to avoid consolidation-related interruptions.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-91650, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Verified instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

What this PR does / why we need it:

We disable consolidation as a hack to prevent flakiness in the karpenter_control_plane_upgrade suite and test. Erroenous consolidation can cause the test to fail where the new Node is consolidated by Karpenter due to Empty or Underutilized before the old node's pods get scheduled to it.

We should discuss the actual problem/fix separately in the upstream. Ref: https://redhat.atlassian.net/browse/OCPBUGS-91966. For now this should prevent flaky test failures blocking PRs from merging.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-91650

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[maxcao13]

/test e2e-aws-autonode

[maxcao13]

/jira refresh

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-91650, which is invalid:

• expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is MODIFIED instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[maxcao13]

/jira refresh

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-91650, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maxcao13
Once this PR has been reviewed and has the lgtm label, please assign sjenning for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 48078ddf-2a76-4e6f-9835-37d55643fba5

📥 Commits

Reviewing files that changed from the base of the PR and between 4d087b3 and 6b690a8.

📒 Files selected for processing (1)

• test/e2e/karpenter_control_plane_upgrade_test.go

---

📝 Walkthrough

Walkthrough

In TestKarpenterUpgradeControlPlane, the Karpenter NodePool's disruption configuration is updated by setting ConsolidateAfter to "Never". A TODO comment is added explaining that this prevents consolidation-driven flakiness in this blocking e2e upgrade test.

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: disabling Karpenter consolidation in the upgrade test.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	Changed file only disables Karpenter consolidation and adds comments; it contains no dynamic Ginkgo titles or other unstable test names.
Test Structure And Quality	✅ Passed	The change only disables consolidation; the test already uses explicit timeouts and cleanup helpers, and the PR adds no new structure or assertion-quality issues.
Topology-Aware Scheduling Compatibility	✅ Passed	Change only disables Karpenter consolidation in an e2e test; no node selectors, anti-affinity, topology spread, or topology-dependent replica logic added.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The change only disables Karpenter consolidation in an existing e2e test; no IPv4 literals, IPv4-only parsing, or public-internet URLs were added.
No-Weak-Crypto	✅ Passed	Changed file only adjusts Karpenter consolidation in an e2e test; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons found.
Container-Privileges	✅ Passed	Only change disables Karpenter consolidation; no privileged, host*, SYS_ADMIN, or allowPrivilegeEscalation=true settings were added.
No-Sensitive-Data-In-Logs	✅ Passed	No new logging of secrets, PII, tokens, or internal data; the change only adds a consolidation workaround comment.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci-robot]

@maxcao13: This pull request references Jira Issue OCPBUGS-91650, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

What this PR does / why we need it:

We disable consolidation as a hack to prevent flakiness in the karpenter_control_plane_upgrade suite and test. Erroenous consolidation can cause the test to fail where the new Node is consolidated by Karpenter due to Empty or Underutilized before the old node's pods get scheduled to it.

We should discuss the actual problem/fix separately in the upstream. Ref: https://redhat.atlassian.net/browse/OCPBUGS-91966. For now this should prevent flaky test failures blocking PRs from merging.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/OCPBUGS-91650

Special notes for your reviewer:

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Tests
• Updated an end-to-end upgrade test to reduce flakiness during control plane upgrade validation.
• Adjusted the test’s node pool behavior to avoid consolidation-related interruptions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.28%. Comparing base (4d087b3) to head (2d72d2f).
⚠️ Report is 46 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8874      +/-   ##
==========================================
+ Coverage   43.19%   43.28%   +0.08%     
==========================================
  Files         767      771       +4     
  Lines       94910    95503     +593     
==========================================
+ Hits        40997    41335     +338     
- Misses      51050    51284     +234     
- Partials     2863     2884      +21     

see 16 files with indirect coverage changes

Flag	Coverage Δ
cmd-support	36.67% <ø> (+0.23%)	⬆️
cpo-hostedcontrolplane	45.31% <ø> (ø)
cpo-other	45.10% <ø> (ø)
hypershift-operator	53.59% <ø> (+0.06%)	⬆️
other	31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[maxcao13]

/test e2e-aws

[maxcao13]

/test e2e-aws-autonode

[cwbotbot]

Test Results

e2e-aws

• Status: ❌ FAIL
• Started: 2026-06-30T16:04:47Z
• View Job
• View Job History

[hypershift-jira-solve-ci]

Now I have all the information I need. Let me consolidate the analysis:

12 test failures, 3 distinct root causes:

1. EnsureNoCrashingPods failures (across 2 different hosted clusters):

   • TestUpgradeControlPlane/Main/EnsureNoCrashingPods — cluster control-plane-upgrade-m5h9p
   • TestNodePool/HostedCluster0/Main/TestNodepoolMachineconfigGetsRolledout/EnsureNoCrashingPods — cluster node-pool-vdlz5
   • TestNodePool/HostedCluster0/Main/TestNTOMachineConfigGetsRolledOut/EnsureNoCrashingPods — same cluster node-pool-vdlz5
   • All show: control-plane-operator restartCount=3, hosted-cluster-config-operator restartCount=3
   • capi-provider has leader election failure (tolerated by the check)

2. TestNodePool/HostedCluster0 condition validation failure:

   • Expected ClusterVersionAvailable=False but got True
   • Expected ClusterVersionProgressing=True but got False
   • This indicates the test framework expected the cluster to be in a "not yet ready" state (no worker nodes scenario), but the cluster was already fully available

3. TestCreateClusterHABreakGlassCredentials/Teardown timeout:

   • Namespace stuck after deletion timeout (context deadline exceeded)

These are all pre-existing flakes unrelated to the PR changes (which only modify karpenter_control_plane_upgrade_test.go). The TestKarpenterUpgradeControlPlane test (the one the PR targets) PASSED.

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2071988108802396160
• Target: e2e-aws
• PR: #8874 — OCPBUGS-91650: disable consolidation in karpenter upgrade test
• Result: 597 tests, 30 skipped, 12 failures

Test Failure Analysis

Error

Container control-plane-operator in pod control-plane-operator-64b984588f-qxh5s has a restartCount > 0 (3)
Container hosted-cluster-config-operator in pod hosted-cluster-config-operator-59978f5f44-8kgwh has a restartCount > 0 (3)

Failed to wait for HostedCluster e2e-clusters-6757m/node-pool-vdlz5 to have valid conditions in 2s: context deadline exceeded
 - incorrect condition: wanted ClusterVersionAvailable=False, got ClusterVersionAvailable=True

Failed to delete test namespace: namespace still exists after deletion timeout: context deadline exceeded

Summary

All 12 test failures are pre-existing flakes unrelated to PR #8874. The PR only modifies karpenter_control_plane_upgrade_test.go (disabling Karpenter consolidation during upgrade), and the TestKarpenterUpgradeControlPlane test it targets passed successfully. The failures occur in three unrelated tests (TestUpgradeControlPlane, TestNodePool/HostedCluster0, TestCreateClusterHABreakGlassCredentials) due to: (1) control-plane-operator and hosted-cluster-config-operator pods restarting 3 times across two independent hosted clusters, tripping the EnsureNoCrashingPods zero-tolerance check; (2) a race condition in TestNodePool/HostedCluster0 where the hosted cluster fully completed version rollout before the test framework could assert the expected "in-progress" conditions; and (3) a namespace deletion timeout during TestCreateClusterHABreakGlassCredentials teardown.

Root Cause

There are three independent failure modes, none related to this PR:

1. EnsureNoCrashingPods — control-plane-operator & hosted-cluster-config-operator restarts (restartCount=3)

Affects: TestUpgradeControlPlane (cluster control-plane-upgrade-m5h9p) and TestNodePool/HostedCluster0 (cluster node-pool-vdlz5)

The EnsureNoCrashingPods check (at util.go:826-829) has a default crash toleration of 0 for most pods. The control-plane-operator and hosted-cluster-config-operator both restarted 3 times across two independent hosted clusters. The capi-provider pod also had a leader election failure, but that is detected and tolerated by the check's exception path. The restarts of control-plane-operator and hosted-cluster-config-operator are not tolerated, causing the test to fail. This is a known pre-existing flake — these operators can restart during control plane rollouts or reconciliation without impacting cluster functionality.

2. HostedCluster condition validation race condition

Affects: TestNodePool/HostedCluster0 (cluster node-pool-vdlz5)

The test framework (util.go:3158, eventually.go:225-228) validates initial HostedCluster conditions, expecting them to be in a "not yet available" state (ClusterVersionAvailable=False, ClusterVersionProgressing=True) because the cluster reportedly has no worker nodes. However, the cluster had already completed its version rollout (Done applying 5.0.0-0.ci-2026-06-30-165442-test-ci-op-86dcqx12-latest), so the actual conditions were ClusterVersionAvailable=True and ClusterVersionProgressing=False. This is a race where the validation ran after the cluster reached a terminal good state, but the test expected an intermediate state. The 2s timeout makes it unable to recover.

3. Namespace deletion timeout during teardown

Affects: TestCreateClusterHABreakGlassCredentials/Teardown

After the HostedCluster ha-break-glass-creds-t7qkf was destroyed, the test namespace e2e-clusters-9p9rm failed to be deleted within the timeout window (577s). The namespace contained finalizers or resources that prevented garbage collection within the deadline. This is a known infrastructure flake.

Recommendations

1. Retry the job (/retest) — All failures are pre-existing flakes unrelated to the PR. The PR's target test (TestKarpenterUpgradeControlPlane) passed, including its EnsureNoCrashingPods check. A retry should pass if these intermittent issues do not recur.

2. For EnsureNoCrashingPods flakes — Consider increasing the crash toleration for control-plane-operator and hosted-cluster-config-operator from 0 to a small value (e.g., 3) to accommodate expected restarts during control plane rollouts, similar to the existing toleration for ingress-operator (20 restarts) and KubeVirt infra (1 restart). The repeated occurrence across two independent clusters suggests these restarts are common during normal operations.

3. For the condition validation race — The ValidateHostedClusterConditions check sets expected conditions assuming a "no worker nodes" state at validation time, but by the time the check runs, the cluster may already be fully available. The 2s timeout is too short to self-correct. Consider either relaxing the condition expectations when the cluster is already in a terminal-good state, or running the check earlier in the lifecycle.

4. For namespace deletion timeout — This is a known infrastructure limitation. Consider increasing the teardown timeout or adding retry logic.

Evidence

Evidence	Detail
PR scope	Only modifies test/e2e/karpenter_control_plane_upgrade_test.go — adds ConsolidateAfter: "Never" to Karpenter NodePool
PR target test result	TestKarpenterUpgradeControlPlane — PASSED (3967.60s), including EnsureNoCrashingPods
Failure 1: EnsureNoCrashingPods	control-plane-operator restartCount=3 and hosted-cluster-config-operator restartCount=3 in two clusters (control-plane-upgrade-m5h9p, node-pool-vdlz5)
Failure 2: Condition race	TestNodePool/HostedCluster0: wanted ClusterVersionAvailable=False, got True; wanted ClusterVersionProgressing=True, got False — cluster already at 5.0.0-0.ci-2026-06-30-165442-test-ci-op-86dcqx12-latest
Failure 3: Namespace timeout	TestCreateClusterHABreakGlassCredentials/Teardown: namespace e2e-clusters-9p9rm stuck for 577s after HostedCluster deletion
Total test results	597 tests, 30 skipped, 12 failures (all unrelated to PR changes)
Affected pods (cluster 1)	capi-provider-587f58546-fcml4 (leader election loss, tolerated), control-plane-operator-64b984588f-qxh5s (restarts=3), hosted-cluster-config-operator-59978f5f44-8kgwh (restarts=3)
Affected pods (cluster 2)	capi-provider-54cdd745d8-8h24l (leader election loss, tolerated), control-plane-operator-d8598665b-xtsq5 (restarts=3), hosted-cluster-config-operator-68f6b6dc68-frflt (restarts=3)

---

[maxcao13]

/pipeline-required

[jparrill]

Dropped some comments. Thanks!

[jparrill]

Just confirming: "Never" is the right call here over a longer duration (e.g. "300s"). The race is timing-dependent and any finite value would still flake eventually. Since this test validates drift-based upgrade, not consolidation behavior, fully disabling it is the clean workaround. 👍

[maxcao13]

Thanks! Addressed @jparrill

[joshbranham]

/lgtm

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[openshift-ci]

@maxcao13: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aks-4-22	2d72d2f	link	true	/test e2e-aks-4-22

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.