Skip to content

Update cluster runtime upgrade with expand health check feature #3885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Update cluster runtime upgrade with expand health check feature #3885

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
db537eb
Update cluster runtime upgrade with expand health check feature
deepukraju Sep 23, 2025
f592b78
update with review comments
deepukraju Sep 25, 2025
1541c9b
Update health check behavior description in upgrade overview
deepukraju Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions operator-nexus/concepts-cluster-upgrade-overview.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,27 @@ az networkcloud baremetalmachine list -g $mrg --subscription $sub --query "sort_
--output table

```

## Nexus tenant workload health check during cluster runtime upgrade

During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the upgrade continues to next stage even after the health check failure. By default the feature is disabled.
Copy link

@seaneagan seaneagan Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to reference the AFEC flag name here? (EnableUpgradeInventoryChecks) or are we relying on other documentation elsewhere for that?

Copy link
Author

@deepukraju deepukraju Oct 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is added in TSG


The tenant workload health checks are performed after different stages of cluster runtime upgrade like control-plane, management-plane and compute servers upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify tenant workload health state after different stages of cluster runtime upgrade.

### Workflow of tenant workload health check

1. **Snapshot Initiation** - Snapshot is collected for tenant type Nexus Kubernetes Cluster and Virtual Machine, before starting upgrade of servers.
2. **Upgrade Stage Transitions** - After upgrade of each stage like control-plane, management-plane and compute servers upgrade are completed, comparison of tenant workloads are initiated.
3. **Comparison Process** - Comparison of current tenant workloads with snapshot taken during start of upgrade. Report comparison status.
4. **Health Check Handling** - On success proceed to next runtime upgrade stage. For failure, based on feature flag is enable or disable its handled as below.

| Runtime upgrade Stage | Feature Enable | Feature Disable |
|--------------------------|----------------------------------------------------------------|---------------------------------|
| Initial Snapshot | Upgrade failure | Upgrade continues to next stage |
| Control Plane Upgrade | Upgrade failure | Upgrade continues to next stage |
| Management Plane Upgrade | Upgrade failure | Upgrade continues to next stage |
| Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage |
Copy link

@seaneagan seaneagan Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage |
| Compute server Upgrade | Upgrade paused, resumed when customer executes the continue API | Upgrade continues to next stage |


## BareMetalMachine (BMM) keyset operations during cluster runtime upgrade

When a server is upgraded to utilize a new OS, the BMM keysets have to be re-established with the new software. This process starts once the runtime upgrade completes for the instance. Servers yet to undergo a runtime upgrade can still be accessed via the BMM keyset. If access to a machine is needed during the upgrade, the console user is available.
Expand Down