diff --git a/operator-nexus/concepts-cluster-upgrade-overview.md b/operator-nexus/concepts-cluster-upgrade-overview.md index ae8b6a9ffd..0704c25f2b 100644 --- a/operator-nexus/concepts-cluster-upgrade-overview.md +++ b/operator-nexus/concepts-cluster-upgrade-overview.md @@ -77,6 +77,27 @@ az networkcloud baremetalmachine list -g $mrg --subscription $sub --query "sort_ --output table ``` + +## Nexus tenant workload health check during cluster runtime upgrade + +During a runtime upgrade, tenant workload (Nexus Kubernetes Cluster and Virtual Machine) health checks are performed for only rack by rack upgrade strategy. This functionality is feature flag enabled to control the cluster runtime upgrade outcome for health check failures. When the feature is enabled, the upgrade is paused if health check fails after the compute rack upgrade. The runtime upgrade can be resumed when customer executes the upgrade API [here](./howto-cluster-runtime-upgrade-with-pauserack-strategy.md). When the feature is disabled the upgrade continues to next stage even after the health check failure. By default the feature is disabled. + +The tenant workload health checks are performed after different stages of cluster runtime upgrade like control-plane, management-plane and compute servers upgrade. It operates in snapshot and comparison modes and provides a mechanism to verify tenant workload health state after different stages of cluster runtime upgrade. + +### Workflow of tenant workload health check + +1. **Snapshot Initiation** - Snapshot is collected for tenant type Nexus Kubernetes Cluster and Virtual Machine, before starting upgrade of servers. +2. **Upgrade Stage Transitions** - After upgrade of each stage like control-plane, management-plane and compute servers upgrade are completed, comparison of tenant workloads are initiated. +3. **Comparison Process** - Comparison of current tenant workloads with snapshot taken during start of upgrade. Report comparison status. +4. **Health Check Handling** - On success proceed to next runtime upgrade stage. For failure, based on feature flag is enable or disable its handled as below. + +| Runtime upgrade Stage | Feature Enable | Feature Disable | +|--------------------------|----------------------------------------------------------------|---------------------------------| +| Initial Snapshot | Upgrade failure | Upgrade continues to next stage | +| Control Plane Upgrade | Upgrade failure | Upgrade continues to next stage | +| Management Plane Upgrade | Upgrade failure | Upgrade continues to next stage | +| Compute server Upgrade | Upgrade paused, resumed when customer executes the upgrade API | Upgrade continues to next stage | + ## BareMetalMachine (BMM) keyset operations during cluster runtime upgrade When a server is upgraded to utilize a new OS, the BMM keysets have to be re-established with the new software. This process starts once the runtime upgrade completes for the instance. Servers yet to undergo a runtime upgrade can still be accessed via the BMM keyset. If access to a machine is needed during the upgrade, the console user is available.