Skip to content

🐛 Fix MachinePool nodeRef UID mismatch after K8s upgrade #12392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

jayesh-srivastava
Copy link
Member

What this PR does / why we need it:
When a K8s upgrade is performed on a Managed cluster, new nodes will come up with new UIDs. However, the MachinePool controller has an early return condition that only validates the count of NodeRefs but doesn't check if the UIDs are still valid. This leads to MachinePools retaining stale NodeRef UIDs after upgrades, causing UID mismatches that persist until manual intervention.
This PR adds UID validation logic before the early return condition.

  • A new name-to-node map with the name nodeNameMap is created.
  • We iterate over the mp.Status.NodeRefs and using the above map, get each each Node.
  • If the Node doesn't exists or the fetched Node's UID is not matching to the UID in NodeRef, we break and continue with further reconciliation. This will set the correct nodeRef in the Machine Pool.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #12388

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Jun 24, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign vincepri for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. area/machine Issues or PRs related to machine lifecycle management labels Jun 24, 2025
@k8s-ci-robot
Copy link
Contributor

@jayesh-srivastava: The label(s) area/pool cannot be applied, because the repository doesn't have them.

In response to this:

/area machine pool

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/needs-area PR is missing an area label label Jun 24, 2025
@jayesh-srivastava
Copy link
Member Author

/area machinepool

@k8s-ci-robot k8s-ci-robot added the area/machinepool Issues or PRs related to machinepools label Jun 24, 2025
Copy link
Contributor

@mboersma mboersma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems reasonable to add this additional verification in general, but I wonder if this problem has been seen in providers other than CAPZ.

// Validate that the UIDs in NodeRefs are still valid
if s.nodeRefMap != nil {
// Create a name-to-node mapping for efficient lookup
nodeNameMap := make(map[string]*corev1.Node)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
nodeNameMap := make(map[string]*corev1.Node)
nodeNameMap := make(map[string]*corev1.Node, len(s.nodeRefMap))

@sbueringer
Copy link
Member

It seems reasonable to add this additional verification in general, but I wonder if this problem has been seen in providers other than CAPZ.

cc @richardcase @justinsb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/machine Issues or PRs related to machine lifecycle management area/machinepool Issues or PRs related to machinepools cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Node UID in MachinePool nodeRef mismatch post Kubernetes upgrade
4 participants