📖 in-place updates: extension-declared Machine preservation on insufficient coverage

[LiangquanLi930]

What this PR does:
Iterates on the In-Place Updates proposal by adding preserveOnInsufficientCoverage, a new optional field on the CanUpdateMachine / CanUpdateMachineSet hook responses. When an extension cannot cover the full desired-state diff, it can set this field to request that CAPI preserve the existing Machine rather than fall back to rolling replacement.

why we need it:
For infrastructure that is expensive or destructive to replace (bare metal, local storage, license-bound nodes), the implicit fallback to replacement can destroy irreplaceable Machines. This lets the extension declare, per hook response, that the Machine should be preserved and the rollout reported as blocked, so operators can intervene.

The decision now lives in the Runtime Extension hook response rather than a user-facing KCP/MD API field, so this iteration introduces no core or KCP API changes.

Change scope:
1. Updates docs/proposals/20240807-in-place-updates.md (proposal)
2. Updates docs/proposals/20240807-in-place-updates-implementation-notes.md (MD and KCP implementation notes)
3. Documentation only — no code or API type changes
4. Field is optional; when omitted or false, existing rolling-replacement behavior is unchanged

Which issue(s) this PR fixes :
NONE

[LiangquanLi930]

/area provider/core

[LiangquanLi930]

/area documentation

[LiangquanLi930]

cc @g-gaston @fabriziopandini @sbueringer @alexander-demicev

[LiangquanLi930]

/cc @g-gaston @fabriziopandini @sbueringer @alexander-demicev

[elmiko]

i think this makes sense to me, should we have an example of the condition that will be added?

[LiangquanLi930]

i think this makes sense to me, should we have an example of the condition that will be added?

@elmiko Thanks for the suggestion. I've added an example of the condition here: https://github.com/kubernetes-sigs/cluster-api/compare/035f8772a8b41b6060c48d4663fac1fae88cd4b6..3d4b49bff1b8c201f32220dd6577cb8c0f3e991a

[elmiko]

looks great, thanks for the update!

/lgtm

[kubernetes-prow]

LGTM label has been added.

Details

Git tree hash: 94de56228b23df0b6eb046f904643ff4c42e2671

[LiangquanLi930]

/assign @enxebre @fabriziopandini @sbueringer

Got LGTM from @elmiko. This is a follow-up to the in-place updates proposal (#11029) — would appreciate an approve when you have a moment.

[LiangquanLi930]

/assign @enxebre @fabriziopandini @sbueringer

[sbueringer]

This needs careful consideration, please don't ping us every few days (4 pings now in 4 days)

[LiangquanLi930]

This needs careful consideration, please don't ping us every few days (4 pings now in 4 days)

Sorry for the frequent pings. I'll try to keep my fingers under control 😅. (I thought the assign hadn't gone through — it didn't seem to take effect — so I retried a couple of times.)

[LiangquanLi930]

Hi @sbueringer, checking in on this, does the current approach look reasonable? Happy to answer any questions. Thanks!

[fabriziopandini]

I'm a little bit perplexed by the entire idea of allowing a change that cannot be performed by a system (or by a particular cluster), and then trying to stop it. Shouldn't the change be blocked upfront?

Never the less, a few high level comments

1. We should not introduce API changes for in place in this phase
2. I do not consider "never create machines" a realistic goal, because there are several lifecycle operations that requires to create machines
3. Stopping rollouts is a very generic goal we should qualify it or break it down in smaller problems
4. There should alway be a clear recovery path when the system gets stuck for any reason.

Let me add a few line more on 2 and 3.

While we usually think in term of workflows, e.g. a rollout, in Kubernetes controllers there is no notion of workflows, just an infinite sequence of micro decisions that starts by assessing the current state of system, and if possible, apply the next change shifting the system towards the desired state (all of this taking into considerations constraints like availability, max surge etc.)

On top of that, the system allows to multiple change the desired state, no matter if the previous change is already fully rolled out or not, so, in the context of this proposal, it is not clear how to identify what the is the rollout to be stopped when the user changed spec, scaled up, remediation and autoscaling kicked in, and then the user changed spec again...

Figuring out this requires a deep investigation in the code and addressing several edge cases and system caveats not yet included in this proposal

• what will happen to MachineDeployment's MS when "stop-rollout" or "unblock stopped rollout" happens
• is it enough to introduce "stop-rollout", when we are still allowing max surge (that leads to a rollout)
• what about remediation, or the deadlock blocker in the rollout logic (or in general other cases that lead to a machine deletion + create a new one)
• how this concept apply to KCP (or control planes in general)
• how we can propagate the "stop rollout" or "unblock stopped rollout" decision from MD controller (when the rollout decision is taken) to the MachineSet or to the Machine controller (where the up-to-date condition is set).
• how do we make such change involving many controllers re-entrant, robust etc.
• ... (for sure there are more)

Frankly speaking I think that we should try to slice the problem is smaller, well scoped improvements aimed at providing a better granular control on how a rollout happens vs trying to achieve "all or nothing". See e.g. #14160 that goes in this direction

[kubernetes-prow]

New changes are detected. LGTM label has been removed.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from enxebre. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[LiangquanLi930]

Thanks @fabriziopandini for the detailed explanation, very helpful.

I've reworked this PR based on your feedback: the preservation decision now lives in the Runtime Extension hook response (preserveOnInsufficientCoverage) instead of a user-facing KCP/MD API field, so this iteration introduces no core or KCP API changes.

PTAL when you have a chance.