WIP: Introduce Node Lifecycle WG

[atiratree]

No description provided.

[atiratree]

/hold

[rthallisey]

Looks like I'm not a member of kubernetes org anymore. I was a few years back, but didn't keep up with contributions recently. You can remove me as a lead and I can reapply after some contributions to this WG.

[atiratree]

We have had impactful conversations with Ryan about this group and its goals. He has experience with cluster maintenance and I look forward to his participation in the WG.

[marquiz]

/cc

[ivelichkovich]

Is it worth including something about DRA device taints/drains?

[atiratree]

Seems relevant to me as this affects the pod and device/node lifecycle. @pohly what do you think about including and discussing kubernetes/enhancements#5055 in the WG?

[pohly]

Part of the motivation for 5055 is to not have to drain entire nodes when doing some maintenance which only affects one device or hardware managed by one driver - unless of course that maintenance ends up with having to reboot the node.

So I guess it depends?

[rthallisey]

For what I understand about device taints, they are a way we can make device health scheduler-aware. This fits into our scope because we need a way to decide what Node should be prioritized for maintenance and a plan to drain that Node. E.g. a Node with all its devices tainted is a great target for Node maintenance. However, I think we should hold onto the DRA device taints feature for when we discuss Node Maintenance and Node Drain designs. I don't think we need it called out in the scope as Node Maintenance and Node Drain should cover it.

[atiratree]

We would also like to handle the pod lifecycle better in any descheduling scenario (not just Node Drain/Maintenance). One option is to use the EvictionRequest API which should give more power to the applications that are being disrupted. So it might be interesting to see if we can make the disruption more graceful in 5055.

[ivelichkovich]

Part of the motivation for 5055 is to not have to drain entire nodes when doing some maintenance which only affects one device or hardware managed by one driver - unless of course that maintenance ends up with having to reboot the node.

The other end of this to consider though is devices that span multiple nodes

[atiratree]

I have added it to the WG. I think it would be good to discuss this feature and its impact . Also, it might be better to hold off beta for some time.

[fabriziopandini]

As per KubeCon discussion I would like to be part of the working group as a representative of SCL and Cluster API.

[atiratree]

Cool, thank you for joining the WG @fabriziopandini. Added.

[JoelSpeed]

Can users not already do this with a PDB? Are we suggesting that node maintenance would override blocking PDBs if they block for some extended period of time?

I'm aware of k8s-shredder, an Adobe project that puts nodes into maintenance and then gives them a week to be clear before removing them. I'm wondering if this case is to say, even in that scenario, don't kill my workload?

[rthallisey]

I think PDBs have a different use-case, so we may need to reword. The PodDisruptionBudget protects the availability of the application. What we're saying is that there's no API that protects both the availability of the infrastructure and the availability of the application. E.g. an accelerator is degraded on a Node, so I don't want to run future workloads there but it's ok for the current one to finish. It's in the best interest of the application and infrastructure provider that an admin remediates the accelerator, so admin-user mutually agree on when that can occur.

[ivelichkovich]

Yeah I agree there's a problem that eviction API / drain doesn't guarantee it will finish within a reasonable time, especially if the node is having issues (things get stuck terminating etc.).

But this at least we can do today right?

an accelerator is degraded on a Node, so I don't want to run future workloads there but it's ok for the current one to finish

You can just taint the node or the devices with NoSchedule.

[rthallisey]

You can just taint the node or the devices with NoSchedule.

Yes, that is a solution assuming that:

1. workloads will eventually drain if we wait long enough
2. all termination steps will be successful

However, 1) can theoretically always work but it is the slowest possible solution and 2) is not guaranteed to work.

[atiratree]

I'm aware of k8s-shredder, an Adobe project that puts nodes into maintenance and then gives them a week to be clear before removing them. I'm wondering if this case is to say, even in that scenario, don't kill my workload?

This is a good example. We want the applications/admins be aware of upcoming maintenances. But also pods in most of descheduling scenarios so that they are given opportunity to migrate or cleanup before the termination, which is hard to do with PDBs.

The goal is not to override PDBs (it is also hard to do without breaking someone), the goal is to have a smarter layer above the PDBs.

[humblec]

If eviction API has been used and if the terminationGracePeriod passed, isnt there a force kill executed ? Is there a scenario where the pod get stuck for days/weeks in this case?

[atiratree]

Just the eviction API alone can cause pods to get stuck. All in all, I would prefer we do not dive deep into the topic and focus mostly on the scope in this PR.

[humblec]

Hmm.. ooc, can you please share any reference issues where eviction API itself get stuck? while I understand the scope to an extent of this WG, I feel , the need to have a new WG itself arised due to the lack of co-ordination or complexity of the Primitive/features we have in K/K around scheduling, Preemption and Eviction.. imo, with the new design if we still not address half of the issues on this area, it dont serve the purpose.

[atiratree]

When the eviction gets through; i.e. results in a delete call then everything works fine. The stuck eviction can be caused by:

• PDB itself: that is the pods don't satisfy the constraints (and never will)
• catching the eviction request via a validation webhook and discarding it (e.g. some projects from https://github.com/atiratree/kube-enhancements/blob/improve-node-maintenance/keps/sig-apps/4212-declarative-node-maintenance/README.md#motivation use it)

co-ordination or complexity of the Primitive/features we have in K/K around scheduling, Preemption and Eviction.. imo, with the new design if we still not address half of the issues on this area, it dont serve the purpose.

Yes, this is one of the important main goals that should be solved even before we introduce a node maintenance API IMO. Added to the goals to make it clear.

[JoelSpeed]

In some scenarios, evicting/removing certain pods would prevent some of these operations. Consider if we were also evicting daemonset pods as an option (is that a goal I wasn't sure?), then we might need ordering somehow to make sure the CSI driver or CNI driver aren't removed until certain other cleanup has happened

[ivelichkovich]

The way we handle this is our internal drain API has a label selector for things to ignore and by default just ignores daemonsets instead of worrying about the ordering here. Daemonsets aren't really supported under drain anyway. That should handle most cases for system level things like csi/cni/etc.

[atiratree]

Yes, daemonset pods should be considered as part of the maintenance scenarios, especially in cases when the node is going to shutdown. I have added it to the goals to make it clear.

We also had discussions with SIG node about static pod termination in the past, and they were generally not against it. But we lack use cases for it so far.

[humblec]

iiuc, this proposal aim to define an order for static pod termination, DaemonSet pods or other system-node-critical Priority class Pod termination in node drain scenario. Is that a correct assumption ?

[atiratree]

I am not saying that we will solve static pod termination, just that we will look into that :) And yes, I think there should definitely be an ordering for both DS/Static.

[dom4ha]

@erictune @sanposhiho @macsko
Some of the problems that are brought up here are actually related to scheduling, or rather a things that are missing in scheduling, that are:

1. no ability to reserve resources upfront (which makes eviction unreliable and cannot be considered atomic)
2. no ability to link pods to workloads they belong to, so admins have no tools to assess the impact of their actions.

So SIG scheduling definitely should be involved in this effort, although the problem space is much wider than the issues above and spans over the whole scheduling space.

We will take those issues into consideration, especially methods to prevent from breaking running workloads by other components. It implies giving sufficient information about the impact of such disturbing admin actions, so definitely there is an overlap between our efforts.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: atiratree
Once this PR has been reviewed and has the lgtm label, please assign pohly for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[atiratree]

For visibility, please let me know, if anyone has a relevant project they would like to see included here.

[soltysh]

Suggested change

	- Improve the Graceful/Non-Graceful Node Shutdown and consider how this affects the node lifecycle.
	To graduate the [Graceful Node Shutdown](https://github.com/kubernetes/enhancements/issues/2000)
	feature to GA and resolve the associated node shutdown issues.
	- Improve the Graceful/Non-Graceful Node Shutdown and consider how this affects the node lifecycle.

Let's stick to general topics, w/o mentioning specific KEPs in the charter.

[soltysh]

Suggested change

	- Consider improving the pod lifecycle of DaemonSets and Static pods during a node maintenance.
	- Consider improving the pod lifecycle of DaemonSets and static pods during a node maintenance.

[soltysh]

Suggested change

	(e.g. "thumbstones").
	(e.g. "tombstones").

[soltysh]

Suggested change

	and other actors want to use a new eviction API to gracefully remove pods. This would enable new
	and other actors should to use a new eviction API to gracefully remove pods. This would enable new

or even stronger must is appropriate.

[soltysh]

Nit: this entire section has 3 separate use-cases:

1. initiate
2. observe
3. discover

Can you just split them accordingly. It's easier to read shorter user stories.

[soltysh]

As mentioned elsewhere, drop these links. I'd focus on goals, not actual KEPs in charter. KEPs can and should be tracked as part of the ongoing work of a particular WG.

[soltysh]

This statement doesn't seem to fit in Area we expect to explore:. I'd drop it entirely.

[soltysh]

Are you saying there will be separate API for descheduling any Pod and a Pod with DRA device? Why both can't just use /evict?

[soltysh]

Same question here, /evict isn't sufficient?

[soltysh]

I'd like to especially stress this section:

We will strive to make these solutions minimalistic and extensible to support advanced 
use cases across the ecosystem.

to ensure we first look into existing APIs and how we can expand them, rather than introducing new ones.

We already struggle with small usage of Eviction API, adding new API will not resolve the problem, but will only make it more complicated for users to find the right one. I believe someone else already stressed that out, but I'd like to see this being one of the key goals for this WG.

[kwilczynski]

@atiratree, even though I don't work for Red Hat any more, I would like to join this WG, this topic is still of interest to me.

[selansen]

@atiratree, I would like to be part of this WG. Pls include me as well.