How to gracefully shutdown a k3s single node cluster (revisited)

[kinghrothgar]

I have converted the majority of my server from systemd units over to k3s. One issue I've discovered however is that it's unclear to me how to gracefully shutdown my k3s cluster or my entire server. I need to make sure that various containers exit in the correct order and that all containers exit before my various file systems used by the containers get unmounted. Also, I need my pods to be killed with SIGTERM, not SIGKILL.

Containers shutting down in the correct order: I found this native kubelet solution for that #4319. However, I don't understand how this would work with the current version of k3s as there is no kubelet? Stopping the systemd unit causes the k3s server to exit without doing anything to the pods. As such, there is no kubelet systemd unit to get stopped and trigger the kubelet to take a systemd-inhibit lock: https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/.

As there does not seem to be a systemd unit tied to the "kubelet", I do not know how to block my various FS mounts from being unmounted before all of the pods have exited. I did add the appropriate After's to the k3s.service which I think might be working as all the containers are child processes of the k3s.service, so even though k3 server exits instantly, I believe systemd is waiting for all the containers to exit before umounting the needed mounts.

Is there something I am missing or misunderstanding? Given what I have been able to search here and google, it is unclear to me how to gracefully shutdown k3s,

[brandond]

The discussion in that thread is what you should be referencing. There does not need to be a separate systemd unit for the kubelet, the kubelet runs in the main k3s process and systemd service.

Have you tried what is suggested in that thread?