K3s eating too much disc space

[smartlocus]

**Hello Guys , i have been running k3s for a while now with about 100 workloads and it seems like k3s has been eating much of my disc space for some reason.

i am running k3s version v1.31.4+k3s1 (a562d09)
go version go1.22.9.

I would appreiate it if someone could tell me how to fix this without losing data **

Filesystem                                                                                                         Size  Used Avail Use% Mounted on
tmpfs                                                                                                              4,8G   20M  4,7G   1% /run
/dev/vda3                                                                                                          733G  508G  196G  73% /
tmpfs                                                                                                               24G     0   24G   0% /dev/shm
tmpfs                                                                                                              5,0M     0  5,0M   0% /run/lock
/dev/vda2                                                                                                          974M  242M  665M  27% /boot
overlay                                                                                                            733G  508G  196G  73% /var/lib/docker/overlay2/356fa53d1c1e649be8762a1351eca16d5113bf94dec5c3bb5428e5762cff1756/merged
overlay                                                                                                            733G  508G  196G  73% /var/lib/docker/overlay2/55fe9e74f8f21e1c16600cbc566a82cb909f88643d75c8aba70ec2251cf2df8d/merged
tmpfs                                                                                                               48G  8,0K   48G   1% /var/lib/kubelet/pods/5e3be513-8194-46aa-8abf-cdbd8e8821c3/volumes/kubernetes.io~secret/apiservice-cert
tmpfs                                                                                                               48G   24K   48G   1% /var/lib/kubelet/pods/dcf7cff6-ba5c-48d8-82b1-69a7af6d8aef/volumes/kubernetes.io~secret/secret
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/d28752a4-6a37-4e08-9c72-b23911f52170/volumes/kubernetes.io~empty-dir/runner-secrets
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/4ab84779-0cbe-46e1-95f8-a10b7432c9ea/volumes/kubernetes.io~projected/rook-config-override
tmpfs                                                                                                               48G  8,0K   48G   1% /var/lib/kubelet/pods/5e3be513-8194-46aa-8abf-cdbd8e8821c3/volumes/kubernetes.io~secret/webhook-cert
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/4ab84779-0cbe-46e1-95f8-a10b7432c9ea/volumes/kubernetes.io~secret/rook-ceph-mds-myfs-a-keyring
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/d28752a4-6a37-4e08-9c72-b23911f52170/volumes/kubernetes.io~projected/init-runner-secrets
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/bea7ea02-a12f-4463-b088-876d79787b15/volumes/kubernetes.io~projected/kube-api-access-hprmm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/9119a643-e7da-4485-a0f8-d6f400cdf6d5/volumes/kubernetes.io~projected/kube-api-access-phmrw
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/f474c9b9-cbe0-4f51-bce1-99e546c6a061/volumes/kubernetes.io~projected/kube-api-access-2b2dv
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/ab68f75b-7a22-46cd-b6bd-c798949f4baa/volumes/kubernetes.io~projected/kube-api-access-zpnhd
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/f65afdcf-589f-4465-b027-be98c85f0f86/volumes/kubernetes.io~projected/kube-api-access-x9k6q
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/873e5fa3-9609-4b92-9b03-0bed78be6d0d/volumes/kubernetes.io~projected/kube-api-access-xn886
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/9431aa73-c834-4502-b115-dc5958b20e74/volumes/kubernetes.io~projected/kube-api-access-nd22v
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/6e65bdff-be56-4da6-b4a6-8c272384cc49/volumes/kubernetes.io~projected/kube-api-access-bs4fs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/9eec1dfe-3195-45fd-8034-5987531a8ce8/volumes/kubernetes.io~projected/kube-api-access-tchbg
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/5e3be513-8194-46aa-8abf-cdbd8e8821c3/volumes/kubernetes.io~projected/kube-api-access-wzcm7
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/f322bb12-664b-4801-aa66-0715ffb35dad/volumes/kubernetes.io~empty-dir/socket-dir
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/f322bb12-664b-4801-aa66-0715ffb35dad/volumes/kubernetes.io~empty-dir/keys-tmp-dir
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/3425c7c8-487b-4bce-ae22-28ed2bb442d9/volumes/kubernetes.io~projected/kube-api-access-sjljf
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/dcf7cff6-ba5c-48d8-82b1-69a7af6d8aef/volumes/kubernetes.io~projected/kube-api-access-8rxxk
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/d28752a4-6a37-4e08-9c72-b23911f52170/volumes/kubernetes.io~projected/kube-api-access-sm6mn
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/4ab84779-0cbe-46e1-95f8-a10b7432c9ea/volumes/kubernetes.io~projected/kube-api-access-7vdpl
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/99772c0a-9296-4da2-9ffe-2849b121ed52/volumes/kubernetes.io~projected/kube-api-access-rsc2j
tmpfs                                                                                                              170M   12K  170M   1% /var/lib/kubelet/pods/d111cf07-005b-451f-84b7-96f6e688fed9/volumes/kubernetes.io~projected/kube-api-access-cbk8f
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/a7b62d68-6741-4652-aceb-bcd3d68b70cc/volumes/kubernetes.io~projected/kube-api-access-pfz28
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/f322bb12-664b-4801-aa66-0715ffb35dad/volumes/kubernetes.io~projected/kube-api-access-xn4bt
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/cbc84ea2-aaf5-4368-8536-639548601284/volumes/kubernetes.io~projected/kube-api-access-fmjt8
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/f620b39d-34d8-40a6-afdf-7c2cddc966be/volumes/kubernetes.io~projected/kube-api-access-l2xsk
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/7827f51aa21af96895ece504cb550e6ed0467487af1a9149776e3fdcb7513888/shm
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/aedfe6e1-1328-4b24-9c87-70e92faef6ca/volumes/kubernetes.io~secret/argocd-repo-server-tls
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/e4de751d0dd18e663c8d5ea25ab6a6e6692fc9ceabc6ca7e70159046b155b947/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/4ee8907b-eaaa-4093-a713-636b4b300a5a/volumes/kubernetes.io~projected/kube-api-access-dq5hh
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/74972c2bdb3005d7f5fd280356daac257dc603b06ad94c450f59d5777c789517/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/dd3688dc-3b5f-449e-b90d-b1db54fe849d/volumes/kubernetes.io~projected/kube-api-access-lpd9w
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/24122fc5-b91a-4810-8e7c-656f4b5107b8/volumes/kubernetes.io~projected/kube-api-access-bgc8m
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/7827f51aa21af96895ece504cb550e6ed0467487af1a9149776e3fdcb7513888/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e4de751d0dd18e663c8d5ea25ab6a6e6692fc9ceabc6ca7e70159046b155b947/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/a90a29b3-5c40-48c7-9608-6f983ac49c24/volumes/kubernetes.io~projected/kube-api-access-6wgb5
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/74972c2bdb3005d7f5fd280356daac257dc603b06ad94c450f59d5777c789517/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/0278496fac8274a15752c7d912d6adb2308e671593e1acdebcf4267b9c5275ec/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/10b76ca14d31c66bab22a45db312e29b3c2016a281ef811ef0b54f797ee9ad79/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9ba01efda4134422f998aaf98b04b5babb4e7e4ca1e7efdd5ede243c770270ed/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/0278496fac8274a15752c7d912d6adb2308e671593e1acdebcf4267b9c5275ec/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/00549abe-b062-4c89-8a07-a940dafdeaf6/volumes/kubernetes.io~projected/kube-api-access-p7k9f
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9ba01efda4134422f998aaf98b04b5babb4e7e4ca1e7efdd5ede243c770270ed/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/50830474c1437b8737786bf88ae887fecb172166de3c6a7cd807fc0719ec5830/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/10b76ca14d31c66bab22a45db312e29b3c2016a281ef811ef0b54f797ee9ad79/rootfs
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/99e3d6b5-d930-48b5-bdaa-fad456d9bdc4/volumes/kubernetes.io~secret/ceph-admin-secret
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/7575cecf-75f0-450a-9220-1f598d7c2f68/volumes/kubernetes.io~projected/rook-config-override
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/7575cecf-75f0-450a-9220-1f598d7c2f68/volumes/kubernetes.io~secret/rook-ceph-mgr-a-keyring
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/7575cecf-75f0-450a-9220-1f598d7c2f68/volumes/kubernetes.io~projected/kube-api-access-bsrc7
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/99e3d6b5-d930-48b5-bdaa-fad456d9bdc4/volumes/kubernetes.io~projected/kube-api-access-pq952
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/e1da116a-5647-4ed4-9dcb-9bb9f08b8905/volumes/kubernetes.io~projected/kube-api-access-jw52q
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/50830474c1437b8737786bf88ae887fecb172166de3c6a7cd807fc0719ec5830/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6476be91a2cdc5efad8f180045872c65ce1ebebf3eb28e8bc1ed320151ec0748/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/ef8176b2206adf4abcdf7c6ddb7a41d3a454a0d0be0f772df5fcf4a2659e127a/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/e5891a68afc62d55117a052207e1abb580f2e690e71959a500f17ff67b5eecac/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/c37d0611-34af-4eda-9a1b-bde38051eac8/volumes/kubernetes.io~projected/kube-api-access-m4ttm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/80cb382681984287df15ef591602a6872fad4772bb0f769d4084118356d5bbfc/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/7f439d3d-98fe-4328-a98f-5162ca7c2df1/volumes/kubernetes.io~projected/kube-api-access-8jkt2
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e5891a68afc62d55117a052207e1abb580f2e690e71959a500f17ff67b5eecac/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/fcfb70515215b166338c029ac602dfed90ca1e5d0b84a6d307b02b8e3667a3e3/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6476be91a2cdc5efad8f180045872c65ce1ebebf3eb28e8bc1ed320151ec0748/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/892ea79a24a11882764db51b7fb956e343ea1ebf95c1f0eb1d0c8babb239242b/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ef8176b2206adf4abcdf7c6ddb7a41d3a454a0d0be0f772df5fcf4a2659e127a/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/149597f6-b829-469d-be73-b35f5a887958/volumes/kubernetes.io~projected/kube-api-access-wttfw
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/285255dd-0563-428d-a404-f101bfb519c7/volumes/kubernetes.io~projected/kube-api-access-dmj6r
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/8b1de76eff419419889ed269c283bdc7e3068d2a95afaa856fc26b37d81df156/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/cde32feab6a5328f65b5e48d4ccf586c091a2bf1b863e29c57d454826174383b/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4564d32d2f6359e194d2b72c10640bfa85d11c687189bc133c865c69bd0da57f/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/0029dad58749b75a5442620a2f46f7bd4f100fd5e5fae1332af7e4c6b61a2480/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9984abd304fdeeac172f24e9e8b11b255e8651ba5e68cd6d8601b3caf187f07f/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/6f8790d02a6263ecc7ff292c931f7cfbae00ee08c936eb37b0465c90f51825a5/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/baabe18f-1b6d-4b56-af53-45292b52e2e0/volumes/kubernetes.io~projected/kube-api-access-bfsxl
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/ab0f322f1ee1637c1a192b00f076342377c35aeb9c500051e42ab7323318a9cf/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/4d2a2596c9bd23b69b4b47c391076ecbe4e6ce01fdb4770cfe33b9565ae407f8/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/3c85995bec02db6ecd92ac862d5d3553a7003514ce2590d0b24ed211d044dd63/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/aae5a967b2c90b0c5690692b465455de1f4fa300215b764c3785e81710ce03cc/shm
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/aa1575e5-2246-4913-a0bd-0868d8fc3169/volumes/kubernetes.io~empty-dir/keys-tmp-dir
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d195d9185f36766e106757be5d73497541cc3145310d28220b24e3d6d5052b57/rootfs
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/e96ab11b-8b58-4c79-b21b-3831c0987944/volumes/kubernetes.io~secret/argocd-repo-server-tls
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c2ef65b63df13d479aa239a9c31d2b0a9dc34189ffa4bbad28cb516d47f51237/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e6935d2b23ab107d1eba9f917cf6dcd93cbf403f5815d91401414f844a2b086c/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/41008bde1a371a3f3645323405a3461715d57135f2759104230b2157043134b5/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/8ccb65c6f1f05973f8d01d97f425bdf39c180c0d053f94920737bfb2b7d84180/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c16518df244aa8f1c61436de62513768e1d06e9ec005c68f6e3d2896657fd003/rootfs
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/3fa0be6f-2633-493c-b64b-f105a568da3e/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/aa1575e5-2246-4913-a0bd-0868d8fc3169/volumes/kubernetes.io~projected/kube-api-access-6fj6r
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/e96ab11b-8b58-4c79-b21b-3831c0987944/volumes/kubernetes.io~projected/kube-api-access-9w6cj
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ab0f322f1ee1637c1a192b00f076342377c35aeb9c500051e42ab7323318a9cf/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/5298ec68-0ec2-4ef7-9450-a4a473a17c2b/volumes/kubernetes.io~projected/kube-api-access-jmm6z
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/64a2df9ddd00c01872c46a86697cc459bc678a9d74b4f038f241db04f35a7d39/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/dd8beba8-0a7d-4933-8a58-11fb08f7c96d/volumes/kubernetes.io~projected/kube-api-access-tgp6f
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/3fa0be6f-2633-493c-b64b-f105a568da3e/volumes/kubernetes.io~projected/kube-api-access-bmn2t
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8b1de76eff419419889ed269c283bdc7e3068d2a95afaa856fc26b37d81df156/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/ae6ba475-226f-4f73-90a0-3b45aea146b2/volumes/kubernetes.io~projected/kube-api-access-gq9wr
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/9234cc36b0c107f080690f4cdb62e1ea31d1efb7ec50270f3f0850d4770d929c/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/6c211d59-8bd9-4c0d-9c24-ecf12aeee241/volumes/kubernetes.io~projected/kube-api-access-l2mq7
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4564d32d2f6359e194d2b72c10640bfa85d11c687189bc133c865c69bd0da57f/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9984abd304fdeeac172f24e9e8b11b255e8651ba5e68cd6d8601b3caf187f07f/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/9234cc36b0c107f080690f4cdb62e1ea31d1efb7ec50270f3f0850d4770d929c/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/85188860-6390-4b90-ae6b-852788b209c4/volumes/kubernetes.io~projected/kube-api-access-qk9xl
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/4291f0a4-492b-4471-a5c6-0822c8e6f2f4/volumes/kubernetes.io~projected/kube-api-access-zjdk8
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/06ed3aa4-78de-4645-8fab-eaffd1ddb929/volumes/kubernetes.io~secret/argocd-dex-server-tls
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/ca354f1a-70d8-40e6-b871-2811658ce13a/volumes/kubernetes.io~projected/clustermesh-secrets
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/ca354f1a-70d8-40e6-b871-2811658ce13a/volumes/kubernetes.io~projected/hubble-tls
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/0029dad58749b75a5442620a2f46f7bd4f100fd5e5fae1332af7e4c6b61a2480/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6f8790d02a6263ecc7ff292c931f7cfbae00ee08c936eb37b0465c90f51825a5/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/ca354f1a-70d8-40e6-b871-2811658ce13a/volumes/kubernetes.io~projected/kube-api-access-72q8n
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/79c7bafa90ea203bd14f755f5719a3a161ba2e95fe323813d9923214eb003d4b/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/fc25aab33c22b6064752ab88a5c4959250a2483dd14436a9f9b5e19b8082fa6b/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/b15916c1cc437e344879a63cbeebcbc675a903825e3952711e32ac5623456fcf/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/cde32feab6a5328f65b5e48d4ccf586c091a2bf1b863e29c57d454826174383b/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/8ccb65c6f1f05973f8d01d97f425bdf39c180c0d053f94920737bfb2b7d84180/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b15916c1cc437e344879a63cbeebcbc675a903825e3952711e32ac5623456fcf/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/41008bde1a371a3f3645323405a3461715d57135f2759104230b2157043134b5/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/aae5a967b2c90b0c5690692b465455de1f4fa300215b764c3785e81710ce03cc/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/3c85995bec02db6ecd92ac862d5d3553a7003514ce2590d0b24ed211d044dd63/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c2ef65b63df13d479aa239a9c31d2b0a9dc34189ffa4bbad28cb516d47f51237/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/4d2a2596c9bd23b69b4b47c391076ecbe4e6ce01fdb4770cfe33b9565ae407f8/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/9f78eb40-f3d0-4068-adf9-ae0a43cd7cc2/volumes/kubernetes.io~projected/kube-api-access-9k5kw
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/76201b19-c403-4e0e-88d2-89d214a89223/volumes/kubernetes.io~projected/kube-api-access-xx258
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/06ed3aa4-78de-4645-8fab-eaffd1ddb929/volumes/kubernetes.io~projected/kube-api-access-mzh8x
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/fb8b28417b619e57bd4288c96db7c68cc149abf5e0f2c2f69c05261a48615209/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/d6d2343aaed240a1a6cd81b496a37a0cfd0008866aea0fe0f284d7a48f5c0f4a/shm
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/3492591d-2d8a-4dcc-a2fe-933ad121075b/volumes/kubernetes.io~secret/argocd-dex-server-tls
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/3492591d-2d8a-4dcc-a2fe-933ad121075b/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/3492591d-2d8a-4dcc-a2fe-933ad121075b/volumes/kubernetes.io~projected/kube-api-access-l74gq
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/d33a6f94-1ac1-4d26-95b5-33fde7c598d5/volumes/kubernetes.io~projected/kube-api-access-nrtw9
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/5bb33905-0076-41b7-bf54-66941597e276/volumes/kubernetes.io~projected/kube-api-access-pfzb2
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/c1017b7d5424c36d0c68564596fcd310cead20613e2ca2f3960e0abdd78d752a/rootfs
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/47e79690-b292-4651-8f31-837d2b2a95f7/volumes/kubernetes.io~secret/rook-ceph-mds-myfs-b-keyring
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/47e79690-b292-4651-8f31-837d2b2a95f7/volumes/kubernetes.io~projected/rook-config-override
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/865a411f6e03ad49c1de406e9206c9aabb860ebdf612480cf40b3ce3e69e3cea/shm
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/1f91cf500786e2d69118e950f25d7ffa19c27ecbd0be03d52fba529dde327b96/shm
tmpfs                                                                                                               48G  4,0K   48G   1% /var/lib/kubelet/pods/817df6ee-b9a2-46cd-84f5-ec8b645f042e/volumes/kubernetes.io~secret/rook-ceph-admin-keyring
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/817df6ee-b9a2-46cd-84f5-ec8b645f042e/volumes/kubernetes.io~projected/rook-config-override
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/817df6ee-b9a2-46cd-84f5-ec8b645f042e/volumes/kubernetes.io~projected/kube-api-access-tv5ls
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/e515008c-5497-4b20-a83a-399bb8a4ee4e/volumes/kubernetes.io~projected/kube-api-access-njrn7
tmpfs                                                                                                               48G     0   48G   0% /var/lib/kubelet/pods/4b000b64-e63c-4c39-9a71-0d51f34b773a/volumes/kubernetes.io~secret/argocd-repo-server-tls
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/fda8b646-6826-4fe4-99f8-057aa57521a7/volumes/kubernetes.io~projected/kube-api-access-98vsz
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/ffd551d2-7e6d-4c06-bb74-0f5199bce09c/volumes/kubernetes.io~projected/kube-api-access-k8l5d
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/fb8b28417b619e57bd4288c96db7c68cc149abf5e0f2c2f69c05261a48615209/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/d6d2343aaed240a1a6cd81b496a37a0cfd0008866aea0fe0f284d7a48f5c0f4a/rootfs
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/868f7de5-d794-4f42-bd52-bb78ecd9b78c/volumes/kubernetes.io~projected/kube-api-access-8jt75
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/f7fa6fe5-90bb-4405-90e3-40414c70236f/volumes/kubernetes.io~projected/kube-api-access-vvdd5
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/65ddf705-4cb2-4c12-85de-ec4367f97867/volumes/kubernetes.io~projected/kube-api-access-5tqks
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/c9459f30-65a1-49f2-97e5-80e66b95d6e2/volumes/kubernetes.io~projected/kube-api-access-snszm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/98d8e4f4-8b70-4088-adaa-84b5c4ce17b3/volumes/kubernetes.io~projected/kube-api-access-vf4p4
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/4b000b64-e63c-4c39-9a71-0d51f34b773a/volumes/kubernetes.io~projected/kube-api-access-n6b2j
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/e316881d-0061-4372-8b8a-7388f8bf3d62/volumes/kubernetes.io~projected/kube-api-access-zn6qw
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/8536a409-87ba-4026-bf7e-5faab16d26f2/volumes/kubernetes.io~projected/kube-api-access-hk5tf
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/1114a23af21869ea7e2ced58b8ece5bf9443901e43a458d7e655f31185efeae3/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/31e0fdb03a02246eab2f5f2f7936078b28db466c3a8e4143c648f3d9e73bc6ed/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/c7ee5605d8a816c9e1fbe1b4aa036f6ac76c821a8cb24c029ad6c570e9e0de28/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/084b4e8647c5f83b479a7cd9912eca6c0847b43033bea8141691c2fda91e7172/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/6895641d55c5a483ec12228d2dd317519c9a3093cc295e4371da57ac43726f32/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/e291faf08fc154c1bf5229ec9197612e325911d0eb3dedfef8b0ec54cd656ff9/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/404468932b36dc4501685ae842d9dff5b32382f79148c5ba7c3b908eb4ba71f2/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/fb808a3ad38cbab9092b98626f6f47ac1b1849e9491931c843a69d779bbd75ff/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/0001e78df6ed4f39f88aac312e50b76146df687a1dd2f34a4043b892b5579361/shm
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/b3e26d45d1e033fc7e4c64e93db539009531def7ad8696c09702fef42eabee86/rootfs
shm                                                                                                                 64M     0   64M   0% /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/e3a348a120532e98bb18ce8e4876265bcd38fcb5c6aadcfe0cd2fb16b0a9ddad/shm
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/eba8568c-6a4d-4b3c-95cf-da64c0d28765/volumes/kubernetes.io~projected/kube-api-access-75g75
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/4df3d5e7-287f-419d-8a13-bce9434c6066/volumes/kubernetes.io~projected/kube-api-access-zkpmz
tmpfs                                                                                                               48G   12K   48G   1% /var/lib/kubelet/pods/47e79690-b292-4651-8f31-837d2b2a95f7/volumes/kubernetes.io~projected/kube-api-access-2244h
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/101d50f2f264e0b07b79a7a40a2b27f5e441c73f74ea0801176d63c52945dc22/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/ad5645f895467b1aa163a663a0f32a4a9399626bd54abcf703eef7952fecc76a/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/865a411f6e03ad49c1de406e9206c9aabb860ebdf612480cf40b3ce3e69e3cea/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/127519b29642bc93377761d7e1d3ce05bba7ae756d254e0b6357b549e01e6bbf/rootfs
overlay                                                                                                            733G  508G  196G  73% /run/k3s/containerd/

[MrTrustor]

When something similar happened to me, it was because k3s wasn't triggering image garbage collection soon enough (that is, deleting old container images that are not used anymore). If I remember correctly, by default, it triggers only when 80% of the storage space is used. To fix this, I added these lines to my config (/etc/rancher/k3s/config.yaml on my Ubuntu server):

kubelet-arg:
  - "image-gc-low-threshold=20"
  - "image-gc-high-threshold=50"

That way it, it doesn't trigger below 20% of storage used, and always triggers when more than 50% is used.

[brandond]

overlay 733G 508G 196G 73% /run/k3s/containerd/

@MrTrustor is correct. You're not using enough space to trigger cleanup. Default thresholds are 85% high, 80% low. This is covered in the upstream Kubelet image GC threshold documentation.