Resizing volumes leaves backups behiend

[robinAwallace]

Hello,

I have noticed that when resizing volumes the csi-driver creates a backup of the volume. I also see in the resizer function that this backup should be deleted. But the backup is never deleted.

The resize for me looks successfully, as the size is increased correctly. But I can see in the logs that something failed just before successfully resizing the volume.

I0826 09:02:41.593567       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"opensearch-system", Name:"opensearch-master-opensearch-master-0", UID:"54ae8ec5-c1bd-416a-bcc3-c2f4c5453cff", APIVersion:"v1", ResourceVersion:"38745", FieldPath:""}): type: 'Normal' reason: 'Resizing' External resizer is resizing volume pvc-54ae8ec5-c1bd-416a-bcc3-c2f4c5453cff
I0826 09:02:42.243560       1 controller.go:483] Resize volume succeeded for volume "pvc-54ae8ec5-c1bd-416a-bcc3-c2f4c5453cff", start to update PV's capacity
I0826 09:02:42.243574       1 controller.go:590] Resize volume succeeded for volume "pvc-54ae8ec5-c1bd-416a-bcc3-c2f4c5453cff", start to update PV's capacity
I0826 09:02:42.261758       1 controller.go:489] Update capacity of PV "pvc-54ae8ec5-c1bd-416a-bcc3-c2f4c5453cff" to 120Gi succeeded

The backup I see in the hub after resizing:

I expect after a successful resize that all backups are deleted. As this will otherwise be an additional costs.

Version Im using:

image: registry.k8s.io/sig-storage/csi-resizer:v1.4.0
image: ghcr.io/upcloudltd/upcloud-csi:v1.1.0

[thevilledev]

Hey Robin! Thanks for reaching out. We'll take a look.

Note: deleted a spam comment that contained a fishy link

[peknur]

Hi @robinAwallace !

We currently take storage backup before every resize operation and it's also default behavior of our storage API.
Exception to this rule in CSI driver is when creating new volume from other source volume, backup is deleted automatically because source volume still contains that same data.

Managing these backups when using Kubernetes can be bit tricky, because Kubernetes doesn't directly list all the backups reported by ListSnapshots function.

One way of doing it is scheduling upctl to clean up unused backups.

[robinAwallace]

Hi Pekka,

Unfortunate that ListSnapshots is having issues to list all backups. I would say that using upctl would be a less optimal way of solving this issue 🤔

Is a better way of keeping track of backups from resizes of a volume/pvc being planed? Because that would mean less of unforeseen costs and a smoother experience on our part. 🙂 🙏

[peknur]

We have had some initial discussion about that, but we haven't decided anything concrete yet. That would basically require new controller to sync backup metadata between storage backend and cluster.

Meanwhile, I think upctl is probably easiest way to handle this, you could use something like this to get those backup UUIDs

$ upctl storage list --backup -ojson | jq -r '.storages[] | select( .title == "Resize Backup" and .labels[].key == "csi_cluster_id" and .labels[].value == $CLUSTER_ID) | .uuid'

$CLUSTER_ID should be replaced with actual cluster ID.

Note that I'm not that proficient with jq, but that should get you started if needed.

[thevilledev]

Just to wrap up what we discussed with @robinAwallace earlier today. We can implement a separate flag that enforces the CSI driver to remove the dangling storage backup if (and only if) the resize operation was a success. It would be an opt-in feature, as our API documentation for storage resize considers keeping it for longer:

It is advisable to keep the backup until you have ensured that everything works after the resize.

In the meanwhile, that upctl one-liner mentioned above should act as a workaround

We also have some ideas of incorporating object imports for resources created outside of Kubernetes, lets say manual backups as an example, but that will be part of a separate project.

[robinAwallace]

@thevilledev thanks for the summary 👍
@peknur for now we went with your suggested approach, cleaning up the backups with upctl. 🙂