Cleanup orphaned ManagedCluster resources after setting hubAcceptsClient to false

[jaswalkiranavtar]

Describe the enhancement

Currently if spoke sets hubAcceptsClient field to false on ManagedCluster CR the hub will cleanup all rbac resources and namespace from hub.

https://github.com/open-cluster-management-io/ocm/blob/main/pkg/registration/hub/managedcluster/controller.go#L162

We will also update this controller to cleanup IAM resources created by hub for this specific spoke. The only leftover resource will be ManagedCluster. We have a lot of temporary clusters which are created and destroyed every week for testing, which will lead to a lot of orphaned resources ManagedCluster resources.

To clean them up, we are planning to add a CronJob to OCM hub. @qiujian16 also suggested to create a contoller on hub side instead of cronjob. Creating this issue to get recommendation from qiujian and othe maintainers.

[qiujian16]

@zhiweiyin318 @elgnay thought on this?

[zhiweiyin318]

We have ResourceCleanup feature gate in the registration on the Hub, the addons, manifestworks and the RBAC related with the cluster will be cleaned up after the cluster is deleted if enable this feature gate.
We also defined a flag gc-resource-list which the user can define what resources are deleted by gc controller after the cluster is deleted if enable the feature gate, and the default resources in the list are addon and manifestwork. But we did not expose it in the clustermanager yet.
Not sure if this can help in this case?
If we are going to add IAM resource cleanup, we could consider to put this part into gc controller too https://github.com/open-cluster-management-io/ocm/blob/main/pkg/registration/hub/gc/controller.go

[jaswalkiranavtar]

We want to trigger unjoin/deregister from spoke. If my understanding is correct spoke doesn't have permission to delete ManagedCluster resource on hub. So we thought, spoke can update hubAcceptsClient flag to false, which will trigger the whole cleanup.

The reasin we want to trigger from spoke is, we will have a long lived hub but many spokes can come up and go down during a week. So before we destroy spoke, we want to update the flag to false, so that all spoke specific resources are cleaned up from hub.

[zhiweiyin318]

I think gc controller is the place where we do resources cleanup, we can put the resources cleanup code to there.
Currently deleting cluster can trigger resources cleanup in gc controller, I think it's ok to add false hubAcceptsClient in gc controller.

[jaswalkiranavtar]

Ok thanks, will look at it.

[jaswalkiranavtar]

@zhiweiyin318 I took a look at gccontrolller. Looks like it is meant to delete resources like ManifestWork, rbac resources etc.

However my question was to delete ManagedCluster resource itself when everything else gets deleted. That is the reason I proposed to add a CronJob or we can add new controoller to delete it.