Recommended implementation for "optional groups of dependent resources"

[lbroudoux]

Hey there!

I'm wondering what could be the best implementation for the reconciliation of "groups of dependent resources".

Typically, I have a CRD with different blocks representing different application modules. Each block may be enabled/disabled and if enabled will typically lead to the management of many Kubernetes resources (let's say a Deployment, a Service, a ConfigMap, and a Service - all of them to be reconciled together)

I was first thinking of a BulkDependentResource implementation, but it seems like it's only for the same kind of resources. I'd prefer handling a reconciliation grouping approach based on the application modules rather than on technical resources.

So at the moment, I think about implementing a dedicated ModuleXDependentResource for each module and having:

public class ModuleXDependentResource extends AbstractDependentResource<ModuleXResources, PrimaryCR>
      implements Creator<ModuleXResources, PrimaryCR>, Updater<ModuleXResources, PrimaryCR>, Deleter<PrimaryCR> {
}

Thus I have to define an associated ModuleXResources class that would act as my secondary resources handler (basic Kube objects) that must be reconciled together. From what I've seen so far, this would imply getting some work on the public Optional<R> getSecondaryResource(P primary, Context<P> context) method implementation.

Would you say it's the better/recommended way of doing things?

I've also considered some other options:

• Having as many Reconcilers for the same CR as I have different application modules/blocks in the CRD. However I'm afraid, this breaks the principle of "One controller/operator per CRD" and led to race conditions/sync issues when it comes to updating the CR Status,
• Implementing this using Workflows but it looks like I have no real dependency between the reconciliation logic of the different modules. The workflow approach looks then a bit overkill for this.

Thanks for reading! Any suggestion is greatly appreciated! 😉

[metacosm]

My initial thinking would be to use workflows with the same reconcile pre-condition for all the related dependents so that their reconciliation would only be triggered at once.

[lbroudoux]

Thanks @metacosm! I've gone through the Workflows documentation to check this and it looks like a match!

I've got a concern though: with this approach, it looks like my PrimaryReconciler has to be aware (via the @Dependent annotations) of all the "low-level" dependencies of my application modules.
I would prefer having an extra level of indirection to not break the encapsulation:

PrimaryReconciler --> n ModuleReconciliers --> m KubernetesDependentResources

Would you have an idea on the best way to implement this?

[lbroudoux]

Looking at Standalone Workflows and it might be the way to go...
Having different ModuleReconciers taking care of instantiating different Workflows and having the PrimaryReconciler act as a composite orchestrator and update the status depending on the result of different workflows. Does it make sense?

[metacosm]

Personally, I prefer to make things more explicit with the annotations but I can understand that you might want to "encapsulate" things more writing the workflow "manually".

Another alternative would be to create secondary CRs that would be created by the primary reconciler and each of these would be handled by their own reconcilers. These CRs would be "invisible" to users (I mean they'd be able to see them if they really wanted to) but they wouldn't be part of the official API for your reconciler. That would implement the P -> N -> M pattern that you showed above.

[csviri]

Not sure if I follow the Module reconciler, will this have a separate custom resource?

If some of your "sub groups" are reusable, I would propose to create a separate custom resource for them. For example also in kubernetes the deployment controller creates manages replica set(s), not pods directly, therefore reusing a subresource.

If not resuable, that is a questions (event then using some meaningful subresources would not be a sin IMO ) . Yes you can do it with standalone mode, with multiple workflows called explicitly, just this might become more complex regarding informers, to manage (those are not dynamically registered).

But having it simply explicit in a single workflow might be the simplest way to do.

[lbroudoux]

Thanks for all the suggestions. Currently trying an implementation using Standalone Workflows with reconcile precondition and looks like it works since then. Now struggling with resource discriminator as I start to have multiple dependent resources of same type but that's another story 😉

Will keep you posted here to let you if OK or if I found some caveats to this approach.