Figure out how to share inputs for cluster install driven by ACM

[larsks]

How can we share or automate the creation of inputs to ACM for driving a cluster install?

[larsks]

There are a few things that complicate the process of (a) managing the input data for a cluster installation and (b) creating something that can be shared without revealing All The Secrets. I'd like to take a look at (a) first:

We'd like to manage our cluster install configuration with a GitOps-style workflow. In the short term, that means we'd like to store the input manifests git repository, and initialize a cluster install via kustomize build | oc apply -f-. The major sticking point in this process is the way that install-config.yaml is managed. While all the information necessary to generate that file is available in the ClusterDeployment resource, the contents of install-config.yaml have to be provided as a secret. This means there are two places where we need to (a) list hosts, (b) provide libvirt credentials, etc. We either need to enter everything twice, or hack together some sort of templating system to generate the manifests.

[HumairAK]

Not entirely sure if I understand what's going on here -- but could writing a simple exec plugin for kustomize help with this? one that maybe we could share with others who want to do the same thing?

[tumido]

That's a neat idea! Like ACM generator plugin.. that would work nicely. @HumairAK do you know if it's possible to compose kustomize plugins - like to use ksops generator first and then some other plugin on the same resource? Because we probably want the resource to be encrypted in git and then decrypt and generate configs from it...

[HumairAK]

No clue, we would need to look into it.

[larsks]

Ideally I would like to see this fixed in ACM itself :).

I think we could work around this with an exec plugin: we would probably need something that takes a single input document and uses it to generate both the ClusterDeployment resource and the install-config.yaml secret. If there's no "official" support for composing plugins, we can just call the ksops exec plugin ourselves, right? Or just sops -d, I guess.

[larsks]

So, just thinking about this, if we want to DRY everything out, we'll need to read as input and emit in the appropriate places...

• The pull secret (which is used both in the provider connection and in install-config.yaml)
• The baremetal host information (which is used in the ClusterDeployment resource and in install-config.yaml)
• The ssh private key (used in the provider connection and in a secret referenced by the ClusterDeployment...twice)
• The ssh public key (used in the provider connection)
• The host key for the provisioner (used in the ClusterDeployment and in the provider connection

I think our deployment would end up looking like:

• a kustomization.yaml file with maybe (a) a namespace resource and (b) a generator
• a generator configuration that contains all the necessary input data

And that's it. And the entire configuration would then be generated by the kustomize plugin.

[tumido]

Consolidating this into an RFE for our toolbox operate-first/toolbox#24