|
| 1 | += SDD 0029 - Local Commodore Development Environment |
| 2 | + |
| 3 | +:sdd_author: Simon Gerber |
| 4 | +:sdd_owner: VSHN Tech Alignment WG |
| 5 | +:sdd_reviewers: Reviewer names |
| 6 | +:sdd_date: 2023-XX-XX |
| 7 | +:sdd_status: draft |
| 8 | +include::partial$meta-info-table.adoc[] |
| 9 | + |
| 10 | +[NOTE] |
| 11 | +.Summary |
| 12 | +==== |
| 13 | +This SDD outlines a local development environment which is easy to use, quick to bootstrap and which supports local development backed by a full cluster catalog. |
| 14 | +==== |
| 15 | + |
| 16 | +== Motivation |
| 17 | + |
| 18 | +While we have a development environment for working on a single component in isolation (`commodore component compile`), we don't currently have a simple development environment for working on complex configurations which consist of multiple components. |
| 19 | + |
| 20 | +Additionally, the currently available `commodore component compile` hits limits when working on more complex components, for example when advanced ArgoCD features are required for correct installation of the software managed by a component. |
| 21 | + |
| 22 | +Finally, with a more full-fledged local development environment we also enable ourselves to provide a better getting started experience with Commodore and we can explore new avenues for integration testing in component CI pipelines. |
| 23 | + |
| 24 | +=== Goals |
| 25 | + |
| 26 | +We want the local development environment to |
| 27 | + |
| 28 | +* Support quick edit-compile-deploy cycles for single components |
| 29 | +* Enable users to test single- or multi-component configurations in a local Kubernetes cluster (e.g. `kind`, or `k3d`) without having to register the cluster in a Lieutenant instance |
| 30 | +* ... |
| 31 | + |
| 32 | +=== Non-Goals |
| 33 | + |
| 34 | +* TBD |
| 35 | + |
| 36 | +== User Stories |
| 37 | + |
| 38 | +=== As a *component developer* I want *quick and easy edit-compile-deploy cycles* so that *I can quickly iterate on my changes* |
| 39 | + |
| 40 | +==== Background |
| 41 | + |
| 42 | +We've seen Helm charts which make heavy use of install hooks to perform some initialization steps in the right order, which aren't supported at all with the currently available tools. |
| 43 | +We also develop multiple components which use ArgoCD's sync job or sync wave mechanisms to orchestrate more complex installations or upgrades. |
| 44 | +Some changes to these components can't be tested without having a fully Syn-managed cluster available. |
| 45 | + |
| 46 | +Depending on how many components are installed on a shared dev cluster, compile times for the dev cluster's cluster catalog may be in the multiple minutes range, which isn't really acceptable for iterating on changes. |
| 47 | + |
| 48 | +==== With the current tooling |
| 49 | + |
| 50 | +Jane tests components by compiling them locally with `commodore component compile` and the applies the resulting manifests on her kind cluster with `kubectl apply -Rf compiled/`. |
| 51 | +However, because the component she's trying to test installs a Helm chart which uses Helm hooks to perform some in-cluster configuration that needs to happen in a specific order, she isn't able to get the software installed by the Helm chart to work properly. |
| 52 | +At this point, Jane may start chasing red herrings, such as "The installation works fine with `helm install`, so something in Commodore's Helm rendering must be wrong". |
| 53 | +Alternatively, if she's seen a similar issue previously, she'll either test the component on a shared dev cluster or she'll register her local cluster in a Lieutenant instance, since she's aware that ArgoCD will execute helm hooks in the correct order. |
| 54 | + |
| 55 | +With either of those approaches, the compile-deploy cycle for Jane has jumped from a few seconds to tens of seconds since she now needs a full `commodore catalog compile` followed by either manually syncing her component's App in ArgoCD or waiting for ArgoCD's sync cycle. |
| 56 | +If she's decided to test her component on the shared dev cluster because she doesn't want to spend upwards of 15 minutes to get her local cluster Syn-managed, the compile-deploy cycle may even take multiple minutes depending on how many other components are installed on the shared cluster. |
| 57 | + |
| 58 | +==== With the proposed tooling |
| 59 | + |
| 60 | +Jane tests components locally by running `commodore component install` while having her local kind cluster selected in her `kubectl` config. |
| 61 | +After a few seconds, she sees her changes getting applied in her local kind cluster. |
| 62 | + |
| 63 | +=== As a *component maintainer* I want *integration tests in the component CI pipeline* so that *I can see issues with dependency updates when looking at PRs* |
| 64 | + |
| 65 | +===== With the current tooling |
| 66 | + |
| 67 | +We've not engineered any support for running integration tests (i.e. installing components in a kind cluster) in CI pipelines. |
| 68 | +Therefore, John needs to setup a local kind cluster, make it Syn-managed, and install the dependency update branch of the component he's reviewing by adding the component and all of its dependencies to the newly created Syn cluster config for his local cluster. |
| 69 | +Alternatively, John can update the component to the dependency branch on the shared dev cluster without any local tests, and accept that he may break the dev environment with a so-far completely untested dependency update. |
| 70 | + |
| 71 | + |
| 72 | +==== With the proposed tooling |
| 73 | + |
| 74 | +John can create a local kind cluster, and install the dependency update branch of the component into his local cluster by running `commodore component install`. |
| 75 | + |
| 76 | +=== As a *system engineer* I want to *locally test configuration changes touching multiple components* so that *I can confidently say that the changes are safe to roll out* |
| 77 | + |
| 78 | +==== With the current tooling |
| 79 | + |
| 80 | +Project Syn currently has no support for cleanly testing configuration changes which touch multiple components without compiling a full cluster catalog. |
| 81 | +Therefore, Jane usually tests such changes on the dev cluster, accepting that she'll wait multiple minutes for each iteration of her change before she sees the catalog diff. |
| 82 | +If she wants to actually test iterations of the configuration change on the cluster, she has to push or merge each iteration of the change into the master branch of the dev cluster's tenant repo, and then compile and push the catalog changes. |
| 83 | + |
| 84 | +==== With the proposed tooling |
| 85 | + |
| 86 | +Jane sets up a local kind cluster and ensures her current `kubectl` context points to it. |
| 87 | +She then defines a local cluster spec which installs all the components affected by her configuration change. |
| 88 | +She copies or references the current configuration from the regular config hierarchy and runs `commodore catalog install cluster-spec.yml`, to render a local catalog and install it into her local kind cluster. |
| 89 | + |
| 90 | +== Design Proposal |
| 91 | + |
| 92 | +=== Design ideas |
| 93 | + |
| 94 | +==== Local cluster spec in Reclass |
| 95 | + |
| 96 | +* Have a local "cluster spec" Reclass class which is used in place of `global.commodore` in the Kapitan targets and which specifies everything that's required to render a local catalog |
| 97 | +* Render catalog into a local Git repo -> TODO: how to handle secret references? |
| 98 | +* Engineer a Steward variant which either spins up a kind/k3d cluster and bootstraps it from the local catalog or which expects a local kind/k3d cluster to be available. |
| 99 | +* Easy to support multi-component testing, aka `commodore catalog install cluster-spec.yml` in the user stories. |
| 100 | +** We can engineer optional support for referencing remote global or tenant repos in the local cluster spec. |
| 101 | +* edit-compile-deploy cycle may become slower, especially if cluster spec gets big or references a lot of external config repos. |
| 102 | +* A single command `commodore component install` may need some extra inputs |
| 103 | + |
| 104 | +==== Run `commodore component compile` in ArgoCD |
| 105 | + |
| 106 | +* Have tooling to bootstrap an ArgoCD which is configured with an App pointing to the working directory of the developer and runs `commodore component compile` during apply |
| 107 | +* `commodore component install` in the user stories is basically this |
| 108 | +* Unclear how we can offer local multi-component testing ("local catalog") with this approach |
| 109 | + |
| 110 | + |
| 111 | +==== Fake Lieutenant API managed by Commodore |
| 112 | + |
| 113 | +* Commodore provides a fake Lieutenant API based on additional flags for `catalog compile` |
| 114 | +* Just having the fake Lieutenant API doesn't address the remaining aspect of getting a local cluster bootstrapped with ArgoCD linked to a repo on the developer's laptop. |
| 115 | + |
| 116 | +=== Implementation Details/Notes/Constraints [optional] |
| 117 | + |
| 118 | +=== Risks and Mitigations [optional] |
| 119 | + |
| 120 | +== Drawbacks [optional] |
| 121 | + |
| 122 | +== Alternatives |
| 123 | + |
| 124 | +== References |
0 commit comments