Ecosystem installation method to be compatible with blue green #2508
Description
Background
I'm raising more as a placeholder and as focus for conversation, this may be long-running, it may be something more the customer has to do?
Galasa documentation steers the user firmly towards using helm to install and upgrade the ecosystem. However, its a very risky form of upgrading Galasa for customer who want to strive for 100% uptime of their ecosystem. It was quite some time ago now, but we helm upgraded to a newer version and the ecosystem completely broke and it was not a straight forward process to bring the ecosystem back, as we were jumping several versions up and committed many changes in order to make the jump, which needed rolling back.
From that point, we implemented a Blue green deployment strategy, that has been working excellently since. We essentially have two separate ecosystems installed in parallel, and we've copied all galasa ingresses to have our own "constant" parent ingresses. We then update these ingresses to point to either one of the ecosystems, and we're able to switch between them.
There are downsides to it though when we switch:
- The RAS is unique for each ecosystem, it is not duplicated, moved, or copied. Therefore we lose all historic RAS runs, tokens, user configs, secrets etc when moving from the old production to new production. This makes it a non-seamless experience for our testers as they have to generate a new token etc.
- We have our own external DSS, which only production should point to. Therefore, when an ecosystem "becomes" production, it must switch from pointing to an internal DSS, to the external one.
- ....possibly more to come
Once we have a solution that makes blue green easier, and can be automated much easier, we can then potentially start looking at a Continuous Deployment strategy for our ecosystem install, where we actually start using the development stream of Galasa, which will be safer as we can test it first as part of the CD pipeline.
Tasks
- <task>