Update Release Workflow

[YuryHrytsuk]

• Use hotfix branches and let people merge whatever in main branch
• Tag / Pin Release CIs (if there was a commit in main branch that breaks CI, it shall not affect STAG / PROD Release CIs)
• Tag / Pin Test CIs (test against pinned configuration, detach from Release CIs in main branch)
• Tag ansible playbooks (infra repo)
• Make sure webserver always runs on latest (hotfix) version
• make release-prod-all version=1.80.4-post0 force-retag=true
• Add mandatory db backup step before releasing simcore
• Use separate tags for simcore and ops / config (so we can release a newer configuration patch version while keeping the same simcore tag[avoid simcore services restart because image tag changed on simcore])
• Link commits from ops-environments repository in release issue (trace when commit gets to stag / prod)
• Grafana dashboards (e.g. in shared folder) should not break tests
• Create tags by specifying Commit SHA (currently it tags current commit) --> it reduces chance of tagging wrong commit (e.g. when person forgets to pull latest changes)

---

How to update broken e2e prod test?

• wait for changes to reach prod (after prod release)
• temporarily use latest changes and revert it after release

Changes to release workflow:

• tag osparc-infra
• follow hotfix approach of simcore (allow merging to "main" branch)
• properly pin ci releases

Tasks

• keep history of tags (when release tag is moved to another commit, we should somehow preserve "history" of what commit was tagged before)