Convert CircleCI config to Github Actions (#8147)

* convert circle ci config to github actions

* use new docker compose syntax

* only run github actions on PRs and master branch

* CI

* fix yarn lockfile

* CI

* CI

* Ci

* CI

* CI

* sudo!

* sudo

* bump ubuntu version

* CI

* use different package names for ubuntu

* CI

* enable ssh debug

* CI

* CI

* add sbt action

* ci

* ci

* CI

* CI

* CI

* CI

* enable ssh debugging

* set postgres url

* CI

* CI

* ci

* ci

* ci

* CI

* ci

* ci

* ci

* ci

* ci

* Update .github/workflows/build_test_deploy.yml

Co-authored-by: robert-oleynik <[email protected]>

* Update .github/workflows/build_test_deploy.yml

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

* Update .github/workflows/build_test_deploy.yml

Co-authored-by: robert-oleynik <[email protected]>

* fix directory permissions

* fix GA composite action path

* add postgres env to e2e tests

* CI

* Replace postgres hostname with localhost:5432

* change port for CI postgres container

* log docker containers

* fix postgres ports

* mkdir Organization_X

* unzip test dataset

* fix unzip order

* add cache for sbt and yarn

* update hash files expression

* use cache of setup-node

* remove node caching

* rework build_test_deploy step

* fix indentation

* fix action uses

* fix github actions expression

* enable corepack

* prepare e2e tests

* add tmate sessions

* cache node modules

* reorder steps and set explicit context

* fix quoting

* fix quoting

* fix yarn.lock path

* fix docker flags

* reorder backend tests

* ci run as non root

* remove unnecessary cache

* fix node_modules caching

* add dbtool asserts and fix permissions

* add postgres user env

* move environment to job declaration

* use postgres env from global scope

* fix user permissions

* fix postgres url

* fix path and reorder steps

* fix binary data path

* move e2e test to separate stage

* run e2e tests without retry

* merge all pipeline build and test stages and disable caches

* add <cstdint> header

* fix docker tags

* fix docker tag for compose

* add separate e2e-test

* fix docker tag reference

* add backend tests

* add system dependencies to backend tests

* add tmate debugging sessions

* remove backend-test job

* add node to e2e tests

* add missing environment

* update checkout action to v4

* add tmate debugging session

* merge end-to-end test and build job

* fix DOCKER_TAG

* replace build job with one-to-one port of circle ci

* fix build_test_deploy step

* use docker compose instead of docker-compose

* add tmate debugging session

* fix CI

* replace not-on-master script with checks

* replace string literal for branch checks

* use native sbt, nod and yarn and upadte env

* setup postgres env

* remove slack notification

* install system dependencies

* fix typo

* generate WK_VERSION from github.run_number

* add services and docker action

* fix typos

* fix namespace name

* fix typo

* fix typo

* remove conflicting port

* run e2e tests without docker compose

* ci: pg_isready with correct user

* bucketScanner.cpp add missing dependency

* fix pg_isready command

* use docker compose instead of GitHub services

* fossildb open ports

* fix fossildb startup command

* fix e2e-test command

* use smoke test action

* add missing redis port

* fix smoke test url

* cleanup old commands

* add custom retry action

* add separate backend test step

* add pull policy

* add extra debugging step

* remove backend-tests

* fix docker image export

* remove double frontend tests

* create separate backend test job

* backend tests install system dependencies

* backend tests start postgres

* fix typo

* fix docker driver

* backend tests install node

* prevent backend tests from running on master

* fix typo

* remove node_modules caches

* cleanup jobs

* remove cache

* fix retry e2e test command

* remove coverage step

* move Dockerfiles to .github directory

* setup slack notifications and fix branch enumeration

* fix permissions

* remove debug options

* fix fossildb command

* fix dockerfile naming

* apply suggestions and fix errors

* add missing env

* add cerficate env

* rename secret

* enable slack notifications only on master

* cleanup .github

* cleanup circle ci and old dockerfiles

* fix normalized branch name

* Apply suggestions from code review

Co-authored-by: Norman Rzepka <[email protected]>

* fix branch conditions and environment variables

* add debugging session

* use GITHUB_HEAD_REF

* remove tmate session

* update changelog

* fix CI_BUILD_NUM env

* Update CHANGELOG.unreleased.md

---------

Co-authored-by: robert-oleynik <[email protected]>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Robert Oleynik <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>

Author: 4 people

.circleci/config.yml

@@ -15,7 +15,7 @@ VOLUME /webknossos-datastore/binaryData /tmp
 COPY target/universal/stage .
 
 RUN chown -R webknossos . \
-  && chmod go+x bin/webknossos-datastore \
+  && chmod +x bin/webknossos-datastore \
   && chmod go+w .
 
 RUN echo '#!/bin/bash\numask 002\nbin/webknossos-datastore "$@"\n' > /docker-entrypoint.sh \

.circleci/not-on-master.sh

@@ -16,7 +16,7 @@ VOLUME /webknossos-tracingstore/tracingData /tmp
 COPY target/universal/stage .
 
 RUN chown -R webknossos . \
-  && chmod go+x bin/webknossos-tracingstore \
+  && chmod +x bin/webknossos-tracingstore \
   && chmod go+w .
 
 RUN echo '#!/bin/bash\numask 002\nbin/webknossos-tracingstore "$@"\n' > /docker-entrypoint.sh \
@@ -31,4 +31,4 @@ HEALTHCHECK \
 EXPOSE 9050
 
 ENTRYPOINT ["/docker-entrypoint.sh"]
-CMD ["-J-Xmx20G", "-J-Xms1G", "-Dconfig.file=conf/standalone-tracingstore.conf", "-Dlogger.file=conf/logback-docker.xml", "-Dlogback.configurationFile=conf/logback-docker.xml", "-Dhttp.port=9090", "-Dhttp.address=0.0.0.0"]
+CMD ["-J-Xmx20G", "-J-Xms1G", "-Dconfig.file=conf/standalone-tracingstore.conf", "-Dlogger.file=conf/logback-docker.xml", "-Dlogback.configurationFile=conf/logback-docker.xml", "-Dhttp.port=9050", "-Dhttp.address=0.0.0.0"]

.circleci/slack-notification.sh

@@ -8,13 +8,15 @@ RUN curl -sL "https://deb.nodesource.com/setup_${VERSION_NODE}" | bash - \
 RUN mkdir -p /webknossos
 WORKDIR /webknossos
 
+# Copy compiled Scala output from a previous build step, e.g. output of the Docker-dev image
 COPY target/universal/stage .
 
 RUN addgroup --system --gid 999 webknossos \
   && adduser --system --uid 999 --ingroup webknossos webknossos \
   && mkdir disk \
   && chown -R webknossos . \
-  && chmod go+x bin/webknossos \
+  && chmod +x bin/webknossos \
+  && chmod +x tools/postgres/dbtool.js \
   && chmod go+w .
 
 RUN echo '#!/bin/bash\numask 002\nbin/webknossos "$@"\n' > /docker-entrypoint.sh \

webknossos-datastore/Dockerfile renamed to .github/Dockerfile.datastore

@@ -0,0 +1,24 @@
+name: "Shell Retry"
+description: "Runs a small shell script with retries"
+inputs:
+  run:
+    description: "Command to execute."
+    required: true
+  retries:
+    description: "Maximum number of retries until failed"
+    required: true
+  retry_delay_seconds:
+    description: "Amount of time to sleep between each retry"
+    default: 0
+runs:
+  using: 'composite'
+  steps:
+    - shell: bash
+      run: |
+        for i in {1..${{ inputs.retries }}}; do
+          bash -c "$INPUT_COMMAND" && exit 0
+          sleep "${{ inputs.retry_delay_seconds }}"
+        done
+        exit 1
+      env:
+        INPUT_COMMAND: ${{ inputs.run }}