Integrates GitLab Runner and updates documentation (#1115 and #1082)

  - Adds integrated gitlab runner to all installation scenarios.
    It is now possible to have one runner that can run all
    the jobs sent by DevOps requests.
  - Updates DevOps documentation and ports corrections from
    documentation of release v0.6.0

---------
Co-authored-by: aryanpingle <[email protected]>

.codeclimate.yml

@@ -47,3 +47,8 @@ exclude_patterns:
   - "client/config/"
   - "deploy/config/client/env*.js"
   - "files/"
+  - "CODE_OF_CONDUCT.md"
+  - "CONTRIBUTING.md"
+  - "LICENSE.md"
+  - ".github/ISSUE_TEMPLATE/"
+  - "docs/developer/conduct.md"

CITATION.cff

@@ -1,34 +1,38 @@
 cff-version: 1.2.0
-message: "If you use this software, please cite it as below."
-authors:
-- family-names: "Talasila"
-  given-names: "Prasad"
-  orcid: "https://orcid.org/0000-0002-8973-2640"
 title: "Digital Twin as a Service"
-version: 0.3.0
-date-released: 2023-11-08
+version: 0.6.0
+date-released: 2024-11-11
 url: "https://github.com/into-cps-association/DTaaS"
+message: Please cite the following works when using this software.
 preferred-citation:
-  conference: "The 2023 IEEE International Conference on Digital Twin"
   authors:
-  - family-names: "Talasila"
-    given-names: "Prasad"
-    orcid: "https://orcid.org/0000-0002-8973-2640"
-  - family-names: "Gomes"
-    given-names: "Cláudio"
-    orcid: "https://orcid.org/0000-0000-0000-0000"
-  - family-names: "Mikkelsen"
-    given-names: "Peter Høgh"
-    orcid: "https://orcid.org/0000-0003-2321-758X"
-  - family-names: "Gil"
-    given-names: "Santiago"
-    orcid: "https://orcid.org/0000-0002-1789-531X"
-  - family-names: "Kamburjan"
-    given-names: "Eduard"
-    orcid: "https://orcid.org/0000-0002-0996-2543"
-  - family-names: "Larsen"
-    given-names: "Peter Gorm"
-    orcid: "https://orcid.org/0000-0002-4589-1500"
-  title: "Digital Twin as a Service (DTaaS): A Platform for Digital Twin Developers and Users"
-  month: 8
-  year: 2023
+    - family-names: Talasila
+      given-names: Prasad
+    - family-names: Gomes
+      given-names: Cláudio
+    - family-names: Vosteen
+      given-names: Lars B
+    - family-names: Iven
+      given-names: Hannes
+    - family-names: Leucker
+      given-names: Martin
+    - family-names: Gil
+      given-names: Santiago
+    - family-names: Mikkelsen
+      given-names: Peter H
+    - family-names: Kamburjan
+      given-names: Eduard
+    - family-names: Larsen
+      given-names: Peter G
+  doi: 10.1177/00375497241298653
+  identifiers:
+    - type: doi
+      value: 10.1177/00375497241298653
+  title: Composable digital twins on Digital Twin as a Service platform
+  date-published: 2024-01-01
+  year: 2024
+  journal: SIMULATION
+  publisher:
+    name: 'SAGE Publications Sage UK: London, England'
+  start: '00375497241298653'
+  type: article

README.md

@@ -39,14 +39,15 @@ if there is a suggestion to improve the software.
 If you find this repo useful for your research, please consider citing our paper:
 
 ```bibtex
-@INPROCEEDINGS{talasila2023dtaas,
-  author={Talasila, Prasad and Gomes, Cláudio and Mikkelsen, Peter Høgh and Arboleda, Santiago Gil and Kamburjan, Eduard and Larsen, Peter Gorm},
-  booktitle={2023 IEEE Smart World Congress (SWC)},
-  title={Digital Twin as a Service (DTaaS): A Platform for Digital Twin Developers and Users}, 
-  year={2023},
-  pages={1-8},
-  keywords={digital twins;physical twin;automation;life cycle;composition},
-  doi={10.1109/SWC57546.2023.10448890}}
+@article{talasila2024composable,
+author = {Prasad Talasila and Cl{\'a}udio Gomes and Lars B Vosteen and Hannes Iven and Martin Leucker and Santiago Gil and Peter H Mikkelsen and Eduard Kamburjan and Peter G Larsen},
+title ={Composable digital twins on Digital Twin as a Service platform},
+journal = {SIMULATION},
+pages = {00375497241298653},
+year={2024},
+doi = {10.1177/00375497241298653},
+publisher = {SAGE Publications Sage UK: London, England}
+}
 ```
 
 ## :hammer_and_wrench: Development Setup

SECURITY.md

@@ -6,7 +6,8 @@ We are currently providing security updates for:
 
 | Version | Supported          |
 | ------- | ------------------ |
-| 0.5.0   | :white_check_mark: |
+| 0.6.0   | :white_check_mark: |
+| 0.5.0   | :x:                |
 | 0.4.0   | :x:                |
 | 0.3.0   | :x:                |
 | 0.2.0   | :x:                |

deploy/services/gitlab/.env

@@ -1,2 +1,2 @@
-GITLAB_HOME='/Users/<Username>/DTaaS/deploy/services/gitlab'
+DTAAS_DIR='/Users/<username>/DTaaS'
 SERVER_DNS='foo.com'

deploy/services/gitlab/INTEGRATION.md

@@ -66,7 +66,7 @@ the following files:
    _deploy/config/client/env.local.js_.
 1. _deploy/docker/.env.local_ Add localpath and username.
 
-If the DTaaS application is hosted at <https://localhost>, then configure
+If the DTaaS application is hosted at <https://foo.com/>, then configure
 the following files:
 
 1. **DTaaS Client Authorization** token in

deploy/services/gitlab/README.md

@@ -29,7 +29,7 @@ Edit the `.env` file available in this directory to contain the following variab
 
 | Variable    | Example Value                                | Explanation                                                                                                                  |
 | :---------- | :------------------------------------------- | :--------------------------------------------------------------------------------------------------------------------------- |
-| GITLAB_HOME | '/home/Desktop/DTaaS/deploy/services/gitlab' | Full path to the DTaaS gitlab directory. This is an absolute path with no trailing slash.                                    |
+| DTAAS_DIR | '/Users/<username>/DTaaS' | Full path to the DTaaS directory. This is an absolute path with no trailing slash.                                    |
 | SERVER_DNS  | either `foo.com` or `localhost`                               | The server DNS, if you are deploying with a dedicated server. Remember not use _http(s)_ at the beginning of the DNS string. |
 
 **NOTE**: The DTaaS client uses the `react-oidc-context` node package, which
@@ -114,5 +114,7 @@ for further help.
 ## Pending Tasks
 
 This README helps with installation of Gitlab along side DTaaS application.
-But the OAuth2 integration between Gitlab and DTaaS will still be still pending.
-Follow the [INTEGRATION](./INTEGRATION.md) guide to setup the Gitlab integration.
+But the OAuth2 integration between Gitlab and DTaaS will still be pending.
+Follow the [integration guide](./INTEGRATION.md) and
+[runner setup guide](../runner/GITLAB-RUNNER.md) to setup the Gitlab
+integration.

deploy/services/gitlab/compose.gitlab.yml

@@ -15,9 +15,9 @@ services:
         nginx['redirect_http_to_https'] = false
         letsencrypt['enable'] = false
     volumes:
-      - '${GITLAB_HOME}/config:/etc/gitlab'
-      - '${GITLAB_HOME}/logs:/var/log/gitlab'
-      - '${GITLAB_HOME}/data:/var/opt/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/config:/etc/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/logs:/var/log/gitlab'
+      - '${DTAAS_DIR}/deploy/services/gitlab/data:/var/opt/gitlab'
     shm_size: '256m'
     labels:
       - "traefik.enable=true"

deploy/services/runner/GITLAB-RUNNER.md

@@ -0,0 +1,150 @@
+# GitLab Runner Integration
+
+This document outlines the steps needed to create a Docker container named
+`gitlab-runner` which will contain a single runner that will be responsible for
+the execution of Digital Twins. There are two installation scenarios:
+
+1. __Localhost Installation__ - You are using the integrated runner locally with
+   a GitLab instance hosted at `https://localhost/gitlab`.
+1. __Server Installation__ - You are using the integrated runner with a GitLab
+   instance hosted on a production server. This server may be a remote server
+   and not necessarily your own, and may have TLS enabled with a self-signed
+   certificate.
+
+Following the steps below sets up the integrated runner which can be used to
+execute digital twins from the Digital Twins Preview Page.
+
+## Prerequisites
+
+A GitLab Runner picks up CI/CD jobs by communicating with a GitLab instance.
+For an explanation of how to set up a GitLab instance that integrates with a
+DTaaS application, refer to [our GitLab instance document](../gitlab/README.md)
+and [our GitLab integration guide](../gitlab/INTEGRATION.md).
+
+The rest of this document assumes you have a running DTaaS application with a
+GitLab instance running.
+
+## Runner Scopes
+
+A GitLab Runner can be configured for three different scopes:
+
+| Runner Scope    | Description |
+|-----------------|-------------|
+| Instance Runner | Available to all groups and projects in a GitLab instance. |
+| Group Runner    | Available to all projects and subgroups in a group. |
+| Project Runner  | Associated with one specific project. |
+
+We suggest creating __instance runners__ as they are the most straightforward, but
+any type will work. More about these three types can be found on
+[the official GitLab documentation page](https://docs.gitlab.com/ee/ci/runners/runners_scope.html).
+
+## Obtaining A Registration Token
+
+First, we will obtain the token necessary to register the runner for the GitLab
+instance. Open your GitLab instance (remote or local) and depending on your
+choice of runner scope, follow the steps given below:
+
+| Runner Scope    | Steps |
+|-----------------|-------|
+| Instance Runner |1. On the __Admin__ dashboard, navigate to __CI/CD > Runners__.<br>2. Select __New instance runner__.|
+| Group Runner    |1. On the __DTaaS__ group page, navigate to __Settings > CI/CD > Runners__.<br>2. Ensure the __Enable shared runners for this group__ option is enabled.<br>3. On the __DTaaS__ group page, navigate to __Build > Runners__.<br>4. Select __New group runner__.|
+| Project Runner  |1. On the __DTaaS__ group page, select the project named after your GitLab username.<br>2. Navigate to __Settings > CI/CD > Runners__.<br>3. Select __New project runner__.|
+
+For any scope you have chosen, you will be directed to a page to create a
+runner:
+
+1. Under __Platform__, select the Linux operating system.
+1. Under __Tags__, add a `linux` tag.
+1. Select __Create runner__.
+
+You should then see the following screen:
+
+![Runner Registration Screen](./runner-registration.png)
+
+Be sure to save the generated runner authentication token.
+
+## Configuring the Runner
+
+Depending on your installation scenario, the runner setup reads certain
+configurations settings:
+
+1. __Localhost Installation__ - uses `deploy/docker/.env.local`
+1. __Server Installation__ - uses `deploy/docker/.env.server`
+
+These files are integral to running the DTaaS application, so it will be
+assumed that you have already configured these.
+
+We need to register the runner with the GitLab instance so that they may
+communicate with each other. `deploy/services/runner/runner-config.toml`
+has the following template:
+
+```toml
+[[runners]]
+  name = "dtaas-runner-1"
+  url = "https://foo.com/gitlab/" # Edit this
+  token = "xxx" # Edit this
+  executor = "docker"
+  [runners.docker]
+    tls_verify = false
+    image = "ruby:2.7"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    volumes = ["/cache"]
+    network_mode = "host" # Disable this in secure contexts
+```
+
+1. Set the `url` variable to the URL of your GitLab instance.
+1. Set the `token` variable to the runner registration token you obtained earlier.
+1. If you are following the server installation scenario, remove the line
+   `network_mode = "host"`.
+
+A list of advanced configuration options is provided on the
+[GitLab documentation page](https://docs.gitlab.com/runner/configuration/advanced-configuration.html).
+
+## Start the GitLab Runner
+
+You may use the following commands to start and stop the `gitlab-runner`
+container respectively, depending on your installation scenario:
+
+1. Localhost Installation
+
+    ```bash
+    docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local up -d
+    docker compose -f deploy/services/runner/compose.runner.local.yml --env-file deploy/docker/.env.local down
+    ```
+
+1. Server Installation
+
+    ```bash
+    docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server up -d
+    docker compose -f deploy/services/runner/compose.runner.server.yml --env-file deploy/docker/.env.server down
+    ```
+
+Once the container starts, the runner within it will run automatically. You can
+tell if the runner is up and running by navigating to the page where
+you created the runner. For example, an Instance Runner would look like this:
+
+![Status indicator under Admin Area > Runners](./runner-activation.png)
+
+You will now have a GitLab runner ready to accept jobs for the GitLab instance.
+
+## Advanced: Runner Executor
+
+The term `runner` could refer one of two things:
+
+1. The `gitlab-runner` Container
+   This is the Docker container that is created when you execute the commands
+   given above. It is based on the `gitlab/gitlab-runner:alpine` image, and is
+   used to spawn one or more _executors_ that actually execute the CI/CD jobs.
+
+   These executors are spawned using a packaged version of Docker within the
+   `gitlab-runner` image.
+1. The Runner Executor
+   These are agents spawned by the `gitlab-runner` container, not as children
+   but as __siblings__ of the container. These runner executors will not show up
+   on running commands such as `docker ps`, but you can check their status by
+   running `gitlab-runner status` inside the `gitlab-runner` container.
+
+Keeping this distinction in mind is important, as the GitLab documentation
+sometimes uses them interchangeably.

deploy/services/runner/compose.runner.local.yml

@@ -0,0 +1,15 @@
+# Environment variables taken from deploy/docker/.env.local
+
+services:
+  gitlab-runner:
+    container_name: gitlab-runner
+    # Runner image version is independent of the gitlab-ce image version
+    image: gitlab/gitlab-runner:alpine-v17.5.3
+    volumes:
+      - "./runner-config.toml:/etc/gitlab-runner/config.toml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock:ro"
+      # Maps the self-signed certificate for localhost to the container
+      - "${DTAAS_DIR}/deploy/docker/certs/localhost/fullchain.pem:/etc/gitlab-runner/certs/localhost.crt:ro"
+
+    # To make https://localhost accessible from the container
+    network_mode: host

deploy/services/runner/compose.runner.server.yml

@@ -0,0 +1,13 @@
+# Environment variables taken from deploy/docker/.env.server
+
+services:
+  gitlab-runner:
+    container_name: gitlab-runner
+    # Runner image version is independent of the gitlab-ce image version
+    image: gitlab/gitlab-runner:alpine-v17.5.3
+    volumes:
+      - "./runner-config.toml:/etc/gitlab-runner/config.toml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock:ro"
+      # Maps the self-signed certificate for your server to the container
+      # Remove this if you are not using a self-signed certificate
+      - "${DTAAS_DIR}/deploy/docker/certs/${SERVER_DNS}/fullchain.pem:/etc/gitlab-runner/certs/${SERVER_DNS}.crt:ro"

deploy/services/runner/runner-config.toml

@@ -0,0 +1,13 @@
+[[runners]]
+  name = "dtaas-runner-1"
+  url = "https://foo.com/gitlab/" # Edit this
+  token = "xxx" # Edit this
+  executor = "docker"
+  [runners.docker]
+    tls_verify = false
+    image = "ruby:2.7"
+    privileged = false
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    volumes = ["/cache"]
+    network_mode = "host" # Disable this in secure contexts

docs/admin/client/config.md

@@ -15,7 +15,6 @@ This page describes various configuration options for react website.
       REACT_APP_WORKBENCHLINK_JUPYTERLAB: "Endpoint for the Jupyter Lab link",
       REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK:
         "Endpoint for the Jupyter Notebook link",
-      REACT_APP_WORKBENCHLINK_LIBRARY_PREVIEW: "Endpoint fot the Library page preview",
       REACT_APP_WORKBENCHLINK_DT_PREVIEW: "Endpoint for the Digital Twins page preview",
       REACT_APP_CLIENT_ID: 'AppID genereated by the gitlab OAuth provider',
       REACT_APP_AUTH_AUTHORITY: 'URL of the private gitlab instance',
@@ -37,7 +36,6 @@ This page describes various configuration options for react website.
       REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/',
       REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab',
       REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '',
-      REACT_APP_WORKBENCHLINK_LIBRARY_PREVIEW: '/preview/library',
       REACT_APP_WORKBENCHLINK_DT_PREVIEW: '/preview/digitaltwins',
       REACT_APP_CLIENT_ID: '1be55736756190b3ace4c2c4fb19bde386d1dcc748d20b47ea8cfb5935b8446c',
       REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/',
@@ -61,7 +59,6 @@ This page describes various configuration options for react website.
       REACT_APP_WORKBENCHLINK_VSCODE: '/tools/vscode/',
       REACT_APP_WORKBENCHLINK_JUPYTERLAB: '/lab',
       REACT_APP_WORKBENCHLINK_JUPYTERNOTEBOOK: '',
-      REACT_APP_WORKBENCHLINK_LIBRARY_PREVIEW: '/preview/library',
       REACT_APP_WORKBENCHLINK_DT_PREVIEW: '/preview/digitaltwins',
       REACT_APP_CLIENT_ID: '1be55736756190b3ace4c2c4fb19bde386d1dcc748d20b47ea8cfb5935b8446c',
       REACT_APP_AUTH_AUTHORITY: 'https://gitlab.foo.com/',

docs/admin/gitlab/index.md

@@ -66,8 +66,7 @@ should be available at <https://foo.com/gitlab>.
 
 **NOTE**: The GitLab instance operates with the `dtaas-frontend` network, which
 requires the DTaaS server to be running before you start it. You may refer to secure
-installation scenarios
-(../overview.md) for the same.
+[installation scenarios](../overview.md) for the same.
 
 The commands to start and stop the instance are:
 
@@ -138,6 +137,6 @@ for further help.
 ## Pending Tasks
 
 This README helps with installation of Gitlab along side DTaaS application.
-But the OAuth2 integration between Gitlab and DTaaS will still be still pending.
-Follow the [integration guide](integration.md) and
-[runner setup](runner.md) to setup the Gitlab integration.
+But the OAuth2 integration between Gitlab and DTaaS will still be pending.
+Follow the [integration guide](integration.md) and the
+[runner setup guide](runner.md) to setup the Gitlab integration.