Sensitive Data Archive

The SDA contains all components of NeIC Sensitive Data Archive. It can be used as part of a Federated EGA or as a standalone Sensitive Data Archive.

For more information about the different components, please refer to the README files in their respective folders.

How to run the SDA stack

The following instructions outline the steps to set up and run the SDA services for development and testing using Docker. These steps are based on the provided Makefile commands.

Prerequisites

Ensure you have the following installed on your system:

• Go: The required version is specified in the sda Dockerfile. Verify using

  $ make go-version-check

• Docker: Version 24 or higher. Verify using

  $ make docker-version-check 

• Docker Compose: Version 2 or higher. For Linux, ensure the Compose plugin is installed.

In preparation for local development, it is essential to verify that $GOPATH/bin is part of the system PATH, as certain distributions may package outdated versions of build tools. SDA uses Go Modules, and it is advisable to clone the repository outside the GOPATH. After cloning, initialize the environment and obtain necessary build tools using the bootstrap command:

$ make bootstrap

Build Docker images

Build the required Docker images for all SDA services:

$ make build-all

You can also build images for individual services by replacing all with the folder name (postgresql, rabbitmq, sda, sda-download, sda-sftp-inbox), for example

$ make build-sda

To build the sda-admin CLI tool:

$ make build-sda-admin

$ make build-sda-doa

Running the services

Start services with Docker Compose

The following command will bring up all services using the Docker Compose file sda-s3-integration.yml (configured for S3 as the storage backend):

$ make sda-s3-up

Shut down all services and clean up resources

The following command will shut down all services and clean up all related resources:

$ make sda-s3-down

For the setup with POSIX as the storage backend, use make sda-posix-up and make sda-posix-down to start and shut down services.

For the setup including the sync service, use make sda-sync-up and make sda-sync-down to start and shut down services.

Running the integration tests

This will build all required images, bring up the services, run the integration test, and then shut down services and clean up resources. The same test runs on every pull request (PR) in GitHub.

• Integration test for the database:

  make integrationtest-postgres

• Integration test for RabbitMQ:

  make integrationtest-rabbitmq

• Integration test for all SDA setups (including S3, POSIX and sync):

  make integrationtest-sda

• Integration test for SDA using POSIX as the storage backend:

  make integrationtest-sda-posix

• Integration test for SDA using S3 as the storage backend:

  make integrationtest-sda-s3

• Integration test for SDA including the sync service:

  make integrationtest-sda-sync

• Integration test for SDA-DOA using POSIX as the storage backend:

  make integrationtest-sda-doa-posix

• Integration test for SDA-DOA using S3 as the storage backend:

  make integrationtest-sda-doa-s3

Running the integration tests without shutting down the services

This will run the integration tests and keep the services running after the tests are finished.

• Integration test for SDA using POSIX as the storage backend:

  make integrationtest-sda-posix-run

• Integration test for SDA using S3 as the storage backend:

  make integrationtest-sda-s3-run

• Integration test for SDA including the sync service:

  make integrationtest-sda-sync-run

• Integration test for SDA-DOA using POSIX as the storage backend:

  make integrationtest-sda-doa-posix-run

• Integration test for SDA-DOA using S3 as the storage backend:

  make integrationtest-sda-doa-s3-run

After that, you will need to shut down the services manually.

• Shut down services for SDA using POSIX as the storage backend

  make integrationtest-sda-posix-down

• Shut down services for SDA using S3 as the storage backend

  make integrationtest-sda-s3-down

• Shut down services for SDA including the sync service:

  make integrationtest-sda-sync-down

• Shut down services for SDA-DOA using POSIX as the storage backend

  make integrationtest-sda-doa-posix-down

• Shut down services for SDA-DOA using S3 as the storage backend

  make integrationtest-sda-doa-s3-down

Linting the Go code

To run golangci-lint for all Go components:

$ make lint-all

To run golangci-lint for a specific component, replace all with the folder name (sda, sda-auth, sda-download), for example:

$ make lint-sda

Running the static code tests

For Go code, this means running go test -count=1 ./... in the target folder. For the sftp-inbox this calls mvn test -B inside a Maven container.

To run the static code tests for all components:

$ make test-all

To run the static code tests for a specific component, replace all with the folder name (sda, sda-admin, sda-download, sda-sftp-inbox), for example:

$ make test-sda

Testing and developing the helm charts locally

Developing and testing the Helm charts (or other deployment manifests) requires a Kubernetes environment. One of the most lightweight distributions available is k3d.

install k3d

The simplest way to install k3d is by using the official install script.

• wget:

wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

• curl:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

Install kubectl

If kubectl is not installed, run the following command to download the latest stable version. (substitue linux/amd64 with darwin/arm64 if you are using a Mac).

curl -sLO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

Create a cluster

Once installed a cluster can be created using the make k3d-create-cluster command, you can create a cluster named k3s-default. The new cluster's connection details will automatically be merged into your default kubeconfig and activated. The command below should show the created node.

kubectl get nodes

The Nginx ingress controller is deployed and will bind to ports 80 and 443 of the host system. A deployed service with an ingress definition can then be targeted by setting the Host: HOSTNAME header for that service.

curl -H "Host: test" http://localhost/

For testing ingress endpoints with other applications like a web browser, the hostname in the ingress definition should have the .127.0.0.1.nip.io ending in order to not have to modify the /etc/hosts file, ex. app.127.0.0.1.nip.io.

Remove the cluster

Removing the cluster can be done using the make k3d-delete-cluster command or as shown below if a specific name is used during creation.

Deploy the components

Deployment of the charts can be done as describe below in more detail, or by using the corresponding command in the Makefile

Makefile commands

• make k3d-deploy-dependencies-federated - bootstrap dependencies, F-EGA use case
• make k3d-deploy-dependencies-isolated - bootstrap dependencies, standalone use case
• make k3d-import-images - build and import images into the default cluster named k3s-default
• make k3d-deploy-postgres - deploy the sda-db chart without TLS
• make k3d-deploy-rabbitmq-federated - deploy the sda-mq chart without TLS, F-EGA use case
• make k3d-deploy-rabbitmq-isolated - deploy the sda-mq chart without TLS, standalone use case
• make k3d-deploy-sda-s3-federated - deploy the sda-svc chart with S3 storage without TLS, F-EGA use case
• make k3d-deploy-sda-s3-isolated - deploy the sda-svc chart with S3 storage without TLS, standalone use case
• make k3d-deploy-sda-posix - deploy the sda-svc chart with POSIX storage without TLS
• make k3d-cleanup-all-deployments - Remove all deployed components and dependencies

Bootstrap the dependencies

This script requires yq, the GO version of crypt4gh as well as xxd and jq to be installed.

Bootstrap the dependencies with the command: make k3d-deploy-dependencies-[federated|isolated].

Deploy the Sensitive Data Archive components

Start by building and importing the required containers using the make k3d-import-images.

The Postgres and RabbitMQ Needs to be deployed first using the following commands: make k3d-deploy-postgres and make k3d-deploy-rabbitmq-[federated|isolated].

Once the DB and MQ are installed the SDA stack can be installed, here the desired storage backend needs to specified as well (posix or s3), make k3d-deploy-sda-posix or make k3d-deploy-sda-s3-[federated|isolated].

Testing with ingress

Once everything is deployed it is posible to interact with the services using the following hostnames:

• api.127.0.0.1.nip.io
• auth.127.0.0.1.nip.io
• broker.127.0.0.1.nip.io
• download.127.0.0.1.nip.io
• inbox.127.0.0.1.nip.io

Cleanup all deployed components

Once the testing is concluded all deployed components can be removed with the command make k3d-cleanup-all-deployments