Merge branch 'RM-4731_load_test_setup' into 'master'

RM-4730/4731: cdoc2-shares-server load testing setup

See merge request cdoc2/cdoc2-gatling-tests!5

.gitignore

@@ -6,6 +6,8 @@ cdoc2-capsule-server/capsule-server-test-results.log
 /cdoc2-shares-server/src/test/resources/keys/test-clients/*.p12
 cdoc2-shares-server/shares-server-test-results.log
 
+variables.sh
+
 # ---> Maven
 target/
 pom.xml.tag

cdoc2-capsule-server/setup-load-testing/README.md

@@ -83,7 +83,7 @@ Note: `http_server_requests_*` metrics appear after you have made some requests
 
 ### Start load tests
 
-Run from `cdoc2-gatling-tests` 
+Run from `cdoc2-gatling-tests/cdoc2-capsule-server` 
 
 * Create test keys:`mvn clean compile exec:java -Damount=10`
 * Create and edit `src/test/resources/application.conf`:

cdoc2-shares-server/setup-load-testing/README.md

@@ -1,13 +1,13 @@
-Contains semi-automated setup for load-testing. Can be tested locally, but for real load-testing
-needs separate Linux host with Docker and separate postgres database
+Contains semi-automated setup for load-testing. 
 
+Requires Linux host with Docker installed and Postgres DB where empty DB or existing cdoc2-shares-server
+exists.
 
 ### GitHub
 
 Create GitHub Personal Access Token with registry read rights and login to docker
 * `docker login ghcr.io -u $GH_USERNAME`
 
-
 Alternatively 
 
 * Create .gh.env file with following contents:
@@ -25,66 +25,47 @@ CR_PAT=<github personal access token with registry read access>
 Checkout https://github.com/open-eid/cdoc2-shares-server
 
 ### Edit variables.sh
-```
+```bash
 cp variables.sh.sample variables.sh
 vim variables.h
 ```
 
-### Copy server keys
-If using development keys from cdoc2-shares-server repo:
-```
-./copy_shares_server_files.sh
+### Copy server keys and certificates
+View `scp.sh` that copies development keys and configuration files from `cdoc2-shares-server` 
+to `$DEST_HOST`
+```bash
+./scp.sh
 ```
 
-Copies `cdoc2server.p12` and `servertruststore.jks` file under `keys`. 
-Copies `test_sid_trusted_issuers.jks` file under `sid-trusted-issuers`. 
 
 #### cdoc2server.p12
 This file contains server TLS certificate and private key.
 
-#### servertruststore.jks
+#### test_sid_trusted_issuers.jks
 Must contain `gatling-ca.pem` (development truststore already contains). 
-Used by shares-server to authenticate clients (client certificate must be signed by cert in truststore). 
+Used by shares-server to authenticate clients (client certificate must be signed by cert in truststore).
 
-### Setup cdoc2 database
+#### logback.xml
+Logging configuration
 
-* Edit `cdoc2-shares-server/server-db/liquibase.properties`
-* Run `mvn liquibase:update` inside `cdoc2-shares-server/server-db`
 
+### Setup cdoc2-shares-server database
 
-### Start cdoc2-shares-server servers
-
-Copy configuration to LOAD_TEST_HOST:
-```
-scp.sh
+```bash
+bash run-cdoc2-shares-server-liquibase.sh
 ```
 
-Review `docker run` settings in `run_cdoc2-shares-server.sh` files (options `--cpus` and `--memory`). 
-
+### Start cdoc2-shares-server servers
 
-On LOAD_TEST_HOST run:
+On DEST_HOST run:
 ```
 ./run_cdoc2-shares-server.sh
 ```
 
-### Install Prometheus/Grafana
-
-* `cp prometheus-sample.yml prometheus.yml`
-* Edit `prometheus.yml`, update `cdoc2-shares-server.host`/`cdoc2-shares-server.host` hostnames and `username` and `password` for `/actuator/prometheus` endpoint
-* Run `prometheus/prometheus.sh` on load host or dedicated prometheus host and check http://<prometheus.host>:9090/targets
-* Run Grafana `prometheus/grafana.sh`
-  - Open http://<grafana.host>:3000 (admin:admin) in browser
-  - [Configure Prometheus data source](https://grafana.com/docs/grafana/latest/datasources/prometheus/configure-prometheus-data-source/)
-  - (Optional) Install https://grafana.com/grafana/dashboards/17360-spring-http-example/
-  - (Optional) Install https://grafana.com/grafana/dashboards/12271-jvm-micrometer/
-
-Note: `http_server_requests_*` metrics appear after you have made some requests against cdoc2 servers 
-
 ### Start load tests
 
-Run from `cdoc2-gatling-tests` 
+Run from `cdoc2-gatling-tests/cdoc2-shares-server` 
 
-* Create test keys:`mvn clean compile`
 * Create and edit `src/test/resources/application.conf`:
   ```
   cp src/test/resources/application.conf.sample src/test/resources/application.conf
@@ -97,10 +78,9 @@ Run from `cdoc2-gatling-tests`
 
 ### Random notes:
 
-#### bash access
+#### Check that cdoc2-shares is up
 
-running container: `docker exec -it cdoc2-shares-server /bin/bash`
-image: `docker run -it --entrypoint /bin/bash ghcr.io/open-eid/cdoc2-shares-server:latest`
+`curl -k https://$DEST_HOST:18442/actuator/health`
 
 #### logs for running container
 

cdoc2-shares-server/setup-load-testing/run-cdoc2-shares-server-liquibase.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+# initialize or update database for cdoc2-shares-server
+source variables.sh
+
+DOCKER_IMAGE=${DOCKER_REGISTRY}/${DOCKER_REPOSITORY}/cdoc2-shares-server-liquibase:$SHARES_SERVER_VERSION
+
+docker run --rm \
+--env DB_URL=$POSTGRES_URL/$POSTGRES_DB \
+--env DB_PASSWORD=$POSTGRES_PASSWORD \
+--env DB_USER=$POSTGRES_USER \
+$DOCKER_IMAGE

cdoc2-shares-server/setup-load-testing/run-cdoc2-shares-server.sh

@@ -2,40 +2,42 @@
 
 source variables.sh
 
-DOCKER_IMAGE=ghcr.io/open-eid/cdoc2-shares-server:latest
+DOCKER_IMAGE=${DOCKER_REGISTRY}/${DOCKER_REPOSITORY}/cdoc2-shares-server:$SHARES_SERVER_VERSION
 
 #current directory, outside of container
 WORKDIR=$(pwd)
 
 #cdoc2 conf dir inside container
-CDOC2_CONF_DIR=/conf
+CDOC2_CONF_DIR=/config
+
+echo CDOC2_DB_URL=$POSTGRES_URL
 
 CONTAINER_NAME=cdoc2-shares-server
-#NETWORK_NAME=cdoc2-network
 
 docker pull $DOCKER_IMAGE
 
 docker stop $CONTAINER_NAME
 docker rm -f $CONTAINER_NAME
 
-# decrease thread count (default 250) to run on machine with 1 GB RAM
+
 docker run -d --restart on-failure:3 --name $CONTAINER_NAME --user "$(id -u):$(id -g)" \
-    -p 8443:8443 \
-    -p 18443:18443 \
-    -v $WORKDIR/shares-server:/conf \
+    -p 8442:8442 \
+    -p 18442:18442 \
+    -v $WORKDIR/config/sid-trusted-issuers/test_sid_trusted_issuers.jks:/config/sid_trusted_issuers.jks \
+    -v $WORKDIR/config/application.properties.docker:/config/application.properties \
+    -v $WORKDIR/config/keystore/cdoc2server.p12:/config/cdoc2server.p12 \
     --env BPL_JVM_THREAD_COUNT=250 \
     --env JAVA_OPTS="-Dspring.config.location=$CDOC2_CONF_DIR/application.properties" \
-    --env CDOC2_CONF_DIR=$CDOC2_CONF_DIR \
-    --env CDOC2_DB_URL=$CDOC2_DB_URL \
-    --env CDOC2_DB_USERNAME=$CDOC2_DB_USERNAME \
-    --env CDOC2_DB_PASSWORD=$CDOC2_DB_PASSWORD \
-    --env CDOC2_ACTUATOR_USERNAME=$CDOC2_ACTUATOR_USERNAME \
-    --env CDOC2_ACTUATOR_PASSWORD=$CDOC2_ACTUATOR_PASSWORD \
+    --env POSTGRES_URL=${POSTGRES_URL} \
+    --env POSTGRES_DB=${POSTGRES_DB} \
+    --env POSTGRES_USER=${POSTGRES_USER} \
+    --env POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
+    --env management.endpoints.metrics.username=$CDOC2_ACTUATOR_USERNAME \
+    --env management.endpoints.metrics.password=$CDOC2_ACTUATOR_PASSWORD \
+    --env server.port=8442 \
+    --env management.server.port=18442 \
     --log-opt mode=non-blocking \
-   --cpu-shares 410 \
     --cpus 2 \
-    --cpuset-cpus 0-1\
+    --memory 3072m \
     $DOCKER_IMAGE
-#    --network $NETWORK_NAME \
-#    -v /var/log/cdoc2:/var/log/cdoc2 \
-#    -v /opt/cdoc2/cdoc2-shares-server/conf/logback.xml:/workspace/logback.xml \
+

cdoc2-shares-server/setup-load-testing/scp.sh

@@ -1,5 +1,19 @@
 #!/usr/bin/env bash
 
+
 source variables.sh
 
-scp -r ../setup-load-testing $LOAD_TEST_HOST:~/
+
+echo "Coping to $DEST_HOST:$DEST_DIR"
+
+ssh $DEST_HOST "mkdir -p $DEST_DIR"
+ssh $DEST_HOST "mkdir -p $DEST_CONF_DIR"
+
+scp $CDOC2_SHARES_SERVER_DIR/shares-server/config/application.properties.docker $DEST_HOST:$DEST_CONF_DIR
+scp -r $CDOC2_SHARES_SERVER_DIR/shares-server/src/test/resources/sid-trusted-issuers $DEST_HOST:$DEST_CONF_DIR/
+scp -r $CDOC2_SHARES_SERVER_DIR/shares-server/src/test/resources/keystore $DEST_HOST:$DEST_CONF_DIR/
+scp $CDOC2_SHARES_SERVER_DIR/shares-server/src/test/resources/logback.xml $DEST_HOST:$DEST_CONF_DIR/
+
+scp variables.sh $DEST_HOST:$DEST_DIR/
+scp run-cdoc2-shares-server.sh $DEST_HOST:$DEST_DIR/
+scp run-cdoc2-shares-server-liquibase.sh $DEST_HOST:$DEST_DIR/

cdoc2-shares-server/setup-load-testing/variables.sh.sample

@@ -1,13 +1,29 @@
 #!/usr/bin/env bash
 
+DOCKER_REGISTRY=ghcr.io
+DOCKER_REPOSITORY=open-eid
+SHARES_SERVER_VERSION=0.4.1-SNAPSHOT.0-3553a87e54851276c82ee734eebcc207aaf90088
+
 # directory containing source cloned from https://github.com/open-eid/cdoc2-shares-server
-# used to copy development keys
-CDOC2_SHARES_SERVER_DIR=$(pwd)/../../cdoc2-shares-server
+# used to copy development keys and configuration files
+CDOC2_SHARES_SERVER_DIR=$(pwd)/../../../cdoc2-shares-server
+
+# linux host with docker installed in format username@host
+DEST_HOST=<user>@<host>
+
+# will contain cdoc2-shares-server configuration and scripts required to run cdoc2-shares-server in Docker
+DEST_DIR="~/shares-server"
+
+# cdoc2-shares-server configuration directory. Will contain application.properties and certificates
+DEST_CONF_DIR="${DEST_DIR}/config"
+
+# cdoc2-shares-server existing database (emtpy or existing) properties
+POSTGRES_URL=<DB_HOST>:5432
+POSTGRES_DB=cdoc2-shares
+POSTGRES_USER=cdoc2
+POSTGRES_PASSWORD=<DB_PASSWORD>
 
-CDOC2_DB_URL=jdbc:postgresql://<DB_HOST>/cdoc2
-CDOC2_DB_USERNAME=<DB_USERNAME>
-CDOC2_DB_PASSWORD=<DB_PASSWORD>
 
-# host running cdoc2-shares-server Docker containers, with username
-# username@host
-LOAD_TEST_HOST=ubuntu@<RUN_HOST>
+# used to access https://$DEST_HOST:18442/actuator/prometheus and other endpoint
+CDOC2_ACTUATOR_USERNAME=username
+CDOC2_ACTUATOR_PASSWORD=change_it