ClearML server

.github/issue_template.md

@@ -0,0 +1,16 @@
+---
+
+name: "Issue template"
+about: "Minimal info about an issue"
+title: ""
+ref: "devel"
+
+---
+
+## Description
+
+
+<!--- ## Screenshots (if appropriate): -->
+
+## References
+<!--- Create a list of references, (in a bullet style) -->

.github/pull_request_template.md

@@ -0,0 +1,27 @@
+<!--- Provide a general summary of your changes in the Title above -->
+## Description
+
+
+## Motivation and context
+<!--- Why is this change required? What problem does it solve? -->
+<!--- If it fixes an open issue, please link to the issue here. -->
+
+
+<!--- ## Screenshots (if appropriate): -->
+
+## How has this been tested?
+<!--- Please describe in detail how you tested your changes. -->
+<!--- Include details of your testing environment, and the tests you ran to -->
+<!--- see how your change affects other areas of the code, etc. -->
+
+
+## Checklist
+<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
+<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
+- [ ] All TODOs in the code have been resolved or linked to a proper issue.
+- [ ] Code has been (auto)formatted.
+- [ ] Documentation (e.g., README, CHANGELOG, Wiki) has been updated.
+- [ ] All automated checks have passed.
+
+---
+Clickup task: [hash](link)

.github/workflows/pre-commit.yaml

@@ -0,0 +1,11 @@
+---
+name: Pre-Commit
+
+on:
+  push:
+
+jobs:
+  pre-commit:
+    uses: ros-controls/ros2_control_ci/.github/workflows/reusable-pre-commit.yml@master
+    with:
+      ros_distro: humble

.pre-commit-config.yaml

@@ -0,0 +1,30 @@
+repos:
+-   repo: https://gitlab.com/vojko.pribudic.foss/pre-commit-update
+    rev: v0.9.0
+    hooks:
+    -   id: pre-commit-update
+
+-   repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v6.0.0
+    hooks:
+    -   id: check-added-large-files
+    -   id: check-case-conflict
+    -   id: check-json
+    -   id: check-merge-conflict
+    -   id: check-symlinks
+    -   id: check-toml
+    -   id: check-xml
+    -   id: check-yaml
+    -   id: debug-statements
+    -   id: destroyed-symlinks
+    -   id: detect-private-key
+    -   id: end-of-file-fixer
+    -   id: mixed-line-ending
+    -   id: pretty-format-json
+    -   id: trailing-whitespace
+
+-   repo: https://github.com/codespell-project/codespell
+    rev: v2.4.1
+    hooks:
+    -   id: codespell
+        args: [-L, passt, -w]

README.md

@@ -1 +1,67 @@
-# robolab_mlops
+# robolab_mlops
+
+Collection of all services used in Robotics Laboratory (mainly for the [Aegis robot](https://github.com/AGH-CEAI/aegis_ros)).
+
+## List of services with their default ports
+
+* ClearML - [clearml](http://localhost:8080)
+* Containers registry - [containers_registry](http://localhost:5000)
+* Portainer - [portainer](http://localhost:8999)
+* PPA repository - [ppa_packages](http://localhost:80/debian)
+
+**Before first run, be sure to read the whole README!**
+
+## Start all services
+
+```bash
+# In the main directory
+docker compose up -d
+```
+
+## Shutdown all services
+
+```bash
+# In the main directory
+docker compose down
+```
+
+---
+
+### Private Packages Repo (PPA)
+
+You can find instructions for setting up a private repository [here](./ppa_packages/README.md).
+
+**Adding private repo**
+```bash
+echo "deb [trusted=yes] http://192.168.0.100/debian ./" | tee -a /etc/apt/sources.list > /dev/null
+```
+### Containers registry
+
+You can find instructions for using the self-hosted container registry [here](./containers_registry/README.md).
+
+#### Building & pushing a particular release tag
+```bash
+export REGISTRY_HOSTNAME=geonosis:5000
+export AEGIS_ROS_VERSION=<TAG>
+export AEGIS_CONTAINER_VERSION=<TAG>
+
+# if the building is happening on the same machine as PPA server, add argument:
+# --add-host $(hostname):$(hostname -I | awk '{print $1}')
+podman build . -t ceai/aegis_dev:${AEGIS_CONTAINER_VERSION} --build-arg AEGIS_ROS_TAG=${AEGIS_ROS_VERSION}
+podman tag ceai/aegis_dev:${AEGIS_CONTAINER_VERSION} ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+podman image inspect --format='{{json .Config.Labels}}' ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+
+podman push ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+```
+
+#### Pulling & entering a particular release tag
+```bash
+export REGISTRY_HOSTNAME=geonosis:5000
+export AEGIS_CONTAINER_VERSION=<TAG>
+
+podman pull ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+podman image inspect --format='{{json .Config.Labels}}' ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+
+toolbox create --image ${REGISTRY_HOSTNAME}/ceai/aegis:${AEGIS_CONTAINER_VERSION}
+toolbox enter aegis-${AEGIS_CONTAINER_VERSION}
+```

clearml/README.md

@@ -0,0 +1,18 @@
+# ClearML
+
+Deploy instructions can be found in the [official docs](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_linux_mac).
+
+## Highlight
+Before first run you need to create all catalogs and setup the Elasticsearch params:
+
+```bash
+echo "vm.max_map_count=524288" > /tmp/99-clearml.conf
+sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
+sudo sysctl -w vm.max_map_count=524288
+sudo service docker restart
+```
+
+Please continue the setup procedure for:
+* [Web Login Authentication](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#web-login-authentication)
+* [Hashing passwords](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#using-hashed-passwords)
+* [Removing Fileserver Artifacts for Deleted Tasks](https://clear.ml/docs/latest/docs/deploying_clearml/clearml_server_config#removing-fileserver-artifacts-for-deleted-tasks)

clearml/compose.yml

@@ -0,0 +1,211 @@
+services:
+
+  apiserver:
+    command:
+    - apiserver
+    container_name: clearml-apiserver
+    image: clearml/server:latest
+    restart: unless-stopped
+    volumes:
+    - /srv/data/hosting/volumes/clearml/logs:/var/log/clearml
+    - /srv/data/hosting/volumes/clearml/config:/opt/clearml/config
+    - /srv/data/hosting/volumes/clearml/data/fileserver:/mnt/fileserver
+    depends_on:
+      - redis
+      - mongo
+      - elasticsearch
+      - fileserver
+    environment:
+      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
+      CLEARML_ELASTIC_SERVICE_PORT: 9200
+      CLEARML_MONGODB_SERVICE_HOST: mongo
+      CLEARML_MONGODB_SERVICE_PORT: 27017
+      CLEARML_REDIS_SERVICE_HOST: redis
+      CLEARML_REDIS_SERVICE_PORT: 6379
+      CLEARML_SERVER_DEPLOYMENT_TYPE: linux
+      CLEARML__apiserver__pre_populate__enabled: "true"
+      CLEARML__apiserver__pre_populate__zip_files: "/opt/clearml/db-pre-populate"
+      CLEARML__apiserver__pre_populate__artifacts_path: "/mnt/fileserver"
+      CLEARML__services__async_urls_delete__enabled: "true"
+      CLEARML__services__async_urls_delete__fileserver__url_prefixes: "[${CLEARML_FILES_HOST:-}]"
+      CLEARML__secure__credentials__services_agent__user_key: ${CLEARML_AGENT_ACCESS_KEY:-}
+      CLEARML__secure__credentials__services_agent__user_secret: ${CLEARML_AGENT_SECRET_KEY:-}
+    ports:
+    - "8008:8008"
+    networks:
+      - backend
+      - frontend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+
+  elasticsearch:
+    networks:
+      - backend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    container_name: clearml-elastic
+    environment:
+      bootstrap.memory_lock: "true"
+      cluster.name: clearml
+      cluster.routing.allocation.node_initial_primaries_recoveries: "500"
+      cluster.routing.allocation.disk.watermark.low: 500mb
+      cluster.routing.allocation.disk.watermark.high: 500mb
+      cluster.routing.allocation.disk.watermark.flood_stage: 500mb
+      discovery.type: "single-node"
+      http.compression_level: "7"
+      node.name: clearml
+      reindex.remote.whitelist: "'*.*'"
+      xpack.security.enabled: "false"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    image: elasticsearch:8.17.0
+    restart: unless-stopped
+    volumes:
+      - /srv/data/hosting/volumes/clearml/data/elastic_7:/usr/share/elasticsearch/data
+      - /usr/share/elasticsearch/logs
+
+  fileserver:
+    networks:
+      - backend
+      - frontend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    command:
+    - fileserver
+    container_name: clearml-fileserver
+    image: clearml/server:latest
+    environment:
+      CLEARML__fileserver__delete__allow_batch: "true"
+    restart: unless-stopped
+    volumes:
+    - /srv/data/hosting/volumes/clearml/logs:/var/log/clearml
+    - /srv/data/hosting/volumes/clearml/data/fileserver:/mnt/fileserver
+    - /srv/data/hosting/volumes/clearml/config:/opt/clearml/config
+    ports:
+    - "8081:8081"
+
+
+  mongo:
+    networks:
+      - backend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    container_name: clearml-mongo
+    image: mongo:7.0.22
+    restart: unless-stopped
+    command: --setParameter internalQueryMaxBlockingSortMemoryUsageBytes=196100200
+    volumes:
+    - /srv/data/hosting/volumes/clearml/data/mongo_4/db:/data/db
+    - /srv/data/hosting/volumes/clearml/data/mongo_4/configdb:/data/configdb
+
+  redis:
+    networks:
+      - backend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    container_name: clearml-redis
+    image: redis:7.4.1
+    restart: unless-stopped
+    volumes:
+    - /srv/data/hosting/volumes/clearml/data/redis:/data
+
+  webserver:
+    command:
+    - webserver
+    container_name: clearml-webserver
+    # environment:
+    #  CLEARML_SERVER_SUB_PATH : clearml-web # Allow Clearml to be served with a URL path prefix.
+    image: clearml/server:latest
+    restart: unless-stopped
+    depends_on:
+      - apiserver
+    ports:
+    - "8080:80"
+    networks:
+      - backend
+      - frontend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+
+  async_delete:
+    depends_on:
+      - apiserver
+      - redis
+      - mongo
+      - elasticsearch
+      - fileserver
+    container_name: clearml-async_delete
+    image: clearml/server:latest
+    networks:
+      - backend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    restart: unless-stopped
+    environment:
+      CLEARML_ELASTIC_SERVICE_HOST: elasticsearch
+      CLEARML_ELASTIC_SERVICE_PORT: 9200
+      CLEARML_MONGODB_SERVICE_HOST: mongo
+      CLEARML_MONGODB_SERVICE_PORT: 27017
+      CLEARML_REDIS_SERVICE_HOST: redis
+      CLEARML_REDIS_SERVICE_PORT: 6379
+      PYTHONPATH: /opt/clearml/apiserver
+      CLEARML__services__async_urls_delete__fileserver__url_prefixes: "[${CLEARML_FILES_HOST:-}]"
+    entrypoint:
+      - python3
+      - -m
+      - jobs.async_urls_delete
+      - --fileserver-host
+      - http://fileserver:8081
+    volumes:
+      - /srv/data/hosting/volumes/clearml/logs:/var/log/clearml
+      - /srv/data/hosting/volumes/clearml/config:/opt/clearml/config
+
+  agent-services:
+    networks:
+      - backend
+    extra_hosts:
+      - "geonosis:192.168.0.100"
+    container_name: clearml-agent-services
+    image: clearml/clearml-agent-services:latest
+    deploy:
+      restart_policy:
+        condition: on-failure
+    privileged: true
+    environment:
+      CLEARML_HOST_IP: ${CLEARML_HOST_IP}
+      CLEARML_WEB_HOST: ${CLEARML_WEB_HOST:-}
+      CLEARML_API_HOST: http://apiserver:8008
+      CLEARML_FILES_HOST: ${CLEARML_FILES_HOST:-}
+      CLEARML_API_ACCESS_KEY: ${CLEARML_AGENT_ACCESS_KEY:-$CLEARML_API_ACCESS_KEY}
+      CLEARML_API_SECRET_KEY: ${CLEARML_AGENT_SECRET_KEY:-$CLEARML_API_SECRET_KEY}
+      CLEARML_AGENT_GIT_USER: ${CLEARML_AGENT_GIT_USER}
+      CLEARML_AGENT_GIT_PASS: ${CLEARML_AGENT_GIT_PASS}
+      CLEARML_AGENT_UPDATE_VERSION: ${CLEARML_AGENT_UPDATE_VERSION:->=0.17.0}
+      CLEARML_AGENT_DEFAULT_BASE_DOCKER: "ubuntu:18.04"
+      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID:-}
+      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY:-}
+      AWS_DEFAULT_REGION: ${AWS_DEFAULT_REGION:-}
+      AZURE_STORAGE_ACCOUNT: ${AZURE_STORAGE_ACCOUNT:-}
+      AZURE_STORAGE_KEY: ${AZURE_STORAGE_KEY:-}
+      GOOGLE_APPLICATION_CREDENTIALS: ${GOOGLE_APPLICATION_CREDENTIALS:-}
+      CLEARML_WORKER_ID: "clearml-services"
+      CLEARML_AGENT_DOCKER_HOST_MOUNT: "/opt/clearml/agent:/root/.clearml"
+      SHUTDOWN_IF_NO_ACCESS_KEY: 1
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /srv/data/hosting/volumes/clearml/agent:/root/.clearml
+    depends_on:
+      - apiserver
+    entrypoint: >
+      bash -c "curl --retry 10 --retry-delay 10 --retry-connrefused 'http://apiserver:8008/debug.ping' && /usr/agent/entrypoint.sh"
+
+networks:
+  backend:
+    driver: bridge
+  frontend:
+    driver: bridge

compose.yml

@@ -0,0 +1,8 @@
+name: robolab-ops
+include:
+  - path: clearml/compose.yml
+    env_file:
+      - /srv/data/hosting/secrets/clearml.env
+  - containers_registry/compose.yml
+  - portainer/compose.yml
+  - ppa_packages/compose.yml