Merge remote-tracking branch 'origin/main' into checkout

Author: howardyu60211

.deploy/local/slurm/README.md

@@ -0,0 +1,188 @@
+# Local Dockerized Slurm for Clustron
+
+A self-contained Slurm cluster in Docker so the backend's `internal/slurm`
+functions (account/user/association management and job submission) can be
+exercised against a **real `slurmrestd`** during local development.
+
+> **"Slurm can't run in a container" — only half true.** Production *compute
+> nodes* are awkward to containerize (they want cgroups, GPUs, MPI, privileged
+> host access). But a single-host **dev/test** cluster — `slurmctld` +
+> `slurmdbd` + `slurmrestd` + a dynamic compute node — runs in Docker just fine.
+> This stack is built on the well-maintained
+> [giovtorres/slurm-docker-cluster](https://github.com/giovtorres/slurm-docker-cluster)
+> image and needs **no source compile** (it pulls a prebuilt image).
+
+## What you get
+
+| Container | Role | Port |
+| --- | --- | --- |
+| `mysql` (MariaDB) | slurmdbd accounting storage | internal |
+| `slurmdbd` | accounting daemon; generates the shared `jwt_hs256.key` | internal `6819` |
+| `slurmctld` | controller; mints JWTs via `scontrol token` | internal `6817` |
+| `slurmrestd` | **REST API with JWT auth** (`X-SLURM-USER-TOKEN`) | **host `6820`** |
+| `cpu-worker` | one dynamic compute node so a partition exists | internal `6818` |
+
+The backend talks to `slurmrestd` exactly as it does in dev/stage:
+`POST /slurmdb/<ver>/accounts_association`, `GET /slurm/<ver>/jobs`, etc., authenticating
+with a JWT in the `X-SLURM-USER-TOKEN` header.
+
+## Prerequisites
+
+- Docker + Docker Compose (v2). On Windows, run these from your **WSL** shell.
+- Outbound network on first run (to pull `giovtorres/slurm-docker-cluster` and
+  `mariadb`).
+
+## Quick start
+
+From `clustron-backend/`:
+
+```bash
+make slurm-up          # == ./.deploy/local/slurm/deploy.sh
+make slurm-token       # prints a root JWT  (== ./.deploy/local/slurm/mint-token.sh root)
+```
+
+`make slurm-up` waits for every daemon to become healthy and then prints which
+REST API versions the running image serves, e.g.:
+
+```
+/slurm/v0.0.43
+/slurm/v0.0.44
+/slurmdb/v0.0.43
+/slurmdb/v0.0.44
+```
+
+### Point the backend at it
+
+Copy the `slurm:` block from [`config.slurm-local.yaml`](./config.slurm-local.yaml)
+into your `clustron-backend/config.yaml`, then paste the `make slurm-token`
+output into `slurm_root_token`:
+
+```yaml
+slurm:                                       # MUST be nested under `slurm:`
+  slurm_restful_base_url: "http://localhost:6820"
+  slurm_restful_version:  "v0.0.44"          # pick one from the deploy.sh output
+  slurm_root_token:       "eyJhbGciOi..."    # from `make slurm-token`
+```
+
+By default `make slurm-token` mints a 1-hour (3600s) root token. For an
+effectively unexpired token, pass a long lifespan via `SLURM_LIFESPAN`:
+
+```bash
+make slurm-token SLURM_LIFESPAN=infinite   # exp ~year 2094 — paste once, forget
+```
+
+`SLURM_USER` and `SLURM_LIFESPAN` are forwarded to `mint-token.sh` (defaults
+`root` / `3600`), so `./.deploy/local/slurm/mint-token.sh root infinite` is the
+equivalent direct call.
+
+Now `make run` the backend. Creating a group will create a matching Slurm
+account (see below).
+
+## The group → account flow
+
+`group.Service.Create` runs a saga: **DB group → LDAP base group → LDAP admin
+group → Slurm account**. The Slurm step calls `slurm.CreateAccountAssociation`
+(the `sacctmgr add account` equivalent — `POST /slurmdb/<ver>/accounts_association`)
+with the group's LDAP CN as the account name, so the account is created **with**
+its cluster association and is usable for jobs. (The bare `/accounts` endpoint
+creates an account with no association.) If it fails, the saga compensates the
+earlier steps (LDAP groups deleted) and the DB transaction rolls back.
+
+Verify the REST path the backend uses, without the backend, with:
+
+```bash
+./smoke-test.sh                       # create + delete "clustron-smoke", asserting both
+# override the API version if needed:
+SLURM_RESTFUL_VERSION=v0.0.44 ./smoke-test.sh myaccount
+```
+
+Inspect accounts and their associations directly inside the cluster:
+
+```bash
+docker compose exec slurmctld sacctmgr -i show account
+docker compose exec slurmctld sacctmgr -i show assoc account=clustron-smoke
+```
+
+> **Heads-up — group creation now depends on Slurm.** Because the account step
+> is part of the saga, group creation will **fail** if `slurmrestd` is
+> unreachable or `slurm_root_token` is missing/expired. Keep this stack up (and
+> the token fresh) while working on groups, or expect group creation to error.
+
+## Per-user job tokens (out of scope here)
+
+The backend's *job* endpoints need a per-user JWT from an external
+[slurm-token-helper](https://github.com/NYCU-SDC/slurm-token-helper)
+(`GET /api/token/{username}`). That integration is deferred to a separate task —
+this stack is scoped to the group → Slurm-account flow, which uses
+`slurm_root_token` and needs no helper. For ad-hoc job testing you can mint a
+per-user token straight from `scontrol`:
+
+```bash
+./mint-token.sh alice 3600
+curl -H "X-SLURM-USER-TOKEN: $(./mint-token.sh alice)" http://localhost:6820/slurm/v0.0.44/jobs
+```
+
+## Slurm version / REST API version
+
+`SLURM_VERSION` (default `25.11.4`) selects the image tag and therefore which
+OpenAPI plugins `slurmrestd` exposes. The backend's `slurm_restful_version`
+**must** match one of them. The project uses **`v0.0.44`**, which Slurm 25.11
+serves. SchedMD removes old plugin versions over time, so if you need a specific
+version, confirm it in the `make slurm-up` output and pin `SLURM_VERSION`
+accordingly:
+
+```bash
+SLURM_VERSION=25.05.6 make slurm-up   # if you specifically need an older API
+```
+
+## Teardown
+
+```bash
+make slurm-down        # == ./cleanup.sh ; stops everything and wipes volumes
+```
+
+This removes the MariaDB and Slurm state volumes, resetting all accounts.
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `compose.yaml` | the cluster definition (prebuilt image, no build) |
+| `deploy.sh` / `cleanup.sh` | bring up (with version probe) / tear down |
+| `mint-token.sh` | mint a root or per-user JWT via `scontrol token` |
+| `smoke-test.sh` | create, read, and delete an account via the backend's REST paths (asserts deletion) |
+| `config.slurm-local.yaml` | the `slurm_*` block to paste into `config.yaml` |
+
+## Troubleshooting
+
+**`sinfo` shows the node `down` / jobs stay `PENDING` ("Required node not
+available") after a host or Docker restart.** The compute node is dynamic
+(`slurmd -Z`); when its container restarts, slurmctld marks it
+`down` ("Node unexpectedly rebooted") and the image's `ReturnToService=1` only
+auto-returns *non-responsive* nodes. Just re-run:
+
+```bash
+make slurm-up        # resumes any down nodes and sets ReturnToService=2
+```
+
+or fix it by hand:
+
+```bash
+docker exec clustron-slurm-slurmctld scontrol update nodename=c1 state=resume
+```
+
+**`srun` fails / hangs when run as root.** Slurm refuses to run jobs as root or
+SlurmUser. Submit as a normal user inside the cluster:
+
+```bash
+docker exec clustron-slurm-slurmctld su slurm -s /bin/bash -c "srun -N1 hostname"
+```
+
+## Notes / caveats
+
+- Compute is minimal: one dynamic node, jobs may sit `PENDING`. Accounting and
+  job *submission/listing* work; this stack is not for running real workloads.
+- `slurmctld`, `slurmrestd` and the worker run `privileged: true` (Slurm needs
+  it). Fine for local dev; do not copy this layout to production.
+- The cluster is independent of the core `.deploy/local` stack (separate Compose
+  project `clustron-slurm-local`), so `make prepare`/`make run` are unaffected.

.deploy/local/slurm/cleanup.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Stop the cluster and wipe its volumes (resets the slurmdbd accounting database).
+set -euo pipefail
+cd "$(dirname "$0")"
+echo ":: Removing Clustron local Slurm cluster and volumes ..."
+docker compose down -v

.deploy/local/slurm/compose.yaml

@@ -0,0 +1,156 @@
+# Local Dockerized Slurm cluster for Clustron backend development.
+#
+# Self-contained: uses the prebuilt giovtorres/slurm-docker-cluster image
+# (no source compile). Brings up slurmdbd + slurmctld + slurmrestd (JWT auth on
+# :6820) + one dynamic compute node, backed by MariaDB for accounting.
+#
+# This lets the backend's internal/slurm functions (CreateAccount, CreateUser,
+# CreateAssociation, job submission, ...) talk to a real slurmrestd.
+#
+# Quick start (from clustron-backend/):
+#   make slurm-up        # or: ./.deploy/local/slurm/deploy.sh
+#   make slurm-token     # mint a root JWT for slurm_root_token
+# See README.md in this directory for full details.
+
+name: clustron-slurm-local
+
+x-slurm-image: &slurm-image giovtorres/slurm-docker-cluster:${SLURM_VERSION:-25.11.4}
+
+services:
+  mysql:
+    image: mariadb:12
+    hostname: mysql
+    container_name: clustron-slurm-mysql
+    environment:
+      MYSQL_RANDOM_ROOT_PASSWORD: "yes"
+      MYSQL_DATABASE: slurm_acct_db
+      MYSQL_USER: slurm
+      MYSQL_PASSWORD: password
+    volumes:
+      - var_lib_mysql:/var/lib/mysql
+    networks:
+      - slurm-network
+    healthcheck:
+      test: ["CMD", "healthcheck.sh", "--connect", "--innodb_initialized"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 30s
+
+  slurmdbd:
+    image: *slurm-image
+    init: true
+    command: ["slurmdbd"]
+    container_name: clustron-slurm-slurmdbd
+    hostname: slurmdbd
+    environment:
+      MYSQL_USER: slurm
+      MYSQL_PASSWORD: password
+    volumes:
+      - etc_munge:/etc/munge
+      - etc_slurm:/etc/slurm
+      - var_log_slurm:/var/log/slurm
+    expose:
+      - "6819"
+    depends_on:
+      mysql:
+        condition: service_healthy
+    networks:
+      - slurm-network
+    healthcheck:
+      test: ["CMD-SHELL", "pidof slurmdbd"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+
+  slurmctld:
+    image: *slurm-image
+    init: true
+    command: ["slurmctld"]
+    container_name: clustron-slurm-slurmctld
+    hostname: slurmctld
+    privileged: true
+    working_dir: /data
+    volumes:
+      - etc_munge:/etc/munge
+      - etc_slurm:/etc/slurm
+      - slurm_jobdir:/data
+      - var_log_slurm:/var/log/slurm
+    expose:
+      - "6817"
+    depends_on:
+      slurmdbd:
+        condition: service_healthy
+    networks:
+      - slurm-network
+    healthcheck:
+      test: ["CMD-SHELL", "scontrol ping"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+
+  slurmrestd:
+    image: *slurm-image
+    init: true
+    command: ["slurmrestd"]
+    container_name: clustron-slurm-slurmrestd
+    hostname: slurmrestd
+    privileged: true
+    volumes:
+      - etc_munge:/etc/munge
+      - etc_slurm:/etc/slurm
+      - var_log_slurm:/var/log/slurm
+    ports:
+      # Host:Container. The backend's slurm_restful_base_url points here.
+      - "6820:6820"
+    depends_on:
+      slurmctld:
+        condition: service_healthy
+    networks:
+      - slurm-network
+    healthcheck:
+      test: ["CMD-SHELL", "test -S /var/run/slurmrestd/slurmrestd.socket"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+
+  cpu-worker:
+    image: *slurm-image
+    init: true
+    command: ["slurmd-cpu"]
+    working_dir: /data
+    privileged: true
+    environment:
+      COMPOSE_PROJECT_NAME: clustron-slurm-local
+    volumes:
+      - etc_munge:/etc/munge
+      - etc_slurm:/etc/slurm
+      - slurm_jobdir:/data
+      - var_log_slurm:/var/log/slurm
+    expose:
+      - "6818"
+    depends_on:
+      slurmctld:
+        condition: service_healthy
+    networks:
+      - slurm-network
+    healthcheck:
+      test: ["CMD-SHELL", "pidof slurmd"]
+      interval: 10s
+      timeout: 5s
+      retries: 5
+      start_period: 20s
+
+volumes:
+  etc_munge:
+  etc_slurm:
+  var_log_slurm:
+  slurm_jobdir:
+  var_lib_mysql:
+
+networks:
+  slurm-network:
+    driver: bridge

.deploy/local/slurm/config.slurm-local.yaml

@@ -0,0 +1,25 @@
+# Slurm settings for the local Dockerized cluster (.deploy/local/slurm).
+#
+# IMPORTANT: these keys must be NESTED under a top-level `slurm:` key — the
+# backend maps them via `Slurm slurm.Config `yaml:"slurm"`` in internal/config.
+# Flat/top-level keys are silently ignored (you'll get empty base URL + version
+# and a `Post "/slurmdb//accounts": unsupported protocol scheme ""` error).
+#
+# Copy this whole `slurm:` block into your clustron-backend/config.yaml.
+#
+# Do NOT also set SLURM_* env vars: setting even one of them replaces the entire
+# slurm config struct (config.Merge is per-top-level-field), wiping the others.
+# Put these in ONE place — here, or as env vars.
+#
+#   slurm_restful_version: must be a version the running image serves. Slurm
+#     25.11 serves v0.0.42 / v0.0.43 / v0.0.44; `make slurm-up` prints the list.
+#   slurm_root_token: paste the output of `make slurm-token`. Tokens expire
+#     (default 1h) — re-mint when you start getting 401s.
+#
+# (slurm_token_helper_url is only needed for the per-user job endpoints, which
+# are out of scope here; it's defined-but-not-required, so it's omitted.)
+
+slurm:
+  slurm_restful_base_url: "http://localhost:6820"
+  slurm_restful_version: "v0.0.44"
+  slurm_root_token: "PASTE_OUTPUT_OF_make_slurm-token"