MCP tool for retrieving code from Freva data analysis plugins

.env.example

@@ -21,11 +21,13 @@ CLIMATECLAW_OLLAMA_KEEP_ALIVE=-1  # Keep Ollama models loaded after use. -1 = in
 CLIMATECLAW_CODE_SERVER_PORT=8051
 CLIMATECLAW_RAG_SERVER_PORT=8050
 CLIMATECLAW_WEB_SEARCH_SERVER_PORT=8052
+CLIMATECLAW_PLUGIN_CODE_SEARCH_SERVER_PORT=8053
 
-CLIMATECLAW_AVAILABLE_MCP_SERVERS=rag-server,code-server,web-search-server  # The list of servers to connect via backends client
+CLIMATECLAW_AVAILABLE_MCP_SERVERS=rag-server,code-server,web-search-server,plugin-code-search-server  # The list of servers to connect via backends client
 CLIMATECLAW_RAG_SERVER_URL="http://haproxy:8050"  # IMPORTANT: Server URLs must be given as 'CLIMATECLAW_{server_name}_SERVER_URL' for successful parsing
 CLIMATECLAW_CODE_SERVER_URL="http://haproxy:8051"
 CLIMATECLAW_WEB_SEARCH_SERVER_URL="http://haproxy:8052"
+CLIMATECLAW_PLUGIN_CODE_SEARCH_SERVER_URL="http://plugin-code-search-server:8053"
 CLIMATECLAW_MCP_REQUEST_TIMEOUT_SEC=600
 
 # MONGODB SETTINGS

.github/workflows/ci_job.yml

@@ -65,9 +65,10 @@ jobs:
 
       - name: Start dev stack
         run: |
-          ./dev.sh up code-server web-search-server -d --wait --build --remove-orphans || {
+          ./dev.sh up code-server web-search-server plugin-code-search-server -d --wait --build --remove-orphans || {
             docker compose ps -a
             docker compose logs web-search-server
+            docker compose logs plugin-code-search-server
             docker compose logs code-server
             exit 1
           }
@@ -102,18 +103,20 @@ jobs:
           CLIMATECLAW_MONGODB_PASSWORD=ci-psswrd
           CLIMATECLAW_MONGODB_DATABASE_NAME=ci-db
           CLIMATECLAW_OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }}
+          CLIMATECLAW_GITLAB_ACCESS_TOKEN=${{ secrets.GITLAB_ACCESS_TOKEN }}
           EOF
 
       - name: Build production base image
         run: docker compose --profile build-only build climateclaw-base
 
       - name: Start production stack
         run: |
-          docker compose up climateclaw code-server web-search-server -d --wait --build --remove-orphans || {
+          docker compose up climateclaw code-server web-search-server plugin-code-search-server -d --wait --build --remove-orphans || {
             docker compose ps -a
             docker compose logs climateclaw
             docker compose logs code-server
             docker compose logs web-search-server
+            docker compose logs plugin-code-search-server
             docker compose logs mongodb
             docker compose logs litellm
             exit 1

README.md

@@ -1,9 +1,13 @@
 # ClimateClaw
-ClimateClaw is a Python service for building AI-assisted climate-data workflows. It provides the API, conversation handling, model prompting, persistent thread storage, and tool orchestration needed to support interactive work with climate data.
 
-The project integrates LiteLLM-native prompting, MongoDB-backed conversation state, and MCP-based tool execution for retrieval, code execution, and domain-specific automation.
+`ClimateClaw` is a Python service for building AI-assisted climate-data workflows. It provides the API, conversation handling, model prompting, persistent thread storage, and tool orchestration needed to support interactive work with climate data.
+
+The project integrates LiteLLM-native prompting, MongoDB-backed conversation states, and MCP-based tool execution for code execution, web/documentation search, code retrieval, and domain-specific automation.
+
+[TOC]
 
 ## Highlights
+
 - FastAPI app with strict auth parity to the production Rust service (`/api/chatbot/*`)
 - Streaming responses via LiteLLM/OpenAI-compatible SSE (`application/x-ndjson`) with code + image variants
 - Persistent conversation threads in MongoDB and JSONL files (`threads/`), plus per-user scratch space (`cache/`)
@@ -15,21 +19,25 @@ The project integrates LiteLLM-native prompting, MongoDB-backed conversation sta
 ## Quick Start (deployment)
 
 ### Requirements
+
 - `podman` or `docker`
 - Credentials & headers for the Freva auth services
 
 ### Configure environment
+
 Create `.env` (used by FastAPI, Docker, and MCP servers). See `.env.example` for guidance.
 
 ### Full stack via Docker Compose
+
 ```bash
 ./prod.sh up -d --build
 ```
+
 Services that start:
+
 - `climateclaw`: FastAPI app (debugpy toggle via `DEBUG=true` for remote debugging session)
 - `code-server`: MCP server running the sandboxed Jupyter kernel and exposing `code_interpreter`
 - `web-search-server`: MCP server doing web search via OpenAI API and exposing `web_search`
-- `rag-server`: MCP server exposing `get_context_from_resources`
 - `litellm`: LiteLLM proxy that reads `litellm_config.yaml`
 - `ollama`: Optional local model runner for LiteLLM backends
 
@@ -38,17 +46,21 @@ Bind mounts expose `/work`, logs, threads, and shared `cache` to other Freva ser
 ## Quick Start (local dev)
 
 ### Requirements
+
 - `podman` or `docker`
 
 ### Configure environment
+
 Create `.env` (used by FastAPI, Docker, and MCP servers). See `.env.example` for guidance.
 
 ### Start docker containers in DEV mode
+
 ```bash
 ./dev.sh up -d --build
 ```
 
 ## Repository Layout
+
 | Path | Purpose |
 | --- | --- |
 | `src/climateclaw/app.py` | FastAPI entrypoint, CORS policy, router registration, app lifespan hooks |
@@ -67,12 +79,13 @@ Create `.env` (used by FastAPI, Docker, and MCP servers). See `.env.example` for
 | `litellm_config.yaml` | Source of truth for model catalog (consumed by `available_chatbots()`) |
 
 Generated artifacts that persist across runs:
+
 - `threads/` (JSONL transcript per thread id)
 - `cache/{user_id}/{thread_id}` (LLM-created files, plots, etc.)
 - `logs/` (when mounted in Docker)
 
 ## Architecture at a Glance
-1. **FastAPI layer** enforces auth via `AuthRequired` (Bearer tokens validated against `x-freva-rest-url`), derives stable UUIDv5 pseudonymous user IDs from usernames, and validates per-request headers.
+1. **FastAPI layer** enforces auth via `AuthRequired` (Bearer tokens validated against `x-freva-rest-url`), derives stable UUIDv5 pseudonymous user IDs from usernames, and validates per-request headers.
 2. **LiteLLM proxy** (`CLIMATECLAW_LITE_LLM_ADDRESS`) provides OpenAI-compatible chat + embeddings endpoints; completions stream into `StreamVariant` classes that normalize assistant text, code blocks, tool hints, images, and server hints.
 3. **Persistence** uses MongoDB for storing threads and user feedback.
 4. **MCP Manager** (`src/climateclaw/services/mcp/mcp_manager.py`) connects to tool servers listed in `CLIMATECLAW_AVAILABLE_MCP_SERVERS`, discovers tools, exposes OpenAI function schemas to LiteLLM, and routes tool invocations with per-thread session ids.
@@ -94,14 +107,15 @@ Generated artifacts that persist across runs:
 | `POST` | `/api/chatbot/stop` | Initiates stopping of an active conversation | JSON body: `thread_id`; requires auth |
 
 ### Streaming contract
+
 - Response type: `application/x-ndjson`
 - Each `data:` line is a JSON object with `variant` discriminators (`Assistant`, `Code`, `CodeOutput`, `CodeError`, `Image`, `ServerHint`, `StreamEnd`, etc.).
 - Code tool calls stream incremental chunks while LiteLLM emits `tool_calls`. When the MCP tool resolves, results are converted back into JSON events and appended to Mongo/disk storage.
 - The first chunk is a `ServerHint` carrying the `thread_id`; conversation variants are stored in-memory during streaming and flushed to MongoDB at the end, ensuring replay safety.
 - Clients can call `/api/chatbot/stop?thread_id=...` to move a conversation into `STOPPING`; the streaming loop exits and cancels in-flight MCP requests (code, rag, web-search) via the shared `ActiveRequest` registry.
 
 ## Persistence, Prompts, and Assets
-- **MongoDB (`mongodb_storage.py`)**: canonical record for threads. Each document stores a UUIDv5 pseudonymous `user_id`, `thread_id`, ISO timestamp, topic (summarized via LiteLLM), and serialized `StreamVariant` list.
+- **MongoDB (`mongodb_storage.py`)**: canonical record for threads. Each document stores a UUIDv5 pseudonymous `user_id`, `thread_id`, ISO timestamp, topic (summarized via LiteLLM), and serialized `StreamVariant` list.
 - **`cache/` scratch**: `create_dir_at_cache()` ensures each user/thread has a writable directory for generated files (plots, CSVs). Entries are sanitized if user IDs contain unsupported characters.
 - **Prompt library**: `prompt_library/baseline` contains `starting_prompt.txt`, `summary_prompt.txt`, and `examples.jsonl`. GPT-5 models currently fall back to baseline prompts (warning logged). Customize by adding new prompt sets and updating `_resolve_baseline_dir()` / `_resolve_gpt5_dir_or_placeholder()`.
 - **Resources**: `resources/stableclimgen` seeds the RAG MCP server. Drop additional corpora per library folder and list them in `CLIMATECLAW_AVAILABLE_LIBRARIES` inside `src/climateclaw/tools/rag/server.py`.
@@ -110,6 +124,7 @@ Generated artifacts that persist across runs:
 - **Code interpreter** (`src/climateclaw/tools/code/server.py`): spins up per-session Jupyter kernels, sanitizes input, enforces configurable timeouts, and injects Freva config via environment variables. Outputs include stdout/stderr, display data, and structured errors.
 - **Web search server** (`src/climateclaw/tools/web_search/server.py`): calls OpenAI Web Search (`gpt-4.1`) constrained to ICON model + DKRZ/HPC docs. Honors request cancellation.
 - **RAG server** (`src/climateclaw/tools/rag/server.py`): indexes documentation with custom loaders + splitters, stores embeddings in MongoDB (`embeddings`), and surfaces a single tool `get_context_from_resources`. LiteLLM requests embed queries through the same proxy (`CLIMATECLAW_LITE_LLM_ADDRESS`).
+- **Plugin code search server** (`src/climateclaw/tools/plugin_code_search/server.py`): retrieves code from GitLab repositories used as Freva plugins for climate data analysis (included projects: *Coming Decade*, *ClimXtreme*, *RegiKlim*). Returns scraped code, doc and/or config files as query-relevant context.
 - **Header gate** (`src/climateclaw/tools/header_gate.py`): wraps each MCP ASGI app so critical headers become ContextVars and requests fail fast when missing/invalid (e.g., missing Mongo URI yields SSE-friendly JSON-RPC errors).
 - **Manager** (`src/climateclaw/services/mcp/mcp_manager.py`): caches clients, discovers tool schemas, exports OpenAI function definitions, and pins MCP session ids to thread ids for deterministic tool contexts.
 
@@ -128,6 +143,7 @@ Generated artifacts that persist across runs:
 - **Ports**: HAProxy binds `CLIMATECLAW_TARGET_PORT` for the backend and `4000` for litellm; MCP frontends bind their configured ports (e.g., 8050/8051/8052) while container instances stay internal.
 
 ## Troubleshooting
+
 - **Auth failures**: verify headers include both `Authorization` and `x-freva-rest-url`. Inspect FastAPI logs for the exact HTTP status.
 - **Missing models**: ensure `litellm_config.yaml` is readable and contains `model_name` keys. `available_chatbots()` aborts the process if it cannot find any entries.
 - **MCP issues**: backend logs warn but continue when tool discovery fails; LiteLLM will simply not emit tool calls. Use `settings.AVAILABLE_MCP_SERVERS` to enable/disable targets explicitly.

docker-compose.dev.yml

@@ -22,12 +22,17 @@ services:
       - "8600:${CLIMATECLAW_BACKEND_PORT:-8502}" # localhost:8600/docs give FastAPI docs
       - "${CLIMATECLAW_DEBUG_PORT:-5678}:${CLIMATECLAW_DEBUG_PORT:-5678}" # Port for debugpy
     volumes:
+      - /Users/fischer/DKRZ/Coding/ClimateClaw/work:/work:ro # mounts sshfs work directory for easy access to code and resources from climateclaw container
       - ./cache/:/app/cache
       - ./logs/:/app/logs
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:${CLIMATECLAW_BACKEND_PORT:-8502}/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:${CLIMATECLAW_BACKEND_PORT:-8502}/healthz'')"',
+        ]
       interval: 30s
       timeout: 5s
       retries: 3
@@ -72,12 +77,17 @@ services:
       - 8051:8051
     cpus: 4
     volumes:
+      - /Users/fischer/DKRZ/Coding/ClimateClaw/work:/work:ro
       - ./cache/:/app/cache
       - ./logs/:/app/logs
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8051/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:8051/healthz'')"',
+        ]
       interval: 30s
       timeout: 5s
       retries: 3
@@ -102,7 +112,40 @@ services:
     volumes:
       - ./logs/:/app/logs
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8052/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:8052/healthz'')"',
+        ]
+      interval: 30s
+      timeout: 5s
+      retries: 3
+      start_period: 10s
+
+  plugin-code-search-server:
+    image: plugin-code-search-server-${CLIMATECLAW_INSTANCE_NAME}
+    build:
+      context: .
+      dockerfile: docker/plugin-code-search-server/Dockerfile
+    env_file: .env
+    environment:
+      CLIMATECLAW_DEV: ${CLIMATECLAW_DEV}
+      CLIMATECLAW_DEBUG: ${CLIMATECLAW_DEBUG:-0}
+      CLIMATECLAW_MCP_HOST: 0.0.0.0
+      CLIMATECLAW_MCP_PORT: 8053
+      CLIMATECLAW_GITLAB_ACCESS_TOKEN: ${CLIMATECLAW_GITLAB_ACCESS_TOKEN:-}
+    ports:
+      - 8053:8053
+    networks:
+      - climateclaw
+    volumes:
+      - ./logs/:/app/logs
+    healthcheck:
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:8053/healthz'')"',
+        ]
       interval: 30s
       timeout: 5s
       retries: 3
@@ -130,7 +173,11 @@ services:
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://localhost:4000/health/liveliness'')"',
+        ]
       interval: 30s
       timeout: 10s
       retries: 3
@@ -216,7 +263,7 @@ services:
     user: 0:0
     hostname: volume-prep
     restart: "no"
-    command:  find /settings -mindepth 1 -maxdepth 1 -exec rm -rf {} +
+    command: find /settings -mindepth 1 -maxdepth 1 -exec rm -rf {} +
     volumes:
       - settings-data:/settings:rw
 
@@ -228,7 +275,14 @@ services:
     depends_on:
       volume-prep:
         condition: service_completed_successfully
-    command: ["clone", "--depth", "1", "https://github.com/freva-org/freva-service-config.git", "/settings"]
+    command:
+      [
+        "clone",
+        "--depth",
+        "1",
+        "https://github.com/freva-org/freva-service-config.git",
+        "/settings",
+      ]
     volumes:
       - settings-data:/settings:rw
 
@@ -241,11 +295,11 @@ services:
       git-prep:
         condition: service_completed_successfully
     command: >
-        sh -eu -c '
-          ls -rlt /settings &&
-          python /settings/dev-utils.py gen-certs --cert-dir /certs &&
-          cp -a /settings/keycloak/import/. /import/
-        '
+      sh -eu -c '
+        ls -rlt /settings &&
+        python /settings/dev-utils.py gen-certs --cert-dir /certs &&
+        cp -a /settings/keycloak/import/. /import/
+      '
     volumes:
       - settings-data:/settings:ro
       - certs-data:/certs:rw
@@ -288,11 +342,11 @@ services:
       - "8080:8080"
       - "8443:8443"
     command: |
-         start-dev
-         --hostname-strict=false
-         --hostname=http://keycloak.localhost:8080
-         --import-realm
-         -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
+      start-dev
+      --hostname-strict=false
+      --hostname=http://keycloak.localhost:8080
+      --import-realm
+      -Dkeycloak.migration.strategy=OVERWRITE_EXISTING
 
   keycloak-bootstrap:
     image: docker.io/alpine
@@ -355,12 +409,10 @@ services:
     volumes:
       - certs-data:/certs:ro
 
-
 networks:
   climateclaw:
     driver: bridge
 
-
 volumes:
   settings-data:
   certs-data:

docker-compose.yml

@@ -1,5 +1,4 @@
 services:
-
   climateclaw-base:
     # Build manually with:
     # docker compose -f docker-compose.dev.yml --profile build-only build climateclaw-base
@@ -26,7 +25,11 @@ services:
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:${CLIMATECLAW_BACKEND_PORT:-8502}/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:${CLIMATECLAW_BACKEND_PORT:-8502}/healthz'')"',
+        ]
       interval: 5s
       timeout: 3s
       retries: 20
@@ -55,7 +58,11 @@ services:
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8051/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:8051/healthz'')"',
+        ]
       interval: 5s
       timeout: 3s
       retries: 20
@@ -76,7 +83,11 @@ services:
     volumes:
       - /container/da/climateclaw-links/${CLIMATECLAW_INSTANCE_NAME}/logs:/app/logs
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://127.0.0.1:8052/healthz')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://127.0.0.1:8052/healthz'')"',
+        ]
       interval: 5s
       timeout: 3s
       retries: 20
@@ -102,7 +113,11 @@ services:
     networks:
       - climateclaw
     healthcheck:
-      test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
+      test:
+        [
+          "CMD-SHELL",
+          'python -c "import urllib.request; urllib.request.urlopen(''http://localhost:4000/health/liveliness'')"',
+        ]
       interval: 30s
       timeout: 10s
       retries: 3
@@ -142,7 +157,6 @@ services:
       retries: 3
       start_period: 15s
 
-
 networks:
   climateclaw:
     driver: bridge