feat(cirrus): config-only deployment of the proxy on the self-hosted k3s cluster (#102)

* feat(cirrus): deploy the proxy on the self-hosted k3s cluster

OpenRouter + DSE-nimbus only, logging to in-cluster MinIO.

* docs(cirrus): README/LOGGING/AGENTS/CHANGELOG for the cirrus deployment

* refactor(cirrus): make the cirrus deployment config-only, touching nothing NRP reads

Drops the shared-code changes (PROXY_CONFIG, APP_CORS, AWS_S3_ADDRESSING_STYLE,
default_provider) and the consolidate_logs.py extraction from both NRP CronJobs.
The provider set now arrives as a ConfigMap mounted over /app/config.json, and
the log flush uses the pre-existing LOG_BUCKET / AWS_S3_ENDPOINT_URL env vars
(boto3 already path-styles a custom endpoint). Parquet consolidation is dropped
for now: raw JSONL only.

Diff outside cirrus/ is documentation.

Author: cboettig

AGENTS.md

@@ -12,6 +12,28 @@ This is an LLM proxy service that routes chat completion requests to NRP, OpenRo
 
 > **CORS gotcha:** browser CORS is enforced by the **haproxy ingress** (`ingress.yaml` annotations), *not* the app's `CORSMiddleware` (which is effectively dead config). `cors-allow-headers` is an explicit list — a new custom request header (e.g. `X-Client`) must be added there or the browser preflight blocks the whole request. See the comment in `ingress.yaml`.
 
+## Two deployments
+
+The canonical deployment is on **NRP** (`open-llm-proxy.nrp-nautilus.io`,
+namespace `biodiversity`) — that is what the manifests at the repo root, and
+everything below, describe. A second, narrower deployment runs on the lab's
+**self-hosted k3s cluster** (`llm-proxy.carlboettiger.info`, namespace
+`llm-proxy`): OpenRouter + DSE-nimbus only, raw-JSONL logs in the in-cluster
+MinIO mirror, Traefik instead of HAProxy. See [cirrus/](cirrus/README.md).
+
+Two rules for agents working on it:
+
+1. **cirrus is config-only.** Everything cirrus-specific lives in `cirrus/` as
+   deployment config (a ConfigMap mounted over `config.json`, env vars, Traefik
+   CRDs). It runs this repo's unmodified code. Do **not** make an app change
+   "for cirrus" and do **not** point its manifests at a branch — both deployments
+   git-clone `main` at pod boot, so a shared-code edit reaches NRP at its next
+   restart whether or not that was the intent. If cirrus genuinely needs an app
+   change, propose it on its own merits for both deployments.
+2. **Check which cluster `kubectl` is pointed at** (`kubectl config
+   current-context`) before touching anything — a laptop configured for cirrus
+   generally cannot reach NRP, and vice versa.
+
 ## Evaluating Logs
 
 Logs land in three tiers by age (see [LOGGING.md](LOGGING.md) for the full spec):

CHANGELOG.md

@@ -34,6 +34,31 @@ See [Releases](README.md#releases) for how a release is cut.
   dialect (as `kimi` does, not `qwen3`'s `enable_thinking`, which it ignores), so its
   reasoning can actually be toggled. Prefix matching remains provider-declaration-ordered;
   the `gemma4-nimbus` / `qwen3-cirrus` shadowing noted in #105 is left as-is.
+- **Second deployment on the self-hosted k3s cluster (cirrus):
+  `https://llm-proxy.carlboettiger.info`.** Manifests in `cirrus/` (namespace
+  `llm-proxy`), documented in [cirrus/README.md](cirrus/README.md).
+  **Config-only and NRP-inert:** it runs this repo's unmodified `llm_proxy.py`
+  from `main`, and adds no file the NRP deployment reads — no application code,
+  no `config.json` change, no CronJob change. Everything cirrus-specific is
+  deployment config:
+  - **Providers: OpenRouter + DSE-nimbus (`qwen`) only**, from a ConfigMap
+    mounted over `/app/config.json` in the pod. No NRP ELLM, no Anthropic direct.
+  - **Logs to the in-cluster MinIO mirror** (`minio-svc.minio.svc.cluster.local:9000`,
+    bucket `logs-open-llm-proxy`), via the existing `LOG_BUCKET` /
+    `AWS_S3_ENDPOINT_URL` env vars and a MinIO service account scoped to that
+    bucket — not the MinIO root user. **Raw JSONL tier only:** the Parquet rollup
+    and `sessions/**` view live inside the NRP CronJob manifests and can't be
+    reused without extracting them, which would touch NRP. Deferred.
+  - **Traefik replaces HAProxy** for ingress: CORS and the 600s backend timeouts
+    become `Middleware` / `ServersTransport` CRDs (`cirrus/middleware.yaml`), with
+    cert-manager + external-dns provisioning TLS and the A record.
+  - No dns-cache sidecar / `dnsPolicy: None` (that works around NRP CoreDNS
+    flakiness, #28); pinned to the `cirrus` node, where MinIO and Traefik live.
+  - Known limitation, inherent to staying config-only: an unrouted model id
+    returns `500` rather than a useful `400`, because
+    `get_provider_for_model`'s fallback is a hard-coded `PROVIDERS["nrp"]` and
+    cirrus has no `nrp` provider. Correct model ids are unaffected.
+
 - **Route OpenRouter floating aliases (`~…`).** OpenRouter publishes always-latest
   aliases whose model id carries a literal leading tilde — e.g.
   `~deepseek/deepseek-v4-flash-latest` (canonical slug identical), as distinct from

LOGGING.md

@@ -7,6 +7,12 @@ Logs are written to two places:
 1. **Pod stdout** — available immediately via `kubectl`, lost on pod restart
 2. **S3 bucket `logs-open-llm-proxy`** — flushed every 60 seconds (configurable via `FLUSH_INTERVAL` env var) as JSONL chunk files, persisted indefinitely
 
+> This document describes the **NRP** deployment (bucket on NRP Ceph). The cirrus
+> deployment writes the same record format to a same-named bucket on the
+> in-cluster MinIO mirror, but has **only the raw JSONL tier** — the Parquet
+> rollup and `sessions/**` view below are NRP-only for now. See
+> [cirrus/README.md](cirrus/README.md).
+
 ### S3 layout (tiered rollup)
 
 Three tiers, each holding a different age of data. A daily CronJob rolls

README.md

@@ -4,6 +4,8 @@ OpenAI-compatible LLM proxy for NRP Nautilus. Routes requests to LLM providers,
 
 The canonical deployment lives at `https://open-llm-proxy.nrp-nautilus.io` in the `biodiversity` namespace, but the proxy is designed to be deployed independently into any namespace — each instance writes logs to its own namespace-scoped S3 bucket.
 
+A second deployment runs on the lab's self-hosted k3s cluster at `https://llm-proxy.carlboettiger.info` (OpenRouter + DSE-nimbus only, logging to the in-cluster MinIO mirror). It is **config-only** — it runs this repo's unmodified code from `main` with a ConfigMap mounted over `config.json`, and changes nothing the NRP deployment reads. Manifests and operations: [cirrus/](cirrus/README.md).
+
 ## Architecture
 
 ```

cirrus/README.md

@@ -0,0 +1,198 @@
+# open-llm-proxy on cirrus (self-hosted k3s)
+
+A second deployment of the same proxy, on the lab's own k3s cluster instead of
+NRP. Public endpoint:
+
+    https://llm-proxy.carlboettiger.info
+
+## Config-only, by design
+
+**This directory contains no application code and changes nothing the NRP
+deployment reads.** The pods clone `main` and run the *same* `llm_proxy.py` as
+NRP; everything cirrus-specific is expressed as deployment config:
+
+- the provider set comes from a **ConfigMap mounted over `/app/config.json`**
+  (`config-configmap.yaml`) — no `config.json` edit, no new config file in the
+  repo, no code change;
+- the log destination comes from `LOG_BUCKET` + `AWS_S3_ENDPOINT_URL`, env vars
+  the app already supported;
+- ingress/CORS/timeouts are Traefik CRDs, entirely outside the app.
+
+The upside is that **there is nothing to diverge**: cirrus cannot drift from
+NRP's code, because it has none of its own, and a change made for cirrus cannot
+regress NRP, because there is nothing to change. The cost is that cirrus gets
+only what upstream `llm_proxy.py` already does — see *Known limitations* below.
+That trade is deliberate: features cirrus wants should land in the shared app
+through the normal release cycle, not as a cirrus fork.
+
+Keep it that way. If a future cirrus need seems to require an app change, the
+right move is to propose that change on its own merits for both deployments —
+not to point these manifests at a branch.
+
+## How it differs from NRP
+
+| | NRP (`../deployment.yaml`) | cirrus (this directory) |
+|---|---|---|
+| Host | `open-llm-proxy.nrp-nautilus.io` | `llm-proxy.carlboettiger.info` |
+| Namespace | `biodiversity` | `llm-proxy` |
+| Providers | nrp ELLM, OpenRouter, Anthropic, nimbus, gemma4-nimbus, qwen3-cirrus | **OpenRouter + DSE-nimbus only** |
+| Ingress / CORS | HAProxy annotations | Traefik `Middleware` CRD (`middleware.yaml`) |
+| Log store | NRP Ceph `s3://logs-open-llm-proxy` | in-cluster MinIO `s3://logs-open-llm-proxy` |
+| Log tiers | raw JSONL → daily Parquet → monthly Parquet + `sessions/**` | **raw JSONL only** (see below) |
+| Replicas | 3, spread across nodes | 2, pinned to `cirrus` |
+
+## Models
+
+Routing is exact-match-then-prefix over the mounted config:
+
+- `qwen` → DSE-nimbus (`https://vllm-nimbus.carlboettiger.info`,
+  `nvidia/Qwen3.6-35B-A3B-NVFP4`), with `enable_thinking` supported.
+- `anthropic/…`, `openai/…`, `qwen/…`, `deepseek/…` and the other vendor
+  prefixes, plus `~…` floating aliases → OpenRouter.
+
+## Known limitations
+
+Both follow directly from the config-only rule, and both are fixable upstream
+whenever the shared app is touched for other reasons:
+
+1. **An unrecognized model id returns `500`, not a helpful `400`.** Upstream
+   `get_provider_for_model` falls back to `PROVIDERS["nrp"]` unconditionally,
+   and cirrus has no `nrp` provider, so the lookup raises. It only affects
+   typo'd/unrouted model names; correct ids are unaffected. (Fixing it properly
+   means making the fallback provider configurable in the shared app.)
+2. **No Parquet consolidation.** The daily/monthly rollup and the `sessions/**`
+   per-turn view are implemented as ~550 lines of Python embedded in the NRP
+   CronJob manifests, which cannot be reused without extracting them into the
+   repo — an NRP-affecting change. So cirrus keeps **raw JSONL indefinitely**:
+   nothing is lost, queries just get slower as volume grows, and the
+   query-ready session view isn't available. Revisit when that extraction
+   happens upstream.
+
+## Deploy
+
+```bash
+kubectl apply -f cirrus/namespace.yaml
+# secrets — see below, one time
+kubectl apply -f cirrus/config-configmap.yaml
+kubectl apply -f cirrus/middleware.yaml
+kubectl apply -f cirrus/service.yaml
+kubectl apply -f cirrus/deployment.yaml
+kubectl apply -f cirrus/ingress.yaml
+```
+
+The app is **git-cloned at pod boot** (no image build), so shipping a change is:
+
+```bash
+kubectl -n llm-proxy rollout restart deployment/open-llm-proxy
+```
+
+That picks up whatever is on `main` — the same code NRP runs. A ConfigMap edit
+also needs a restart (subPath mounts don't live-update).
+
+## Secrets (one time)
+
+Four secrets in the `llm-proxy` namespace:
+
+```bash
+# Client auth. Generate: python3 -c "import secrets; print(secrets.token_urlsafe(32))"
+kubectl -n llm-proxy create secret generic open-llm-proxy-secrets \
+  --from-literal=proxy-key='...'
+# optional additional revocable keys, comma-separated:
+#   --from-literal=proxy-keys-extra='key1,key2'
+
+kubectl -n llm-proxy create secret generic openrouter-key \
+  --from-literal=OPENROUTER_KEY='sk-or-v1-...'
+
+# Same vLLM API key the DSE-nimbus endpoint uses (also in vllm/vllm-api-key)
+kubectl -n llm-proxy create secret generic nimbus-api-key \
+  --from-literal=NIMBUS_API_KEY='...'
+
+# MinIO service account for the log bucket (see next section)
+kubectl -n llm-proxy create secret generic logs-s3 \
+  --from-literal=AWS_ACCESS_KEY_ID='llm-proxy-logs' \
+  --from-literal=AWS_SECRET_ACCESS_KEY='...'
+```
+
+Read the current client key back with:
+
+```bash
+kubectl -n llm-proxy get secret open-llm-proxy-secrets \
+  -o jsonpath='{.data.proxy-key}' | base64 -d; echo
+```
+
+## Log bucket + MinIO service account (one time)
+
+Logs go to the in-cluster MinIO mirror, in a **private** bucket, written by a
+service account scoped to that bucket alone — not the MinIO root user. Created
+via `mc` inside the MinIO pod:
+
+```bash
+POD=$(kubectl -n minio get pod -l k8s-app=minio-app -o name | head -1)
+kubectl -n minio exec "$POD" -- sh -c '
+  mc alias set local http://127.0.0.1:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"
+  mc mb --ignore-existing local/logs-open-llm-proxy
+  mc anonymous set none local/logs-open-llm-proxy
+  cat > /tmp/p.json <<EOF
+{"Version":"2012-10-17","Statement":[
+ {"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],
+  "Resource":["arn:aws:s3:::logs-open-llm-proxy"]},
+ {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],
+  "Resource":["arn:aws:s3:::logs-open-llm-proxy/*"]}]}
+EOF
+  mc admin policy create local llm-proxy-logs-rw /tmp/p.json
+  mc admin user add local llm-proxy-logs "<SECRET>"
+  mc admin policy attach local llm-proxy-logs-rw --user llm-proxy-logs
+'
+```
+
+(`DeleteObject` is not needed today — nothing prunes the raw tier — but is
+included so a future consolidation job can delete chunks after rolling them up.)
+
+## Querying the logs
+
+Raw JSONL only, same record format as NRP (see [../LOGGING.md](../LOGGING.md)):
+
+```
+logs-open-llm-proxy/
+└── 2026-08-06/
+    ├── 02-08-45-<pod>-<pid>-<uuid>.jsonl
+    └── ...
+```
+
+MinIO is reachable at `https://minio.carlboettiger.info` from outside the
+cluster:
+
+```bash
+export AWS_ACCESS_KEY_ID=llm-proxy-logs
+export AWS_SECRET_ACCESS_KEY=...       # from the logs-s3 secret
+
+duckdb -s "
+CREATE SECRET minio (TYPE S3, KEY_ID getenv('AWS_ACCESS_KEY_ID'),
+                     SECRET getenv('AWS_SECRET_ACCESS_KEY'),
+                     ENDPOINT 'minio.carlboettiger.info', URL_STYLE 'path');
+SELECT timestamp, model, user_message_this_turn, latency_ms
+FROM read_ndjson_auto('s3://logs-open-llm-proxy/2026-*/*.jsonl', union_by_name=true)
+ORDER BY timestamp DESC LIMIT 20;
+"
+```
+
+`union_by_name=true` matters here: request and response records have different
+field sets, and the raw tier never went through the schema-flattening step.
+
+Live tail:
+
+```bash
+kubectl -n llm-proxy logs deployment/open-llm-proxy -f
+```
+
+## Smoke test
+
+```bash
+KEY=$(kubectl -n llm-proxy get secret open-llm-proxy-secrets \
+        -o jsonpath='{.data.proxy-key}' | base64 -d)
+
+curl -s https://llm-proxy.carlboettiger.info/health | jq
+curl -s https://llm-proxy.carlboettiger.info/v1/chat/completions \
+  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
+  -d '{"model":"qwen","messages":[{"role":"user","content":"Say hi."}]}' | jq -r '.choices[0].message.content'
+```

cirrus/config-configmap.yaml

@@ -0,0 +1,51 @@
+# Provider set for the cirrus deployment, mounted OVER /app/config.json in the
+# pod (see deployment.yaml). This is why the cirrus deployment needs no code
+# change and no repo-level config change: `llm_proxy.py` always reads
+# `config.json` next to itself, and this ConfigMap replaces that one file after
+# the git-sync initContainer has laid down the checkout.
+#
+# Keep this list a strict subset of the NRP config.json's providers — it exists
+# to *narrow* the surface (OpenRouter + DSE-nimbus), not to diverge from it. If
+# a provider's endpoint or model list changes upstream, mirror it here.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: open-llm-proxy-config
+  namespace: llm-proxy
+data:
+  config.json: |
+    {
+      "providers": {
+        "openrouter": {
+          "endpoint": "https://openrouter.ai/api/v1/chat/completions",
+          "api_key_env": "OPENROUTER_KEY",
+          "models": [
+            "anthropic/",
+            "mistralai/",
+            "amazon/",
+            "openai/",
+            "qwen/",
+            "nvidia/",
+            "z-ai/",
+            "minimax/",
+            "moonshotai/",
+            "deepseek/",
+            "~"
+          ],
+          "extra_headers": {
+            "HTTP-Referer": "https://llm-proxy.carlboettiger.info",
+            "X-Title": "Open LLM Proxy (cirrus)"
+          }
+        },
+        "nimbus": {
+          "endpoint": "https://vllm-nimbus.carlboettiger.info/v1/chat/completions",
+          "api_key_env": "NIMBUS_API_KEY",
+          "models": [
+            "qwen"
+          ],
+          "thinking_models": {
+            "qwen": "enable_thinking"
+          }
+        }
+      }
+    }