fix(anythingllm): caddy /var/log perms + ANYTHINGLLM_IMAGE bootstrap prompt

Post-deploy follow-ups from the INT-S004 rebuild:

1) Hardened non-root caddy:2 crash-looped with "permission denied" writing
   /var/log/caddy/<site>.log — nothing created that host bind-mount dir
   writable. ansible/deploy.yml now ensures /var/log/caddy (0777) before
   `docker compose up` (template + s004.ccc.bot + ai.weown.agency).

2) ANYTHINGLLM_IMAGE is now a required Infisical secret (compose reads it
   fail-loud). bootstrap-s004-infisical.sh prompts for it (plain read — not a
   secret, but carries the private registry namespace, so it stays in Infisical
   not the repo), and the s004.ccc.bot README documents it.

Render parity (template==site deploy.yml), bash -n, YAML all verified.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: ncimino

CHANGELOG.md

@@ -60,6 +60,7 @@ Changes in this section will be promoted to a dated release entry on merge to `m
 
 ### Fixed
 
+- **Post-deploy follow-ups: caddy log-dir perms + `ANYTHINGLLM_IMAGE` bootstrap (2026-06-02)** — (1) the hardened (non-root) `caddy:2` crash-looped with `permission denied` writing `/var/log/caddy/<site>.log` because nothing created that host bind-mount dir writable; `ansible/deploy.yml` now ensures `/var/log/caddy` (mode 0777) before `docker compose up` (template + `s004.ccc.bot` + `ai.weown.agency`). (2) `scripts/bootstrap-s004-infisical.sh` now prompts for the now-required `ANYTHINGLLM_IMAGE`, and the `s004.ccc.bot` README documents it, so a rebuild repopulates the image ref in Infisical.
 - **AnythingLLM published no host port → deploy health probe + runbook tunnel could never connect (2026-06-02)** — the `anythingllm` service had no `ports:` (caddy proxies it over the internal Docker network), yet `ansible/deploy.yml`'s health task polls `http://127.0.0.1:3001/api/ping` on the *host* and the runbook's Phase-4 validation uses `ssh -L 3001:127.0.0.1:3001` — both assume 3001 is published on the droplet loopback. The container was healthy the whole time (`docker inspect` health=healthy, internal ping `{"online":true}`); the probe was checking an unpublished port and failed with `Connection refused`. Added `ports: ["127.0.0.1:3001:3001"]` (loopback-only — not internet-exposed; caddy still serves the public via the internal network) to the template + `s004.ccc.bot` + `ai.weown.agency`.
 - **AnythingLLM image pin `1.7.2` was invalid + hardcoded a private registry path (2026-06-02)** — the compose pinned `reg.mini.dev/anythingllm:1.7.2`, but that tag was never in the Minimus registry (the source box actually ran `:latest`, and Minimus rotates tags + had moved the image to a private namespaced path), so the deploy 404'd on image pull. The image ref is now injected from Infisical (`ANYTHINGLLM_IMAGE`, e.g. `reg.mini.dev/<ns>/anythingllm:v1.12.1`): compose reads `${ANYTHINGLLM_IMAGE:?…}` (fail-loud) and `docker compose up` pulls it under `infisical run`, so the private namespace never enters this public repo and each site pins its own version. The deploy's image-pull task now pulls only the static `caddy` ref. Applied to the template + `s004.ccc.bot` + `ai.weown.agency`; render parity + compose interpolation verified.
 - **Infisical machine-identity auth hung on headless boxes — `infisical run` fell back to an interactive login prompt (2026-06-02)** — `ansible/deploy.yml`, `scripts/backup.sh`, and `scripts/restore.sh` ran `infisical login --method=universal-auth …` followed by a bare `infisical run …`, but a universal-auth (machine identity) login does **not** leave a session that `infisical run` reuses. `run` therefore saw no auth and dropped to the interactive `Select hosting option / Email / Password` prompt, which hangs forever on a no-TTY ansible/cron run (`Login via browser failed. inappropriate ioctl for device`). Fixed by capturing the token — `export INFISICAL_TOKEN="$(infisical login … --plain --silent)"` — which `infisical run` reads directly (the documented headless/CI pattern). Applied to the template + `s004.ccc.bot` + `ai.weown.agency`; render parity and `bash -n` verified.

anythingllm-docker/sites/ai.weown.agency/ansible/deploy.yml

@@ -154,6 +154,18 @@
       delay: 5
       until: image_pull is succeeded
 
+    - name: Ensure caddy log dir is writable (hardened caddy runs non-root)
+      # caddy:2 (Minimus-hardened) runs as a non-root user and writes access
+      # logs to the /var/log/caddy host bind mount. Without a writable dir it
+      # crash-loops with "permission denied". 0777 is acceptable for a
+      # single-tenant log dir; otel-agent also reads these files.
+      ansible.builtin.file:
+        path: /var/log/caddy
+        state: directory
+        owner: root
+        group: root
+        mode: "0777"
+
     - name: Ensure compose stack is running (idempotent — `up -d` is a no-op if nothing changed)
       ansible.builtin.shell: |
         set -euo pipefail

anythingllm-docker/sites/s004.ccc.bot/README.md

@@ -62,8 +62,10 @@ bash scripts/bootstrap-s004-infisical.sh
 ```
 
 It sets `JWT_SECRET` (generated, never rotate), `OPENROUTER_API_KEY` (a **fresh**
-key — the old one expired 2026-06-01), `ADMIN_EMAIL`, `SPACES_ACCESS_KEY`,
-`SPACES_SECRET_KEY`.
+key — the old one expired 2026-06-01), `ADMIN_EMAIL`, `ANYTHINGLLM_IMAGE` (the
+container image ref, e.g. `reg.mini.dev/<ns>/anythingllm:v1.12.1` — kept in
+Infisical so the private registry namespace stays out of this public repo),
+`SPACES_ACCESS_KEY`, `SPACES_SECRET_KEY`.
 
 ### 2. Provision (terraform — first-boot bootstrap)
 

anythingllm-docker/sites/s004.ccc.bot/ansible/deploy.yml

@@ -220,6 +220,18 @@
       delay: 5
       until: image_pull is succeeded
 
+    - name: Ensure caddy log dir is writable (hardened caddy runs non-root)
+      # caddy:2 (Minimus-hardened) runs as a non-root user and writes access
+      # logs to the /var/log/caddy host bind mount. Without a writable dir it
+      # crash-loops with "permission denied". 0777 is acceptable for a
+      # single-tenant log dir; otel-agent also reads these files.
+      ansible.builtin.file:
+        path: /var/log/caddy
+        state: directory
+        owner: root
+        group: root
+        mode: "0777"
+
     - name: Ensure compose stack is running (idempotent — `up -d` is a no-op if nothing changed)
       ansible.builtin.shell: |
         set -euo pipefail

anythingllm-docker/sites/s004.ccc.bot/scripts/bootstrap-s004-infisical.sh

@@ -117,6 +117,13 @@ _maybe_push_secret OPENROUTER_API_KEY "Paste the OpenRouter key value (blank to
 read -rp "  ADMIN_EMAIL (blank to skip): " ADMIN_EMAIL
 if [ -n "${ADMIN_EMAIL:-}" ]; then _push ADMIN_EMAIL "$ADMIN_EMAIL"; else echo "  • skipped ADMIN_EMAIL (left blank)"; fi
 
+# ANYTHINGLLM_IMAGE - the container image ref. NOT a secret, but it carries the
+# private registry namespace, so it lives in Infisical (not this public repo).
+# Required: compose reads ${ANYTHINGLLM_IMAGE} fail-loud and the deploy pulls it
+# under `infisical run`. plain read (not a secret); blank = skip.
+read -rp "  ANYTHINGLLM_IMAGE (e.g. reg.mini.dev/<ns>/anythingllm:v1.12.1; blank to skip): " ANYTHINGLLM_IMAGE
+if [ -n "${ANYTHINGLLM_IMAGE:-}" ]; then _push ANYTHINGLLM_IMAGE "$ANYTHINGLLM_IMAGE"; else echo "  • skipped ANYTHINGLLM_IMAGE (left blank)"; fi
+
 # SPACES_* — required for offsite backups; blank = skip.
 _maybe_push_secret SPACES_ACCESS_KEY "SPACES_ACCESS_KEY (DO Spaces, for backups; blank to skip): "
 _maybe_push_secret SPACES_SECRET_KEY "SPACES_SECRET_KEY (blank to skip): "

anythingllm-docker/template/ansible/deploy.yml.jinja

@@ -220,6 +220,18 @@
       delay: 5
       until: image_pull is succeeded
 
+    - name: Ensure caddy log dir is writable (hardened caddy runs non-root)
+      # caddy:2 (Minimus-hardened) runs as a non-root user and writes access
+      # logs to the /var/log/caddy host bind mount. Without a writable dir it
+      # crash-loops with "permission denied". 0777 is acceptable for a
+      # single-tenant log dir; otel-agent also reads these files.
+      ansible.builtin.file:
+        path: /var/log/caddy
+        state: directory
+        owner: root
+        group: root
+        mode: "0777"
+
     - name: Ensure compose stack is running (idempotent — `up -d` is a no-op if nothing changed)
       ansible.builtin.shell: |
         set -euo pipefail