security(backend): --read-only rootfs + full-boot pre-cutover gate (BOOT_CHECK_ONLY)

Implements the read-only-rootfs item deferred at backend-deploy.yml (the old
note at :263-265): the serving container now runs --read-only, and the deploy
ships a pre-cutover gate that validates the FULL boot under the exact flags
before cutover, closing the "gate exits before the app phase" gap that forced
the deferral.

ExecStart + gate both gain --read-only and the shared tmpfs set:
  /tmp (rw,size=512m,mode=1777)          Chromium temp profiles + shm redirect,
                                         nft mktemp, /tmp/fingpt_cache
  /app/staticfiles (uid=1001,gid=1001)   collectstatic at boot
  /home/fingpt (uid=1001,gid=1001)       fontconfig, ~/.edgar, yfinance (uid pin
                                         or the #331 MCP-child EACCES returns)
/app/logs + /app/media stay non-tmpfs on purpose: zero writers today, so a
future stray write must fail LOUDLY. /dev/shm is podman-managed and unaffected.

Gate redesign (entrypoint.sh BOOT_CHECK_ONLY=1, wrapping the final exec):
probes every writable surface + $HOME (covers the fail-SOFT MCP-child writes
/health/ can never catch), backgrounds the image's own CMD byte-identical (no
positional args on the podman line: zero-drift), polls /health/ inside the
container (~90s window), bounded SIGTERM+watchdog-SIGKILL shutdown. Subsumes
--egress-check-only, which stays for manual/local use. Deliberate asymmetries,
pinned by sentinels: gate /app/runtime is a THROWAWAY tmpfs (the real bind
mount's :U re-chown + os.replace() of the shared DuckDB are unsafe under the
serving container) and gate REDIS_URL uses throwaway DB15. Accepted residual:
the :U/:Z relabel path (#322 class) stays covered only by the post-cutover
~130s health window. Deploy step gets an explicit command_timeout: 20m.

Dockerfile: pre-bake Playwright's DEPENDENCIES_VALIDATED marker via a real
one-time chromium launch as uid1001 (the driver's marker write is fail-soft
under --read-only but would force per-process revalidation), with a loud
test -f build guard.

Also fixes the pre-existing prod break where /api/sync_preferred_urls/ (used
by the shipped extension) 500s: /app/data is root-owned, so the manager write
EACCESes. Storage path is now env-configurable (PREFERRED_LINKS_PATH), pinned
to /app/runtime/preferred_links.json in the image, seeded on first use from
the in-image default file; local dev keeps the in-repo path.

Sentinels extended in test_dockerfile_nonroot.py (flag extractor widened to
tmpfs/memory/read-only with equals-form pinning, parity + closed asymmetry
list, frozen tmpfs target sets, boot-check branch structure, marker bake) and
new tests/test_preferred_links_manager.py (env override, seeding, fallback,
write path, Dockerfile pin).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: FlyM1ss

.github/workflows/backend-deploy.yml

@@ -172,6 +172,10 @@ jobs:
           username: ${{ env.DROPLET_USER }}
           key: ${{ secrets.DEPLOY_SSH_KEY }}
           port: 22
+          # The default 10m is now shared by the multi-GB image pull, the pre-cutover
+          # FULL-boot gate (store build + collectstatic + gunicorn up + /health/ poll,
+          # ~2-3min), cutover, and image retirement -- give it explicit headroom.
+          command_timeout: 20m
           envs: REMOTE_IMAGE,GHCR_USERNAME,GHCR_TOKEN,SYSTEMD_UNIT
           script: |
             set -euo pipefail
@@ -202,28 +206,44 @@ jobs:
               podman start fingpt-redis >/dev/null 2>&1 || true
             fi
 
-            # Pre-cutover firewall validation: run the NEW image's REAL entrypoint in
-            # check-only mode on fingpt-net BEFORE the systemctl restart's --replace kills
-            # the currently-serving container. --egress-check-only exercises the byte-
-            # identical root-init PID1 runs at cutover (generate + sentinel checks + nft
-            # load + transition-proof self-test + setpriv drop), then exits before the app
-            # phase; an inline re-copy of that sequence (this step's previous shape) can
-            # silently drift from the boot path it is supposed to prove. On failure, abort
-            # so the OLD container keeps serving (there is no auto-rollback). This closes
-            # the CI-invisible gap: the test job is pytest-only and never touches nft/netns.
-            # The gate runs under the IDENTICAL frozen cap/pids flags as the ExecStart
-            # below (cap-to-step map at the override; parity pinned in
-            # Main/backend/tests/test_dockerfile_nonroot.py). That parity IS the safety
-            # property: check-only mode exits right after the setpriv drop, so all five
-            # caps the root-init needs (nft + chown + setuid/setgid + bounding-set drop)
-            # are exercised HERE -- a too-narrow set aborts the deploy with the old
-            # container still serving, instead of crash-looping the new unit at cutover.
-            echo "Validating egress firewall on the new image (pre-cutover)..."
-            podman run --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --network fingpt-net \
-              --env REDIS_URL=redis://fingpt-redis:6379/0 \
-              "$REMOTE_IMAGE" --egress-check-only \
-              || { echo "ERROR: egress firewall pre-cutover validation FAILED; aborting deploy (old container left serving)"; exit 1; }
-            echo "Egress firewall pre-cutover validation passed."
+            # Pre-cutover FULL-BOOT validation: run the NEW image's REAL entrypoint on
+            # fingpt-net BEFORE the systemctl restart's --replace kills the currently-
+            # serving container. BOOT_CHECK_ONLY=1 (mode lives at the END of
+            # Main/backend/entrypoint.sh, wrapping the final exec) exercises the byte-
+            # identical PID1 path that runs at cutover: the full root-init (generate +
+            # sentinel checks + nft load + transition-proof self-test + setpriv drop --
+            # so this SUBSUMES the old --egress-check-only gate, which stays in the
+            # entrypoint for manual/local use), then the full app phase (store build,
+            # collectstatic, Playwright verify), then explicit writable-surface probes
+            # over the tmpfs set (these cover the fail-SOFT MCP-child $HOME writes a
+            # /health/ poll can never catch), then backgrounds the image's own CMD and
+            # polls /health/ INSIDE the container until gunicorn answers (hence no
+            # --publish: the gate's :8000 stays netns-private). No positional args
+            # after the image ref, so the Dockerfile CMD flows through byte-identical
+            # (zero-drift property). On failure, abort so the OLD container keeps
+            # serving (there is no auto-rollback). This closes the CI-invisible gap:
+            # the test job is pytest-only and never touches nft/netns/--read-only.
+            # The gate runs the IDENTICAL frozen cap/pids/read-only/tmpfs/memory flags
+            # as the ExecStart below (cap-to-step map + tmpfs map at the override;
+            # parity pinned in Main/backend/tests/test_dockerfile_nonroot.py), with
+            # exactly one deliberate asymmetry: /app/runtime here is a THROWAWAY
+            # tmpfs, never the real /home/deploy/fingpt/runtime bind mount. Mounting
+            # the real dir pre-cutover is dangerous: :U would re-chown it under the
+            # serving container, and a new-recipe store build would os.replace() the
+            # shared DuckDB while the old container still serves it
+            # (truthlayer/retrieve.py). Same isolation for redis: DB15 is a throwaway
+            # keyspace so gate boot-time writes can never touch the serving
+            # container's rate-limit/budget counters in DB0 (only settings_prod.py
+            # and ops/egress_firewall.py consume REDIS_URL, and the firewall
+            # self-test checks host:port reachability only, DB-agnostic).
+            echo "Validating full boot (root-init + read-only rootfs + /health/) on the new image (pre-cutover)..."
+            podman run --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --read-only --tmpfs=/tmp:rw,size=512m,mode=1777 --tmpfs=/app/staticfiles:rw,uid=1001,gid=1001 --tmpfs=/home/fingpt:rw,uid=1001,gid=1001 --tmpfs=/app/runtime:rw,size=512m --memory=1.7g --memory-swap=2g --network fingpt-net \
+              --env-file /home/deploy/fingpt/envs/.env.production \
+              --env REDIS_URL=redis://fingpt-redis:6379/15 \
+              --env BOOT_CHECK_ONLY=1 \
+              "$REMOTE_IMAGE" \
+              || { echo "ERROR: full-boot pre-cutover validation FAILED; aborting deploy (old container left serving)"; exit 1; }
+            echo "Full-boot pre-cutover validation passed."
 
             # Record the currently-deployed image for manual rollback (no auto-rollback).
             PREV_IMAGE=$(podman inspect --format '{{.ImageName}}' "$SYSTEMD_UNIT" 2>/dev/null || echo "unknown")
@@ -260,15 +280,34 @@ jobs:
             # --pids-limit=1024 is the fork-bomb ceiling: worst case today is ~400-600
             # tasks (gunicorn workers x threads + agent fan-out + Playwright/Chromium);
             # revisit if AGENT_MAX_CONCURRENCY or GUNICORN_WORKERS is raised.
-            # A read-only rootfs is DELIBERATELY deferred: the gate exits before the app
-            # phase, so --read-only would ship unvalidated pre-cutover -- and collectstatic
-            # writes /app/staticfiles at boot, which needs a tmpfs/volume design first.
+            # --read-only rootfs: the ONLY writable surfaces are the tmpfs set below,
+            # /dev/shm (podman-managed 64MB tmpfs; gunicorn --worker-tmp-dir, untouched
+            # by --read-only), and the /app/runtime bind mount. The tmpfs map:
+            #   /tmp              Chromium temp profiles + shm redirect
+            #                     (--disable-dev-shm-usage), nft mktemp at root-init,
+            #                     /tmp/fingpt_cache; sized + world-writable-sticky;
+            #   /app/staticfiles  collectstatic writes it at boot (0 files today, kept
+            #                     writable as future-proofing); uid1001 tmpfs;
+            #   /home/fingpt      fontconfig cache, edgartools ~/.edgar import-time
+            #                     marker, yfinance cache; MUST carry uid=1001,gid=1001
+            #                     or the MCP-child EACCES bug (#331 class) returns.
+            # /app/logs and /app/media are vestigial with ZERO writers -- deliberately
+            # NOT tmpfs, so a future stray write fails LOUDLY instead of vanishing
+            # into RAM. Playwright's DEPENDENCIES_VALIDATED marker is pre-baked at
+            # image build (Dockerfile) so first browser launch needs no rootfs write.
+            # The pre-cutover gate above boots the FULL app under these exact flags
+            # (BOOT_CHECK_ONLY=1) before cutover, so a missing/mis-owned tmpfs aborts
+            # the deploy with the old container still serving. Accepted residual: the
+            # gate's /app/runtime is a throwaway tmpfs, so the :U chown / :Z SELinux
+            # relabel path of the REAL bind mount (#322 class) is NOT exercised
+            # pre-cutover; it stays covered only by the post-cutover ~130s health
+            # window in the Verify step below.
             OVERRIDE_DIR="$HOME/.config/systemd/user/${SYSTEMD_UNIT}.service.d"
             mkdir -p "$OVERRIDE_DIR"
             cat > "$OVERRIDE_DIR/override.conf" <<EOF
             [Service]
             ExecStart=
-            ExecStart=/usr/bin/podman run --name ${SYSTEMD_UNIT} --replace --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --cgroups=split --sdnotify=conmon -d --memory=1.7g --memory-swap=2g --network fingpt-net -v /home/deploy/fingpt/runtime:/app/runtime:U,Z --publish 127.0.0.1:8000:8000 --env-file /home/deploy/fingpt/envs/.env.production --env REDIS_URL=redis://fingpt-redis:6379/0 ${REMOTE_IMAGE}
+            ExecStart=/usr/bin/podman run --name ${SYSTEMD_UNIT} --replace --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --read-only --tmpfs=/tmp:rw,size=512m,mode=1777 --tmpfs=/app/staticfiles:rw,uid=1001,gid=1001 --tmpfs=/home/fingpt:rw,uid=1001,gid=1001 --cgroups=split --sdnotify=conmon -d --memory=1.7g --memory-swap=2g --network fingpt-net -v /home/deploy/fingpt/runtime:/app/runtime:U,Z --publish 127.0.0.1:8000:8000 --env-file /home/deploy/fingpt/envs/.env.production --env REDIS_URL=redis://fingpt-redis:6379/0 ${REMOTE_IMAGE}
             EOF
 
             systemctl --user daemon-reload

Main/backend/Dockerfile

@@ -40,7 +40,8 @@ ENV PATH="/app/.venv/bin:$PATH" \
     PIP_NO_CACHE_DIR=1 \
     PIP_DISABLE_PIP_VERSION_CHECK=1 \
     TRUTHLAYER_DB_PATH=/app/runtime/truthlayer.duckdb \
-    HOME=/home/fingpt
+    HOME=/home/fingpt \
+    PREFERRED_LINKS_PATH=/app/runtime/preferred_links.json
 
 COPY . .
 
@@ -83,6 +84,33 @@ RUN groupadd --system --gid 1001 fingpt \
     && mkdir -p /home/fingpt \
     && chown -R fingpt:fingpt /app/staticfiles /app/media /app/logs /tmp/fingpt_cache /app/runtime /home/fingpt
 
+# Pre-bake Playwright's DEPENDENCIES_VALIDATED marker so the --read-only rootfs
+# (backend-deploy.yml) never needs the write. At FIRST browser launch Playwright
+# validates host requirements (an ldd walk over the browser executable) and caches
+# success by writing a 0-byte DEPENDENCIES_VALIDATED marker next to the executable
+# (driver/package/lib/server/registry/index.js, _validateHostRequirementsForExecutable-
+# IfNeeded). The write itself is fail-SOFT (`.catch(() => {})`), so under --read-only
+# nothing crashes -- but the marker never lands and every fresh process re-runs the
+# validation on its first launch. Bake it organically instead: a real one-time
+# chromium launch as the runtime uid (setpriv, same drop the entrypoint performs;
+# there is deliberately no USER line), i.e. exactly the code path that writes the
+# marker in prod today. /ms-playwright ships world-writable in the Playwright base
+# image, so uid1001 can write it. --no-sandbox/--disable-dev-shm-usage: the docker
+# build environment lacks the userns the Chromium sandbox wants and its /dev/shm is
+# tiny; both flags only shape the spawned browser process, not the dependency
+# validation that writes the marker. The trailing `test -f` fails the BUILD loudly
+# if Playwright ever moves the marker. Residual: the marker's validity window is 30
+# days from its mtime (= image build time); past that, Playwright re-validates once
+# per process and fail-soft skips the re-write -- back to pre-marker behavior,
+# never a crash.
+RUN setpriv --reuid=1001 --regid=1001 --init-groups \
+        python -c "from playwright.sync_api import sync_playwright; \
+p = sync_playwright().start(); \
+b = p.chromium.launch(args=['--no-sandbox', '--disable-dev-shm-usage']); \
+b.close(); p.stop(); print('✓ Chromium one-time launch OK')" \
+    && test -f /ms-playwright/chromium_headless_shell-*/DEPENDENCIES_VALIDATED \
+    && echo "✓ DEPENDENCIES_VALIDATED marker baked"
+
 ENTRYPOINT ["/app/entrypoint.sh"]
 
 EXPOSE 8000

Main/backend/datascraper/preferred_links_manager.py

@@ -12,6 +12,16 @@
 
 logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
 
+# In-image/in-repo default storage file. Local dev reads AND writes it directly.
+# In the production container /app/data is root-owned and the rootfs is read-only,
+# so writes there EACCES -> 500 on /api/sync_preferred_urls/ (used by the shipped
+# extension). The prod image therefore pins PREFERRED_LINKS_PATH to
+# /app/runtime/preferred_links.json (Dockerfile ENV; /app/runtime is the writable
+# persistent volume) and this file becomes the read-only SEED the runtime copy is
+# initialized from on first use.
+DEFAULT_STORAGE_PATH = Path(__file__).resolve().parent.parent / 'data' / 'preferred_links.json'
+
+
 class PreferredLinksManager:
     """Manages preferred links with JSON file storage."""
 
@@ -21,13 +31,13 @@ def __init__(self, storage_path: str = None):
 
         Args:
             storage_path: Path to the JSON storage file.
-                         Defaults to backend/data/preferred_links.json
+                         Defaults to $PREFERRED_LINKS_PATH if set (the prod image
+                         points it at the writable /app/runtime volume), else
+                         backend/data/preferred_links.json (local dev).
         """
         if storage_path is None:
-            backend_dir = Path(__file__).resolve().parent.parent
-            self.storage_path = backend_dir / 'data' / 'preferred_links.json'
-        else:
-            self.storage_path = Path(storage_path)
+            storage_path = os.environ.get('PREFERRED_LINKS_PATH') or DEFAULT_STORAGE_PATH
+        self.storage_path = Path(storage_path)
 
         self.storage_path.parent.mkdir(parents=True, exist_ok=True)
 
@@ -37,7 +47,25 @@ def __init__(self, storage_path: str = None):
             self._init_storage()
 
     def _init_storage(self):
-        """Initialize the storage file with default structure."""
+        """Initialize the storage file, seeding from the in-image default file.
+
+        When the storage path is redirected off the default (prod: the writable
+        /app/runtime volume) and the runtime copy is missing, seed it from the
+        read-only in-image default file so curated defaults survive the move.
+        Falls back to the hardcoded defaults if the seed is absent or unreadable.
+        """
+        if DEFAULT_STORAGE_PATH.exists() and DEFAULT_STORAGE_PATH != self.storage_path.resolve():
+            try:
+                seed_data = json.loads(DEFAULT_STORAGE_PATH.read_text())
+                self._write_data(seed_data)
+                logging.info(
+                    f"Seeded preferred links storage at {self.storage_path} "
+                    f"from {DEFAULT_STORAGE_PATH}"
+                )
+                return
+            except (json.JSONDecodeError, OSError) as e:
+                logging.error(f"Error seeding preferred links from default file: {e}")
+
         default_links = [
             "https://finance.yahoo.com",
             "https://www.sec.gov/search-filings",

Main/backend/entrypoint.sh

@@ -108,4 +108,57 @@ python -c "from playwright.async_api import async_playwright; print('✓ Playwri
     exit 1
 }
 
+# Deploy pre-cutover FULL-BOOT gate mode (see backend-deploy.yml). By this point the
+# ENTIRE boot has already run under the deploy's exact --read-only/tmpfs/cap flags:
+# root-init (nft load + sentinels + self-test + setpriv drop), the OPENAI key gate,
+# the cache mkdir, the truth-layer store build, collectstatic, and the Playwright
+# verify above -- so this mode SUBSUMES the --egress-check-only gate. It then proves
+# the two things a later /health/ poll alone can never catch:
+#   (a) every writable surface is really writable by uid1001. MCP stdio children
+#       write $HOME fail-SOFT (logger.error in mcp_client/apps.py), so a missing or
+#       root-owned tmpfs there would pass every health check while silently killing
+#       the sec-edgar child -- exactly the EACCES class fixed in #331;
+#   (b) the image's own CMD ("$@", byte-identical: the deploy gate passes no
+#       positional args) comes up and answers /health/ within a bounded window.
+# Placement is load-bearing (pinned by test_dockerfile_nonroot): this branch wraps
+# the final exec, and the fall-through path still ends in exec "$@".
+if [ "${BOOT_CHECK_ONLY:-}" = "1" ]; then
+    echo "Boot check: probing writable surfaces..."
+    for dir in /app/staticfiles /app/runtime "${CACHE_FILE_PATH:-/tmp/fingpt_cache}" "$HOME"; do
+        probe="$dir/.boot-check-probe"
+        ( touch "$probe" && rm -f "$probe" ) 2>/dev/null \
+            || { echo "FATAL: boot check: cannot write $dir (missing/mis-owned tmpfs under --read-only?)" >&2; exit 1; }
+    done
+    echo "Boot check: writable surfaces OK; starting app for /health/ validation..."
+    "$@" &
+    APP_PID=$!
+    HEALTH_OK=0
+    tries=0
+    # ~90s window (30 x 3s; refused connections fail fast), same python-urllib
+    # probe as the Dockerfile HEALTHCHECK.
+    while [ "$tries" -lt 30 ]; do
+        if python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health/', timeout=5)" 2>/dev/null; then
+            HEALTH_OK=1
+            break
+        fi
+        kill -0 "$APP_PID" 2>/dev/null \
+            || { echo "FATAL: boot check: app exited before /health/ answered" >&2; exit 1; }
+        tries=$((tries + 1))
+        sleep 3
+    done
+    # Bounded shutdown: SIGTERM the app, watchdog hard-SIGKILLs if it hangs, so the
+    # gate can never wedge the deploy past its own window.
+    kill "$APP_PID" 2>/dev/null || true
+    ( sleep 20; kill -9 "$APP_PID" 2>/dev/null ) &
+    WATCHDOG_PID=$!
+    wait "$APP_PID" 2>/dev/null || true
+    kill "$WATCHDOG_PID" 2>/dev/null || true
+    if [ "$HEALTH_OK" = "1" ]; then
+        echo "Boot check: /health/ answered; full boot validated under deploy flags."
+        exit 0
+    fi
+    echo "FATAL: boot check: /health/ did not answer within the ~90s window" >&2
+    exit 1
+fi
+
 exec "$@"