fix(engine): cc-debian13 base — runtime glibc must be >= build glibc (WS2-2)

The distroless_cc flip got the engine past the missing loader, but it then
crashed with `GLIBC_2.38 not found` / `GLIBCXX_3.4.32 not found`: the binary,
built on ubuntu-24.04-arm (glibc 2.39, GCC 13), needs symbol versions newer
than cc-debian12 (glibc 2.36) ships.

local_config_cc is non-hermetic, so the binary inherits the CI runner's glibc.
glibc is forward-compatible only -> the runtime base must be >= the build glibc.
"Build on an older runner" does NOT work here: the Bazel remote cache reused the
24.04-built objects across runner versions (same non-hermetic action hash), so
the binary stayed glibc-2.39. Moving the base FORWARD is the sound fix.

- MODULE.bazel: distroless_cc -> cc-debian13:nonroot (glibc 2.41 / GCC 14),
  re-pinned digest.
- release-onprem-image.yml: revert runner to ubuntu-24.04-arm; document the
  "runtime base glibc >= build glibc" rule.
- BUILD.bazel + ADR-0025: record the debian13 correction.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: BinHsu

.github/workflows/release-onprem-image.yml

@@ -15,21 +15,21 @@ name: Release on-prem image to GHCR
 #
 # WHY a native arm64 runner:
 #   The local Talos cluster runs on apple/container = linux/arm64. An amd64
-#   image (the ubuntu-latest default) would not run there. ubuntu-22.04-arm is
+#   image (the ubuntu-latest default) would not run there. ubuntu-24.04-arm is
 #   GitHub's free arm64 Linux runner for public repos. local_config_cc
 #   auto-detects the runner's arm64 GCC — no toolchain / sysroot setup, no
 #   BUILD changes. This is also why the engine build belongs in CI, not on the
 #   16GB dev host: the grpc + BoringSSL + whisper.cpp/ggml/llama.cpp link line
 #   needs more RAM than that host can spare for a build VM (de-risked 2026-06-16).
 #
-# WHY 22.04, not 24.04 (resolved 2026-06-16, WS2-2 live verify):
-#   The runtime base is distroless cc-debian12 (Debian 12, glibc 2.36). A binary
-#   built on ubuntu-24.04 (glibc 2.39, GCC 13 / GLIBCXX_3.4.32) references
-#   symbol versions newer than Debian 12 provides — the engine crashed at
-#   startup with `GLIBC_2.38 not found` / `GLIBCXX_3.4.32 not found`. glibc is
-#   forward-compatible, not backward: build against an OLDER glibc than the
-#   runtime base. ubuntu-22.04 ships glibc 2.35 (<= 2.36) and a libstdc++ no
-#   newer than Debian 12's, so the binary runs. Keep build glibc <= runtime base.
+# RUNTIME-BASE glibc must be >= the build glibc (resolved 2026-06-16, WS2-2):
+#   The engine binary is dynamically linked and local_config_cc is non-hermetic,
+#   so it inherits the runner's glibc (ubuntu-24.04-arm = glibc 2.39, GCC 13 /
+#   GLIBCXX_3.4.32). glibc is forward-compatible, not backward — the runtime
+#   base must therefore ship glibc >= 2.39. distroless cc-debian12 (glibc 2.36)
+#   was too old: the engine crashed at startup with `GLIBC_2.38 not found` /
+#   `GLIBCXX_3.4.32 not found`. The base is now cc-debian13 (Debian 13, glibc
+#   2.41 / GCC 14) — see packaging/engine/BUILD.bazel + MODULE.bazel.
 #
 # Manual trigger only for now (local-first; the operator pulls on demand). A
 # push trigger can be added once the WS2 on-prem flow stabilises.
@@ -56,7 +56,7 @@ env:
 jobs:
   push-onprem-image:
     name: Build + push core images to GHCR (linux/arm64)
-    runs-on: ubuntu-22.04-arm
+    runs-on: ubuntu-24.04-arm
     timeout-minutes: 45
     steps:
       - name: Checkout

MODULE.bazel

@@ -295,21 +295,28 @@ use_repo(
     "distroless_static_linux_arm64_v8",
 )
 
-# Distroless cc-debian12:nonroot — the base for the C++ ENGINE image (the Go
+# Distroless cc-debian13:nonroot — the base for the C++ ENGINE image (the Go
 # gateway stays on distroless_static). The engine's link line (grpc++ + BoringSSL
 # + protobuf + whisper.cpp/ggml/llama.cpp via rules_foreign_cc CMake + libopus)
 # produces a DYNAMICALLY-linked binary — the CMake recipes don't honour -static,
 # so the binary needs an ELF loader + glibc + libstdc++/libgcc_s at runtime.
 # distroless_static ships none of those: the engine image built on it failed at
 # `exec /usr/local/bin/engine: no such file or directory` (missing PT_INTERP)
-# the first time it was actually run (WS2-2 live verify, 2026-06-16). cc-debian12
-# is Google's designated base for dynamically-linked C/C++ — it adds the loader,
-# glibc, libstdc++, and libgcc_s that `base-debian12` (glibc only) still lacks.
-# Pinned by manifest-list digest (resolved 2026-06-16); re-pin on bump.
+# the first time it was actually run (WS2-2 live verify, 2026-06-16). cc-* is
+# Google's designated base for dynamically-linked C/C++ (loader + glibc +
+# libstdc++ + libgcc_s; base-* has glibc only).
+#
+# WHY debian13, not debian12: local_config_cc is non-hermetic, so the binary
+# inherits the CI runner's glibc (ubuntu-24.04-arm = glibc 2.39). glibc is
+# forward-compatible only — the runtime base must be >= the build glibc.
+# cc-debian12 (glibc 2.36) was too old (`GLIBC_2.38 not found` at startup);
+# cc-debian13 ships glibc 2.41 / GCC 14 libstdc++. Pinned by manifest-list
+# digest (resolved 2026-06-16); re-pin on bump. If the build runner ever moves
+# to a newer glibc than debian13, bump this base in lockstep.
 oci.pull(
     name = "distroless_cc",
-    digest = "sha256:b0ae8e989418b458e0f25489bc3be523718938a2b70864cc0f6a00af1ddbd985",
-    image = "gcr.io/distroless/cc-debian12",
+    digest = "sha256:d3cda6e91129130d7229a1806b6a73d292ef245ab032da7851907798024cefba",
+    image = "gcr.io/distroless/cc-debian13",
     platforms = [
         "linux/amd64",
         "linux/arm64/v8",

docs/adr/0025-oci-packaging-strategy.md

@@ -193,12 +193,20 @@ at `exec /usr/local/bin/engine: no such file or directory`. The 79 MB binary
 was present and arm64-correct; ENOENT-on-exec means a missing ELF interpreter
 — i.e. the binary is dynamically linked (rules_foreign_cc CMake deps don't
 honour -static, as the fork-point comment anticipated) and `static-debian12`
-ships no loader. The fix is `cc-debian12`, NOT the `base-debian12` the
-fork-point originally named: a dynamic C++ binary also needs `libstdc++` +
-`libgcc_s`, which `base` (glibc only) lacks; `cc` is Google's designated base
-for dynamically-linked C/C++. The gateway (Go, genuinely static) stays on
-`static-debian12`. Net cost of the lost bet: one extra CI build cycle, exactly
-as this decision's "cost of trying-and-failing" line predicted.
+ships no loader. The fix is `cc-debian13`, NOT the `base-debian12` the
+fork-point originally named. Two corrections: (1) a dynamic C++ binary needs
+`libstdc++` + `libgcc_s`, which `base` (glibc only) lacks — `cc` is Google's
+designated base for dynamically-linked C/C++; (2) the cc variant must be new
+enough. `local_config_cc` is non-hermetic, so the binary inherits the CI
+runner's glibc (ubuntu-24.04-arm = glibc 2.39). glibc is forward-compatible
+only, so `cc-debian12` (glibc 2.36) still failed with `GLIBC_2.38 not found`;
+`cc-debian13` (glibc 2.41 / GCC 14) satisfies it. Rule: runtime base glibc >=
+build-runner glibc. (The non-hermetic cache makes this a property of the runner,
+not the source — a Bazel remote-cache hit reused the 24.04-built objects across
+runner versions, so "build on an older runner" did NOT work; moving the base
+forward did.) The gateway (Go, genuinely static) stays on `static-debian12`.
+Net cost of the lost bet: a couple of CI build cycles, as this decision's "cost
+of trying-and-failing" line predicted.
 
 Original reasoning (kept for the trail):
 - The base image was already pulled in `MODULE.bazel` for the gateway

packaging/engine/BUILD.bazel

@@ -40,11 +40,16 @@
 #
 #    The fix is `@distroless_cc`, NOT the `@distroless_base` this comment
 #    used to suggest: a dynamically-linked C++ binary needs libstdc++ +
-#    libgcc_s, which `base-debian12` (glibc only) lacks. cc-debian12 is
-#    Google's designated base for dynamically-linked C/C++ (loader + glibc
-#    + libstdc++ + libgcc_s). Pulled in MODULE.bazel. This BUILD's comment
-#    block is the ground-truth fork point — kept in sync with ADR-0025
-#    §"Slice 4 distroless variant decision".
+#    libgcc_s, which `base-*` (glibc only) lacks. cc-* is Google's designated
+#    base for dynamically-linked C/C++ (loader + glibc + libstdc++ + libgcc_s).
+#
+#    The cc variant must also be NEW ENOUGH: local_config_cc is non-hermetic,
+#    so the binary inherits the CI runner's glibc (ubuntu-24.04-arm = 2.39).
+#    glibc is forward-compatible only, so cc-debian12 (glibc 2.36) was too old
+#    (`GLIBC_2.38 not found`); the base is cc-debian13 (glibc 2.41). Keep the
+#    runtime base glibc >= the build-runner glibc. Pulled in MODULE.bazel.
+#    This BUILD's comment block is the ground-truth fork point — kept in sync
+#    with ADR-0025 §"Slice 4 distroless variant decision".
 #
 # 3. Models NOT shipped in image. Engine reads model path via
 #    `AEGIS_MODEL_PATH` env var (`engine_cpp/cmd/engine/main.cc:68-73`)