Skip to content

ADR013-secrets.md

Latest commit

 

History

History
149 lines (97 loc) · 42.1 KB

ADR013-secrets.md

File metadata and controls

149 lines (97 loc) · 42.1 KB
Raw

ADR: platform secrets — OpenBao for tenant credentials

Status: accepted; substrate deployed on the local mock cluster (w4/m5), wired into the product in w4/m6 (bex-api's env-vars API reads/writes tenant credentials through it — see Product usage), wired into prod in w1/m10 (init/unseal + Kubernetes auth run from deploy.yml, replicas: 3 overlay — see Prod deploy path), and extended in w11/m8 with an optional KV-v2 compare-and-set capability for one-variable mobile edits.

Context

docs/ADR012-auth.md covers identity (who is calling) and authorization (what they may touch). Neither answers a third question: once an agent deploys an app on a tenant's behalf (docs/ADR008-vision.md pillar 4, "deploy-from-chat"), where does that app's own credentials — a database password, a third-party API key — actually live? Today the only place is the App CR itself: plaintext in etcd, readable by anyone who can get the CR. That's fine for bex's own platform secrets (Kratos/Hydra secrets, the zot registry credential, etc. — those are w1/m7 t003's concern: platform secrets committed to git via SOPS/sealed-secrets, because they're static and bex-operated). It is not fine for tenant secrets: a growing, per-tenant, per-app set of credentials that must be versioned (rotate without downtime), revocable, and readable only by the scope that owns them — not by every other tenant's app, not by anyone who can kubectl get apps.

This ADR is the substrate for that: a versioned, policy-scoped secret store running as platform infrastructure, so w1/m2's tenants table and w4/m6's product wiring have somewhere to put credentials before either exists.

Decision

1. OpenBao, not HashiCorp Vault

Same governance test CNPG, Ory, and OpenFGA already passed (ADR009-postgresql-management.md §1): open-source, self-hostable, no SaaS dependency. HashiCorp re-licensed Vault under BSL 1.1 in 2023 — source-available, not OSI open source, and explicitly restricts competing-use. That disqualifies it for a platform whose whole premise is "Apache-2.0, run it yourself" (ADR008-vision.md). OpenBao is the Linux Foundation-governed MPL-2.0 fork: same HTTP API, same secrets/auth/policy model, drop-in for everything below.

2. Storage backend: integrated Raft, not Postgres or Consul

OpenBao (like Vault before it) supports several storage backends. Bex already operates Postgres well via CNPG — the instinct is to reuse it, as Kratos/Hydra/OpenFGA all do. Rejected anyway:

  • Postgres backend is community-supported, not the recommended path, and HashiCorp deprecated recommending external-DB backends in favor of integrated storage back in Vault 1.4 (2020) for good reason: OpenBao would need its own DSN Secret bootstrapped out-of-band exactly like Kratos/Hydra — except the thing being bootstrapped is itself the credential store, a chicken-and-egg dependency this design doesn't need to take on.
  • Consul is another moving part bex doesn't otherwise run, for the same benefit Raft gives for free.
  • Integrated Raft storage is self-contained (one PVC per pod, no external dependency), gives built-in snapshotting, and is the chart's own recommended default. Locally: replicas: 1 (single-node raft — same storage engine, no quorum). Production runs replicas: 3 for quorum and tolerates one member or node being unavailable; the one-member-at-a-time drain/unseal runbook is in ADR015-openbao-backup-restore.md.

3. Unseal strategy: Shamir secret sharing via .env, not auto-unseal

This custody shape — high-trust material out-of-band in .env/CI secrets, never in an in-cluster store — is reused for the backup-encryption private key in ADR050-encrypted-platform-backups.md §1.

OpenBao encrypts everything at rest behind a master key split into Shamir shares; a freshly started (or restarted) node is sealed until a threshold of those shares is presented. The alternative — cloud KMS auto-unseal (AWS/GCP) — is rejected for the same reason bex's infra/ targets Hetzner via Cluster API rather than a single hyperscaler: it would tie a self-hostable platform to a specific cloud's KMS.

Instead: scripts/bao-init.sh runs bao operator init once (5 shares / 3 threshold, the OpenBao default), capturing 3 unseal keys + the root token and writing them into .env (gitignored — same rule as bex.kubeconfig) — never printed to stdout or logs, mirroring auth-secrets.sh's never-echo convention. Every subsequent run detects "already initialized," reads the same three keys back out of .env, and unseals idempotently — including after a pod restart (raft persists the encrypted data to the PVC, but the decryption state is always sealed on process start; this is correct behavior, not a bug, and scripts/bao-verify.sh asserts it explicitly).

This is a real trust tradeoff: whoever holds .env (or its prod equivalent — GitHub Actions secrets via scripts/gh-secrets.sh) can unseal OpenBao and mint root-scoped tokens. Accepted — it's the same trust boundary the Ory secrets and bex.kubeconfig already carry.

4. KV v2 mount tenants/

One versioned key-value engine (kv-v2: soft-delete + rollback), mounted at tenants/. Tenant credentials live at tenants/<tenant-id>/... — a path convention, not a mount-per-tenant. A mount per tenant doesn't scale (mounts are a fairly heavyweight, admin-managed concept) and buys nothing here: the policy in §5 already scopes every non-admin caller to the tenants/* subtree as a whole, and per-tenant isolation within that subtree is w4/m6's job (the product layer knows which tenant a caller represents; OpenBao itself doesn't need to).

The product's base SecretKV seam keeps its original unversioned read/write/delete/list contract. A separate optional VersionedSecretKV capability returns one atomic data+version snapshot and supports a full-map KV-v2 write with options.cas=<observed-version>. This is opt-in per request: the mount does not enable cas_required, which would break legacy service env-var and secret-file writers that deliberately retain their existing semantics. CAS 0 is create-only; a stale version is a conflict, never an unconditional overwrite.

Environment groups keep only values in OpenBao (w8/m114). A group's env and files maps live under its own workspace's tenant prefix (secrets.WithTenant): env-groups/<id>/env and env-groups/<id>/files, plus, while a content save is in flight, that save's recovery maps under env-groups/<id>/op/<opId>/{old,new}-{env,files}. Nothing else about a group is in OpenBao. Its id, workspace, name, Environment, service links, timestamps, the key and file names its committed content holds, and its save coordination are one row of the control-plane table env_groups:

  • Names are unique per workspace by the (tenant_id, name) constraint, so concurrent creates and renames have one winner (ENV_GROUP_NAME_EXISTS) without claim records or reservations.
  • Metadata writes compare-and-set on meta_version.
  • Content effects and recovery hold one per-group Postgres advisory session lock across maps and projection Secrets. Statements commit individually so a crashed process leaves its recovery evidence. A contending live instance fails with a revision conflict; lease expiry alone cannot overtake a live writer. Group queries share the leased connection, including bounded detached compensation.
  • A content save claims content_version busy and records its operation (kind, phase, lease) before preparing recovery maps. It records each intended phase before writing the corresponding maps and projection Secrets, then releases the row idle at the next content_generation, which is what revision tokens (egr1_<generation>) encode. An interrupted save is restored or finished by another instance after its lease, from the recovery maps; one that cannot be restored is left repair_required with its operation. An idle row retains its operation until recovery-map cleanup succeeds, without advancing the generation again. A database-session failure cannot revoke an external write already in flight; failed database operations stop further work and preserve recovery evidence.
  • Deleting a group deletes its row, so every later write of that id matches nothing; no tombstone is kept.
  • List, get and Environment membership read rows only — no OpenBao call. Reveals and saves read the maps.

Environment groups therefore need the control-plane store; with it unwired they answer SECRETS_UNAVAILABLE, as with OpenBao unset.

The w8/m114 cutover moved environment-group metadata out of OpenBao into Postgres and removed the legacy layouts, migration command and temporary serving gate.

5. Kubernetes auth method — bex-api authenticates as itself

No shared token. OpenBao's chart enables server.authDelegator by default, granting OpenBao's own ServiceAccount the system:auth-delegator ClusterRole — the permission its Kubernetes auth method needs to call the apiserver's TokenReview API and validate a caller's projected ServiceAccount token. scripts/bao-k8s-auth.sh (idempotent, same shape as authz-model.sh) then:

  • enables the kubernetes auth method,
  • writes a policy (tenants-rw) granting create/read/update/delete/list on tenants/* and nothing else — no sys/*, no other mount, so a compromised bex-api pod can't read seal status, re-key, or touch any other tenant of the store (there is only one, but the point generalizes),
  • binds a role bex-api to that policy, scoped to ServiceAccount bex-api in namespace bex-system (the same ServiceAccount lego/operator/config/api/rbac.yaml already defines).

This is the same shape as Hydra client_credentials or the OpenFGA preshared key: a machine identity, minimal scope, no human password anywhere in the loop.

6. GitOps shape, namespace secrets

deploy/gitops/base/openbao.yaml is a multi-source Argo Application (pinned chart 0.28.4, app v2.5.5) exactly like kratos.yaml/openfga.yaml: the Helm repo provides the chart, this repo's ref: values source provides base/values/openbao.values.yaml. Namespace secrets, not auth — this is tenant-credential infrastructure, a different trust boundary than platform identity/authz, and keeping it a separate namespace keeps RBAC/NetworkPolicy scoping simple later. sync-wave: "1": unlike Kratos/Hydra/OpenFGA, OpenBao has no CNPG Cluster dependency (§2), so it doesn't need to wait for the CNPG operator wave — it's an independent base component alongside Traefik/cert-manager/metrics-server. Cluster-internal only, no ingress: bex-api reaches it via in-cluster Service DNS (openbao.secrets.svc:8200) only, the same shape as Hydra's/OpenFGA's admin APIs.

Verification

scripts/bao-verify.sh (exit 0 = pass) proves, against the current kubeconfig cluster:

  1. Init/unseal is idempotent — running bao-init.sh twice in a row changes nothing the second time.
  2. KV read/write — a value written under tenants/verify/... reads back with the expected data.
  3. Scoped SA login — the bex-api ServiceAccount logs in via the Kubernetes auth method and can read/write under tenants/*, but a request for sys/mounts (or any path outside tenants/*) with that same token is denied (403). (sys/seal-status itself is deliberately unauthenticated in OpenBao — not a useful negative test.)
  4. Restart durability — after deleting the OpenBao pod (the chart's StatefulSet uses updateStrategyType: OnDelete, so a plain kubectl rollout restart doesn't actually recreate it — the pod has to be deleted directly), the pod comes back sealed (expected — no auto-unseal), the KV value from step 2 is unreadable until unsealed, and re-running bao-init.sh's unseal step (same three keys from .env) restores it exactly. This proves both that state survives the pod (raft on the PVC) and that the unseal keys are load-bearing, not decorative.

Local bring-up order (what Argo's waves do in prod): mock cluster → local-path provisioner → secrets namespace labeled pod-security.kubernetes.io/enforce=privileged (local-CAPD quirk below) → OpenBao chart (base values + local overlay) → scripts/bao-init.sh → scripts/bao-k8s-auth.sh → scripts/bao-verify.sh.

Local-CAPD quirks (mock cluster only, don't apply to prod):

  • The secrets namespace needs the same pod-security.kubernetes.io/enforce=privileged label the auth namespace already carries — local-path's hostPath helper pods run in the PVC's namespace and are blocked by the cluster's baseline PSA enforcement otherwise (same reasoning as ADR012-auth.md's local-CAPD quirks section).
  • Unlike Kratos/Hydra/OpenFGA, OpenBao's pod does need pinning to the control-plane node (overlays/local/values/openbao.values.yaml's nodeSelector/tolerations), and this one is easy to misdiagnose: a worker-scheduled pod comes up Running, its port stays in LISTEN, and it even accepts TCP connections — but every HTTP request just hangs forever (confirmed via kubectl exec to 127.0.0.1:8200 from inside the pod itself, which hung identically to a port-forward from outside). The cause is service_registration "kubernetes" {}, which has OpenBao call the apiserver directly from inside its own request-handling path; on a worker node under this cluster's OrbStack/Calico networking that call never returns (the same reachability gap ADR012-auth.md's local-CAPD quirks section pins coredns/local-path/CNPG for). Moving the pod to the control-plane node fixed it immediately, confirming the cause.

Product usage (w4/m6): the env-vars API

The consumer this substrate exists for: bex-api's Render-compatible env-vars API (ADR006-bex-api.md). A tenant (an API key, or a human session) sets a service's environment variables; the values live in OpenBao under that service's path; the operator materializes them into the running app.

flowchart LR
  caller["API key / session"] -->|"legacy writes or PATCH /environment"| api["bex-api Core"]
  api -->|"ordinary KV v2 write or optional revision CAS<br/>(source of truth)"| bao["OpenBao<br/>tenants/&lt;tenant&gt;/services/&lt;svc&gt;/env"]
  api -->|"project"| sec["k8s Secret &lt;svc&gt;-env"]
  api -->|"active spec ref or pending metadata + optional one roll"| cr["App CR"]
  cr --> op["operator"]
  op -->|"envFrom + rolling restart"| pod["app pods (new values)"]
Loading

  • Path layout. One KV v2 mount tenants/ (§4); a service's env map lives at tenants/<tenant>/services/<service>/env. Until w1/m2 grows real tenants, <tenant> is the single default (the same convention m4's authorization uses for the default workspace).

  • Authentication. bex-api logs in with the Kubernetes auth method as its own ServiceAccount (§5, role bex-api, policy tenants-rw), caches the returned client token until just before its lease expires, and re-authenticates on demand (including when a still-cached token is rejected). Off-cluster runs (local dev, scripts/secrets-verify.sh) point BEX_OPENBAO_JWT_PATH at a token minted with kubectl create token bex-api.

  • Materialization + the etcd trade-off. On every write, bex-api stores the map in OpenBao (the source of truth), then projects it into a per-app Kubernetes Secret <service>-env (owned by the App, so it is garbage-collected with it); the operator wires an active reference into envFrom. Legacy item/replace writes set the spec reference and roll immediately through spec.restartedAt. The coherent environment batch (PATCH /v1/services/{id}/environment, w5/m44) projects env and file Secrets together. save_only leaves an existing spec reference untouched; for the first service-local Secret it records app.bex.co/pending-env-secret / app.bex.co/pending-files-secret metadata instead. The operator's generation-only update predicate ignores that metadata write, then consumes the pending name during the next deliberate deploy/restart reconcile (including native-build runtime env). deploy activates every pending reference and bumps restartedAt once. Omitted masked entries remain untouched, and a late projection/App-patch failure compensates source and derived state. Accepted trade-off: a projection copy of the values does live in etcd, in that Secret — OpenBao buys durability, versioning, audit and policy-scoping, not etcd-avoidance. The future alternative that removes the etcd copy is sidecar/agent injection (an OpenBao Agent rendering the secret into the pod at runtime), recorded here as follow-up work, out of scope for m6.

  • deploy reruns the serving build (w4/m180). Render separates save only, deploy and save-rebuild-deploy; bex's rebuild is the batch's rebuild mode: a save_only save followed by deploys.Service.Rebuild, recorded as config_change, the same cause an environment group's rebuild uses (w4/347). The batch's deploy mode must therefore not build. Since spec.restartedAt is part of a source-built artifact's identity, the bump alone dispatched a full BuildKit build (live 2026-10-07: two ~32 s go builds for env-only saves). deploy therefore also writes an image-only spec.releaseConfig bound to the release the patch opens (metadata.generation + 1, sourceGeneration: 0) that selects the live deploy's resolved image. The operator reuses that image while snapshotting the newly saved values, and the deploy row records that image and its commit, so history shows no build. The selection is re-read inside the locked patch. A source-built service with no live build is refused with 409 ENVIRONMENT_DEPLOY_NEEDS_LIVE_RELEASE before anything is written; save only and rebuild still work. Prebuilt-image services already run spec.image, and a static site's environment exists only at build time, so its deploy remains a rebuild. CR-only mode (no deploy store) keeps the previous behavior.

  • Revision-aware one-key update (w11/m8). The sensitive GraphQL service.envVarKeys list remains values-free and now attaches the same opaque whole-environment revision to every key; the explicit service.envVar(key:) reveal returns the revision of the coherent map it read. Sensitive REST/MCP list/get responses expose the same additive revision, and their coherent environment patch accepts the same optional expectedEnvRevision, so adapter semantics and coded errors remain aligned. The revision is a canonical evr1_… encoding of OpenBao metadata, not a credential or authorization bypass, and clients must treat it as an opaque echo-only token. Revision mode updates exactly one existing ordinary key/value assignment: it cannot create a missing key and accepts no secret files, rename, delete, or generated value. The server reads one versioned snapshot and submits the completed map with KV-v2 CAS. A missing key returns ENVIRONMENT_VARIABLE_NOT_FOUND, an invalid key returns ENVIRONMENT_VARIABLE_INVALID, and a stale observation returns ENVIRONMENT_REVISION_CONFLICT, all before any source or Kubernetes projection write. Even a same-value CAS advances the revision so two submissions from one observation resolve to one success and one conflict; because effective data did not change, that case does not roll pods.

  • CAS compensation. A revision-aware projection annotates <service>-env with the opaque source revision that owns it. If the source CAS succeeds but projection or App patching fails, compensation restores the prior OpenBao map with a second CAS against the failed attempt's version, then conditionally restores or deletes the derived Secret only while that failed revision still owns its exact Kubernetes UID/resourceVersion. A fully restored attempt returns ENVIRONMENT_UPDATE_RESTORED; a newer source or projection owner returns ENVIRONMENT_REVISION_CONFLICT without being overwritten; an incomplete rollback returns ENVIRONMENT_RESTORATION_FAILED. Those stable outcomes remain visible through REST code, GraphQL extensions.code, and MCP error text. Revisions, keys, values, OpenBao URLs/paths, response bodies, Kubernetes object names, and CAS versions never enter CAS-conflict or version-store error text. The revision capability is additive; a non-versioned SecretKV continues serving legacy behavior, while a revision-bearing mutation fails closed when that optional capability is absent.

  • Item and batch compensation (w5/m119). Every other env and secret-file write, the single-key verbs and the sparse batch patch, takes one compensation path. Each map goes back to what the write replaced only through a CAS against the version that write committed, so a write another caller committed since is never erased. A superseded map and its projection are left to the newer writer, and the call answers ENVIRONMENT_RESTORATION_FAILED, since its change may survive in that write. That includes the batch patch whose secret-file write fails after its env write was superseded. A restored map's Secret is put back with it. For an empty map, the Secret is kept or created empty rather than deleted (w5/106, below). A projection Kubernetes refuses as invalid answers 400 ENVIRONMENT_VARIABLE_INVALID or SECRET_FILE_INVALID, naming the key once and the rule, with the same params as the name's own refusal. A refusal of anything other than the service's own env or files Secret, such as its App, is not relabeled as a key refusal. A restored map's Secret goes back at the revision the restore committed, so it cannot overwrite a newer writer's projection (w5/m127, below).

  • Revision-owned projections (w5/m127). Every <service>-env and <service>-files Secret records the store revision it holds, in app.bex.co/env-source-revision or app.bex.co/files-source-revision. Every writer projects through one primitive that never moves a Secret back to an earlier revision: the batch patch, the single-key verbs, seeds, compensation and deletes. One exception is a Secret another App controls, a deleted namesake's that garbage collection has not reached: its revision belongs to a purged store, so a recreated service's first write replaces it and takes it over, once a re-read confirms the writer's App is still the one at that name (w5/118). The same re-read gates any Secret the writer's App does not control yet, including an ownerless one a create prepared: a write in flight for a deleted App refuses with the service-changed conflict rather than take over its namesake's prepared Secret (w5/147). Create-time preparation, which runs before the App exists, creates its Secret stamped with the revision it wrote. If a deleted namesake's Secret still holds that name, preparation takes it over and rewrites it ownerless with its own map and revision. A live App's Secret, another create's unadopted one, or a Secret being deleted refuses the create with an uncoded 409 conflict (w5/128). A failed preparation removes the leg it already wrote. An abort removes only the legs preparation wrote, so a leg it never wrote, whose Secret may be a namesake's, is left alone. Both run detached from the request, so a client gone mid-create cannot strand them. A create also claims, and at once releases, each leg it does not seed whose Secret exists, a create that seeds neither included, again detached from the request. A crashed create's abandoned leg, or a deleted namesake's, therefore goes with its store map instead of passing its keys to the new service. A release that cannot remove the map keeps the Secret holding the name until a later create takes the abandoned claim over. A leg anybody still holds refuses the create, as a seeded one does. A service created beside such a leftover before this treats the abandoned preparation as its predecessor and replaces it on its first write, while a failed write still restores it as the service's own (w5/148). A projection a later write already landed is left to that write. An update carries the read object's resourceVersion, so a concurrent change is judged again rather than overwritten. The same revision with other data is a conflict, since one revision holds one map. A delete projects before it commits, so a failed delete leaves its key revoked and the store holding it as the retry anchor. Its projection carries the revision its commit will create, marked app.bex.co/source-revision-provisional. A write that commits that revision replaces it, and the delete's commit then conflicts and re-reads. A delete that finds that revision, or a later one, already projected re-reads before it rolls the workload, since its commit could only conflict. When two deletes read the same revision, the Secret keeps only the keys both keep, so neither re-mounts the key the other revoked. A store without versions keeps unconditional writes. Emptying a map never deletes its Secret (w5/106). Deleting a service's last secret file, or restoring an empty map, leaves the Secret empty and stamped with that revision. A projection that arrives late loses to it instead of recreating the Secret with what the store no longer holds. A write that empties the files map drops its mount, and any mount a save-only write staged, so the next release mounts no empty /etc/secrets. A save-only write that empties it leaves an existing reference for the next deploy, as every save-only write does. A projection a later write superseded mounts the files Secret only if that write left files in it. The App's reference is ordered too (w5/m131). spec.filesFromSecrets is one list a merge patch replaces whole, so every secrets write and every env-group link or unlink patches the App under an optimistic lock. A write that loses re-reads the App and decides again: its files projection reports what the Secret holds now, so the reference ends where the latest files write leaves it. A re-read that finds another App at the name, the service deleted and recreated meanwhile, refuses with the service-changed conflict rather than write to the namesake. That write's compensation projects nothing. It takes back only a change its store write merged into the namesake's map, while that write is still the latest (w5/157). A revision-checked save takes the same course and answers ENVIRONMENT_REVISION_CONFLICT. A deleted service's purge, and a failed preparation's release, retire its store paths rather than delete them: an empty map is committed as a new version and every older version is destroyed, so versions keep counting. A write that read the dead service's map therefore never compare-and-sets over its namesake's, and no projection or rollback mistakes one service's revision for the other's. A workspace's purge deletes its paths outright, the retired ones included (w5/156). A batch projects its env map once, before the patch, since a retried CAS projection would refuse a save the store already holds. A write that keeps losing answers 409 CONFLICT.

  • Environment Group coherent patches (w2/m73). A group owns two value maps (env and files) but one opaque egr1_… revision/claim record. The sparse patch resolves literal-or-generated values in Core, preserves omitted masked entries, and CAS-claims one generation before updating both maps and both projection Secrets. Projection failure compensates the maps and derived objects only while the failed generation still owns the transaction; a newer writer is never overwritten, and an incomplete repair leaves a named fail-closed state. save_only changes source/projection state without a rollout; deploy or rebuild performs at most one selected action per linked service after persistence, and a failed action can be retried with an empty patch at the committed revision. Clone holds a source revision claim while it copies contents server-side, preventing an inconsistent two-map snapshot and keeping plaintext out of the dashboard.

  • Precedence and redeploy (w2/m94). Three rules, all mechanism rather than policy — each is an ordering the operator emits and Kubernetes resolves:

    1. A service's own value beats every linked group's, for env vars and secret files. envFromSources lists each linked group's <evg-id>-env Secret in spec.envFromSecrets order and the service's own <name>-env Secret last, and Kubernetes lets the last envFrom source win a key collision. secretFileMounts does the same for files: the service's own <name>-files Secret is projected last into the single /etc/secrets volume regardless of its position in spec.filesFromSecrets, because the kubelet resolves a duplicate path to the last projected source. The positional move matters — that list records the order refs were added, so a service created with files had its own Secret first and silently lost its file to a group on link (w1/095).
    2. Between groups, the last linked wins. Group Secrets keep spec.envFromSecrets order, so the most recently linked group is last and therefore wins. Render documents that its own group-vs-group precedence is not guaranteed and currently resolves to the most recently created group; bex keeps last-linked deliberately, because changing the rule would silently flip which value already-running services serve. AppView.linkedEnvGroupIds exposes this order on REST, GraphQL and MCP so the Environment page can order the panel and mark the shadowed key or file rather than implying a winner it never computed (w1/091).
    3. A group write deploys only linked services whose Auto-Deploy is on. Render: "If you make changes to an environment group (including deleting it), Render kicks off a new deploy for every linked service that has autodeploys enabled." A gated service's Secret is current — the group Secret is shared — but its spec.restartedAt is left alone, so the pod keeps serving its release and picks the values up on its owner's next deploy. Link, unlink and group delete are staged the same way (w4/m239): the group's Secret refs in spec.envFromSecrets/filesFromSecrets are release identity, so writing them made the operator ship a release no deploy row recorded and stranded the owner's next deploy. A gated link or unlink is instead recorded in the app.bex.co/pending-env-group-links annotation (metadata, outside the identity). rollout.EffectiveGroupSecrets is what readers (mountsGroup, linkedEnvGroupIds, Blueprint export) see, and rollout.FoldStagedGroupLinks moves it into the spec in the one write that mints the owner's next release — a manual/API/hook deploy, restart, rebuild, or any Settings write that already rolls a release. A rollback does not apply staged links. A deleted group is safe to leave mounted until then: the operator marks group Secret sources optional and each release serves its own immutable snapshot. Only the Secret values of the running release, never its mounts, change under a gated service; the write result names it under pendingServiceIds, which is a deliberate skip and not a retryable failure. The gate applies only to repo-backed services: bex defaults spec.autoDeploy to false for an image-backed service because there is no branch to watch, and reading that default as an opt-out would strand every image-backed service on stale values. On the explicit saveMode surfaces (REST /contents, GraphQL, MCP, the dashboard editor), saveMode: "deploy" follows the same gate and reports pendingServiceIds on all three APIs; only saveMode: "rebuild" is the caller's deliberate, ungated rollout. The dashboard reports a deploy as started only for affected services that are neither pending nor failed (w4/224).

  • Native build-time environment (w4/m93). A native build's RUN receives the same effective Secret-backed environment the running pod does: the operator merges the App's linked group Secrets (in spec.envFromSecrets order, each optional like the runtime envFrom projection) with the service's own Secret (last, so its keys win) into one App-owned <name>-native-env Secret beside the build Job, transported through the existing single BuildKit secret mount with the byte-exact codec from w4/m92. Literal spec.env entries still override every Secret source, matching Kubernetes env precedence. A briefly-absent group contributes nothing; a missing own Secret, read failure, or protected source fails the build rather than building with a partial environment. Linking or unlinking a group changes the native artifact fingerprint, so a direct spec edit rebuilds just like an API link's restartedAt bump.

  • Native build-time secret files (w4/m163). A native build command (web, private, worker, cron and static sites alike) reads each effective secret file at /etc/secrets/<name>, the same path the runtime volume uses. The operator merges the exact runtime source list (secretFileSources: linked group file Secrets in order, the service's own <name>-files last whatever its spec position, then a save-only pending files Secret) into one App-owned bld-<name>-native-files Secret beside the build Job; every source is optional like the runtime projection, while a read failure or protected source fails the build. The preparer copies those bytes into its private build volume, and each file reaches the RUN through its own transient required=true BuildKit secret mount: never COPY, ARG, ENV, the build context, the image config or a layer (verified against the pinned BuildKit v0.30.0 image). Only file names appear in the generated Dockerfile and build log. Because BuildKit's cache key ignores secret contents, file bytes join the opaque native revision; a changed, emptied, added or removed file re-runs the build command at an unchanged commit, which the pinned-image integration test demonstrates by showing the stale reuse a fixed revision would cause. spec.filesFromSecrets joins the native artifact fingerprint only; Dockerfile, buildpack, prebuilt and no-build static paths are unchanged and still receive no build-time files. The projection is deleted when the last file goes away and is reclaimed with the App.

  • Authorization + leak discipline. Reading values requires the sensitive-read scope (can_view_sensitive), writing the manage scope (can_create) — enforced by the same OpenFGA Checker as every other verb (a tuple-less key gets 403). Values never appear in logs, error messages, or responses beyond the authenticated GET/PUT bodies: a rejected write names the offending key at most, never its value, and the OpenBao client's errors carry only method + path.

  • Two writers, one read (w4/m120). The store is not the only source of a service's environment. A blueprint's plain envVars: literals deliberately stay manifest-owned on spec.Env — the manifest is their writer, and a sync must be able to converge them — while an interactively-created service's literals are moved into the store at create time (w6/m45). The operator feeds spec.Env straight to the container, where Kubernetes env beats the envFrom projection, so a manifest literal is what the process actually has.

    Reads had never been told. Every env read went through the store only, so a blueprint service answered GET env-vars with [] and GET env-vars/MESSAGE with 404 while its process ran the manifest's value (observed live 2026-09-21). Worse, SetEnvVar had no spec.Env check, so a same-key write succeeded and did nothing observable: it landed in the store, the store was projected into <name>-env, and the container kept reading the manifest's env entry — the API said 200, the Environment tab showed the new value, and the process never changed. That is the exact hazard takeCreateEnvLiterals already documents for the interactive path; the blueprint path had no equivalent protection.

    So the reads are now the union — store ∪ manifest literals, each manifest entry flagged managedBy: "blueprint" on REST, GraphQL and MCP — and the four write verbs (SetEnvVar, SetEnvVars, DeleteEnvVar, PatchEnvironment) refuse a manifest-owned key with the conflict code ENV_VAR_MANIFEST_MANAGED, naming the remedy: edit the manifest and sync, or declare the variable sync: false so the store owns it instead. ValueFrom entries are skipped — they are Secret-key references the operator resolves, not values with a view to live in. A bulk write is all-or-nothing, and a rename is refused on its source key too, since a rename deletes it. Two deliberate consequences: the manifest wins a collision on read (because the runtime does, so reporting the store's value would describe a process that does not exist), and a read-modify-write of the whole set through PUT .../env-vars now fails on a blueprint service rather than shadowing — consistently refused rather than sometimes-accepted-when-the-value-happens-to-match. A service with no manifest has an empty spec.Env, so the merge is a no-op and the wire shape is unchanged.

  • Fail-closed. With BEX_OPENBAO_URL unset bex-api has no store and the env-vars verbs 503 (the rest of the API is byte-for-byte unchanged); a sealed or unreachable OpenBao 503s the credential-touching verb, mirroring the Hydra fail-closed precedent in ADR012-auth.md.

Verified end-to-end on the mock cluster by scripts/secrets-verify.sh: mint a key → PUT env-vars → the value is present in OpenBao (scoped read) → the <svc>-env Secret is materialized → the app's pods roll → the running app serves the new value; a tuple-less key gets 403; with BEX_OPENBAO_URL unset the endpoints 503.

Prod deploy path

Wired end-to-end (w1/m10). The app side is live — the prod bex-api Deployment sets BEX_OPENBAO_URL=http://openbao.secrets.svc:8200 (lego/operator/config/api/deployment.yaml) — and the CI/unseal side runs on every deploy:

  1. One-time (operator, by hand, against the prod kubeconfig): BAO_ALLOW_INIT=1 scripts/bao-init.sh initializes OpenBao (5 Shamir shares / 3 threshold), writing the unseal keys + root token into the operator's local .env. The BAO_ALLOW_INIT=1 opt-in is mandatory for a first init and is set only here — deploy.yml never sets it, so CI can never mint-and-discard the keys against an ephemeral runner. Do not cp .env.example .env around this step: bao-init.sh writes the keys straight into .env, and a cp would overwrite the only copy. Prod runs server.ha.replicas: 3, so bao-init.sh initializes the ordinal-first pod (the raft leader) and then unseals every pod directly — Shamir seal state is per-node, and the openbao Service round-robins across sealed+unsealed members, so a Service-targeted unseal is unreliable in HA. (Single-node and the BAO_ADDR off-cluster paths operate on the one reachable endpoint and are unaffected.)
  2. scripts/gh-secrets.sh pushes BAO_UNSEAL_KEY_1/BAO_UNSEAL_KEY_2/BAO_UNSEAL_KEY_3/BAO_ROOT_TOKEN (now present in .env from step 1) into this repo's GitHub Actions secrets — they're in the script's key list alongside the KRATOS_*/HYDRA_*/OPENFGA_* keys.
  3. .github/workflows/deploy.yml waits for the OpenBao StatefulSet's pods to reach Running, then runs bao-init.sh (idempotent — detects "already initialized," just unseals each pod; refuses to init without BAO_ALLOW_INIT) and bao-k8s-auth.sh after the rollout on every deploy, the same shape as the auth-secrets.sh/authz-model.sh steps.
  4. Production sizing: overlays/prod/values/openbao.values.yaml sets server.ha.replicas: 3, server.podManagementPolicy: Parallel (so the sealed pods don't block each other's creation), and a raft config with retry_join for the three pods (so followers auto-join openbao-0's cluster over the headless openbao-internal Service); it carries no storageClass override (falls back to the cluster's default, e.g. hcloud-volumes). The layer is applied onto the openbao Application by the prod overlay's kustomization patch — mirroring how auth-dbs's local patch shrinks storage only for CAPD. The openbao Application itself is deployed in every environment via deploy/gitops/base/kustomization.yaml.

Live acceptance — PUT /v1/services/{id}/env-vars against prod returns 200 (not 503), the value lands in OpenBao under tenants/default/services/<svc>/env and survives a bex-api pod restart — is the operator's first-run verification; thereafter every deploy.yml run re-unseals idempotently (no re-init).

Alternatives considered

  • HashiCorp Vault — same product, but BSL-licensed (§1). Rejected.
  • Encrypted columns in the control-plane Postgres (pgcrypto) — reinvents dynamic secrets, leasing, versioning, and audit logging that the Vault family already solved, and couples tenant secret material to the same database backing App CR metadata (one compromise or bad migration touches both). Rejected.
  • Cloud secret managers (AWS Secrets Manager, GCP Secret Manager) — defeats self-hostability and ties bex to one cloud, even though infra/ already targets Hetzner via Cluster API. Rejected.
  • A mount per tenant — see §4. Rejected as unnecessary operational overhead.

Consequences

  • Once w4/m6 wires product usage, bex-api becomes hard-dependent on OpenBao for any credential-touching verb — an outage or sealed state should 503 that verb, mirroring the Hydra fail-closed precedent in ADR012-auth.md.
  • Single-node raft locally, replicas: 1 — no quorum. Automated snapshot backup is now shipped (w1/m7 t006): a nightly Raft snapshot → object storage, mirroring etcd-backup — see ADR015-openbao-backup-restore.md. HA (replicas: 3 for quorum) is still the remaining half; until then, losing the single node means restoring from the latest snapshot (same recovery posture kratos-db/hydra-db carry today).
  • Root token + unseal keys are a manual, high-trust bootstrap step; rotating the root token or re-keying the Shamir shares follows ADR037-openbao-rekey-runbook.md. (w7/m37)
  • CI wiring (init/unseal in .github/workflows/deploy.yml, unseal keys in GitHub secrets via scripts/gh-secrets.sh) is in place: every deploy waits for the OpenBao rollout, then re-unseals each pod idempotently and reapplies the Kubernetes auth binding. The one-time first init against prod is a manual operator step — see Prod deploy path.