Follow-up to #341/#342. #342 stopped the silent cross-pod skew and was worth doing, but it bought coherence with manual toil — dev had already been hand-pinned twice in the 18 days before it merged, which is the same decay that produced #341. This issue is about the layer underneath: dev keeps breaking because it has been asked to be two different environments, and because the image it runs is not the image prod ships.
Three separate defects, each independently verifiable.
1. Dev has two contradictory mandates, both written in AGENTS.md
AGENTS.md:10 — "The dev server is for development: deploy your branch there and test against it."
AGENTS.md:81 — ":main — moving; rebuilt on every push to main and by the weekly cron. dev tracks this."
(also :19 and :264 — "dev tracks :main")
These cannot both hold. If dev auto-tracks main, the next merge — or the Monday 06:00 UTC cron — stomps whatever branch someone was testing. If dev holds a branch, it isn't "ahead of prod," it's sideways of it, and it isn't a canary for anything.
Every deploy strategy tried so far (:main+Always, then digest pinning) has been an attempt to satisfy both mandates at once. That is why none of them held.
2. The documented branch workflow is impossible — CI builds no branch images
# .github/workflows/docker.yml
on:
push:
branches: [main]
tags: ['v*']
schedule: ...
workflow_dispatch:There is no pull_request trigger and no non-main branch trigger, so no image for your branch ever exists. "Deploy your branch there and test against it" cannot be done as written; the only route onto dev is to merge to main first, which defeats the point of testing before merge. Hand-pinning dev has been the workaround, and hand-pinning is what decays.
Latent hazard in the same file (checked — has not happened in the last 40 runs, so this is prevention, not diagnosis): a workflow_dispatch run from a feature branch falls into the *) case, which tags only $repo:main. Dispatching a build from a branch would therefore publish that branch's code to the :main tag.
3. The artifact validated on dev is not the artifact promoted to prod
This is the one that matters most, and it is directly demonstrable. Commit 19b3c70 exists as two different images:
:main build of 19b3c70 -> sha256:f9be5dae… ← what dev ran
:19b3c709… / :v0.8.13 -> sha256:1a3409d5… ← what prod ships
Reproduce:
repo=boettiger-lab/mcp-data-server
tok=$(curl -s "https://ghcr.io/token?scope=repository:$repo:pull" \
| python3 -c "import sys,json;print(json.load(sys.stdin)['token'])")
for t in main 19b3c709f45394aaaddd64455af40fb65dd8aec1 v0.8.13; do
printf '%-45s ' "$t"
curl -sI -H "Authorization: Bearer $tok" \
-H "Accept: application/vnd.oci.image.index.v1+json" \
-H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
"https://ghcr.io/v2/$repo/manifests/$t" | grep -i '^docker-content-digest:'
doneTwo consequences, both bad:
:<git-sha> is not immutable, despite AGENTS.md:82 calling it "immutable; one per commit." On a tag push, docker.yml tags $repo:${{ github.sha }} — and for a release tag, github.sha is the commit the tag points at. So tagging v0.8.13 on already-built 19b3c70 silently overwrote :19b3c709… with a freshly built image. Same source, different bytes, because the base image and unpinned deps moved in between.- Cron builds are unreferenceable. The
*) case tags only :main, never a :sha. A Monday rebuild produces an image with no immutable name at all — pinnable only by digest, which is part of why digest pinning became the only workable option.
A canary that runs a different binary than production is not a canary. Everything we validate on dev — guidance changes, the headless matrix, #245's pre-merge gate ambitions — is validated against an artifact prod will never run.
The fix: build once, promote the digest
Three rules, which is where mature pipelines land:
- Build once. One image per commit; never rebuilt per environment.
- Deployments reference an immutable digest, never a moving tag.
- The
image: line is written by automation, not by a human.
Environments then differ only in which digest they point at, and promotion to prod is "point prod at the digest dev already validated" — the identical bytes. This also dissolves the #341 trade-off: dev auto-tracks main and every pod converges, because the pod template changes atomically in one rollout.
The missing piece is small. CI already builds per-commit images; there is no deploy step and no cluster credential in any workflow — every deploy today is a manual kubectl. So:
- Preferred — CI-driven deploy. After the
main build, a job sets the freshly-built digest on dev-duckdb-mcp. Needs one NRP service-account token (patch on that one Deployment) as a repo secret. Dev then tracks main automatically and convergently, with no hand-edited digest.
- Rejected for now — GitOps (Argo/Flux). The more standard answer at scale, but it needs a controller on a shared cluster we don't administer. Revisit only if NRP already runs one.
Release promotion becomes: read the digest dev has been running, pin that in k8s/deployment.yaml. No rebuild on the release tag — or, if a :vX.Y.Z tag is still wanted for humans, retag the existing digest rather than rebuilding it.
And stop asking dev to host branches
Pick one and write it down, so the mandate stops being self-contradictory:
- Preview environments — add a
pull_request trigger building :pr-<n>, deployed to an ephemeral dev-pr-<n>; or
- One long-lived
sandbox-duckdb-mcp that is explicitly hand-pinned and expected to hold arbitrary builds.
Either way dev becomes off-limits to hand-repointing, which is the behaviour that has repeatedly broken it.
The invariant an agent can actually follow
No Deployment references a mutable tag. dev's image: line is written by CI only. To test a branch, use sandbox/preview — never repoint dev.
Mechanically checkable, so it doesn't rely on anyone remembering: fail CI if any image: in k8s/*.yaml lacks @sha256:.
Done when
Related: #341 (symptom), #342 (interim fix), #245 (pre-merge guidance gate — depends on dev being a trustworthy canary).
Follow-up to #341/#342. #342 stopped the silent cross-pod skew and was worth doing, but it bought coherence with manual toil — dev had already been hand-pinned twice in the 18 days before it merged, which is the same decay that produced #341. This issue is about the layer underneath: dev keeps breaking because it has been asked to be two different environments, and because the image it runs is not the image prod ships.
Three separate defects, each independently verifiable.
1. Dev has two contradictory mandates, both written in AGENTS.md
These cannot both hold. If dev auto-tracks
main, the next merge — or the Monday 06:00 UTC cron — stomps whatever branch someone was testing. If dev holds a branch, it isn't "ahead of prod," it's sideways of it, and it isn't a canary for anything.Every deploy strategy tried so far (
:main+Always, then digest pinning) has been an attempt to satisfy both mandates at once. That is why none of them held.2. The documented branch workflow is impossible — CI builds no branch images
There is no
pull_requesttrigger and no non-mainbranch trigger, so no image for your branch ever exists. "Deploy your branch there and test against it" cannot be done as written; the only route onto dev is to merge tomainfirst, which defeats the point of testing before merge. Hand-pinning dev has been the workaround, and hand-pinning is what decays.Latent hazard in the same file (checked — has not happened in the last 40 runs, so this is prevention, not diagnosis): a
workflow_dispatchrun from a feature branch falls into the*)case, which tags only$repo:main. Dispatching a build from a branch would therefore publish that branch's code to the:maintag.3. The artifact validated on dev is not the artifact promoted to prod
This is the one that matters most, and it is directly demonstrable. Commit
19b3c70exists as two different images:Reproduce:
Two consequences, both bad:
:<git-sha>is not immutable, despite AGENTS.md:82 calling it "immutable; one per commit." On a tag push,docker.ymltags$repo:${{ github.sha }}— and for a release tag,github.shais the commit the tag points at. So taggingv0.8.13on already-built19b3c70silently overwrote:19b3c709…with a freshly built image. Same source, different bytes, because the base image and unpinned deps moved in between.*)case tags only:main, never a:sha. A Monday rebuild produces an image with no immutable name at all — pinnable only by digest, which is part of why digest pinning became the only workable option.A canary that runs a different binary than production is not a canary. Everything we validate on dev — guidance changes, the headless matrix, #245's pre-merge gate ambitions — is validated against an artifact prod will never run.
The fix: build once, promote the digest
Three rules, which is where mature pipelines land:
image:line is written by automation, not by a human.Environments then differ only in which digest they point at, and promotion to prod is "point prod at the digest dev already validated" — the identical bytes. This also dissolves the #341 trade-off: dev auto-tracks
mainand every pod converges, because the pod template changes atomically in one rollout.The missing piece is small. CI already builds per-commit images; there is no deploy step and no cluster credential in any workflow — every deploy today is a manual
kubectl. So:mainbuild, a job sets the freshly-built digest ondev-duckdb-mcp. Needs one NRP service-account token (patch on that one Deployment) as a repo secret. Dev then tracksmainautomatically and convergently, with no hand-edited digest.Release promotion becomes: read the digest dev has been running, pin that in
k8s/deployment.yaml. No rebuild on the release tag — or, if a:vX.Y.Ztag is still wanted for humans, retag the existing digest rather than rebuilding it.And stop asking dev to host branches
Pick one and write it down, so the mandate stops being self-contradictory:
pull_requesttrigger building:pr-<n>, deployed to an ephemeraldev-pr-<n>; orsandbox-duckdb-mcpthat is explicitly hand-pinned and expected to hold arbitrary builds.Either way
devbecomes off-limits to hand-repointing, which is the behaviour that has repeatedly broken it.The invariant an agent can actually follow
Mechanically checkable, so it doesn't rely on anyone remembering: fail CI if any
image:ink8s/*.yamllacks@sha256:.Done when
:<git-sha>is genuinely immutable — a release tag no longer re-mints the sha tag of an already-built commitworkflow_dispatchfrom a non-mainref cannot publish:mainmain; no human editsk8s/dev-deployment.yaml's image linek8s/*.yamlimage reference lacking@sha256:Related: #341 (symptom), #342 (interim fix), #245 (pre-merge guidance gate — depends on dev being a trustworthy canary).