freeCodeCamp · raisedadead · Sep 2, 2026 · Aug 28, 2026 · Aug 31, 2026 · Aug 31, 2026
@@ -10,7 +10,7 @@ All facts below were verified against hatchet v0.88.6 source (`pkg/config/server
 | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | Engine image tag == artemis `go.mod` hatchet version (v0.88.6)        | monorepo lockstep: one git tag builds engine images and the Go SDK                                                                                     |
 | `SERVER_GRPC_PORT=7077` everywhere                                    | binary default is **7070**; platform contract (artemis netpol, `HATCHET_ADDR`, upstream release compose) is 7077                                       |
-| `SERVER_SERVICES="all health"`                                        | `HasService()` is an exact match — `all` does NOT enable the `/live` + `/ready` health server (`run.go:161`); without `health` the probes kill the pod |
+| `SERVER_SERVICES="all health"`                                        | `all` routes to `runV1Config`, where `/live` + `/ready` start from `Runtime.Healthcheck` (`SERVER_HEALTHCHECK`, default true), NOT from this list. `health` is inert on v0.88.6; kept so a downgrade to the V0 path, where `HasService()` exact-match does require it, still starts the probes |
 | `SERVER_MSGQUEUE_KIND=postgres`                                       | binary default is rabbitmq; postgres is the supported single-store mode (`oneof=rabbitmq postgres`)                                                    |
 | keyset secret (`hatchet-config`) is generated once, never overwritten | regenerating keysets invalidates every issued worker token                                                                                             |
 | `helm -n artemis` (via `.deploy-flags.sh`)                            | the `release` recipe hardcodes `-n {{ app }}`; the hook's later `-n artemis` wins (pflag last-value-wins), avoiding a spurious `hatchet` namespace     |
@@ -42,6 +42,10 @@ Then:
 just release gxy-management hatchet
 ```
 
+Release outside 02:55–04:05 UTC. The old ticker deactivates on shutdown and the new one claims
+the crons on its next 15 s poll; a missed minute is not backfilled, so a release that straddles
+03:00 or 04:00 skips that night's run.
+
 Hook order (all idempotent):
 
 1. `hatchet-quickstart` (pre, -10) — `hatchet-admin k8s quickstart` generates cookie secrets + 3 encryption keysets into Secret `hatchet-config` (only fills missing keys).
@@ -61,18 +65,56 @@ helm -n artemis list | grep hatchet
 # 2. jobs green
 kubectl -n artemis get jobs -l app.kubernetes.io/instance=hatchet
 
-# 3. engine up + ready (startupProbe budget 120s)
-kubectl -n artemis get pods -l app.kubernetes.io/component=engine
-
-# 4. listening on 7077 (NOT 7070) + health
-kubectl -n artemis logs deploy/hatchet-engine | grep -i "grpc\|listen" | head
-kubectl -n artemis port-forward deploy/hatchet-engine 18733:8733 &
-curl -sf http://127.0.0.1:18733/live && curl -sf http://127.0.0.1:18733/ready
+# 3. BOTH engine replicas up, ready, and on DIFFERENT nodes.
+#    Two distinct nodes is the whole point of the required anti-affinity;
+#    a second Running pod on the same node means the affinity block did
+#    not render and the values file is lying to you.
+kubectl -n artemis get pods -l app.kubernetes.io/component=engine -o wide
+kubectl -n artemis get pods -l app.kubernetes.io/component=engine \
+  -o jsonpath='{range .items[*]}{.spec.nodeName}{"\n"}{end}' | sort -u | wc -l   # must be 2
+
+# 4. listening on 7077 (NOT 7070) + health, on EVERY pod.
+#    `logs deploy/` and `port-forward deploy/` each pick ONE pod, so at
+#    two replicas they leave the other unchecked. Loop by pod name.
+for p in $(kubectl -n artemis get pods -l app.kubernetes.io/component=engine -o name); do
+  echo "== $p"
+  kubectl -n artemis logs "$p" | grep -i "grpc\|listen" | head -3
+  kubectl -n artemis port-forward "$p" 18733:8733 >/dev/null 2>&1 &
+  sleep 2; curl -sf http://127.0.0.1:18733/live && curl -sf http://127.0.0.1:18733/ready; echo
+  kill $! 2>/dev/null
+done
 
 # 5. token minted
 kubectl -n artemis get secret hatchet-client-config -o jsonpath='{.data.HATCHET_CLIENT_TOKEN}' | head -c 16
 ```
 
+### 6. The double-fire gate — run this the MORNING AFTER, not at release time
+
+Two replicas both run every service (`SERVER_SERVICES="all"`). A cron schedule has one owner
+while that ticker heartbeats within 10 s (`ticker.sql:120`; heartbeat every 5 s, refresh every
+15 s). A ticker that misses two heartbeats hands its schedules over with up to 15 s of dual
+ownership, and a fire carries no dedup key. The controllers and scheduler are held by the tenant
+partitioner. If either mechanism fails, the nightly crons fire twice — and `tombstone-purge`
+performs destructive R2 deletes. Nothing at release time proves this; only a night does.
+
+Baseline measured 2026-08-31, before the two-replica change: a day with no deploys is **exactly
+two** runs, `tombstone-purge` at 03:00 UTC and `drift-detect` at 04:00 UTC, one fire each.
+
+```sh
+kubectl -n artemis exec artemis-postgresql-0 -- psql -U postgres -d hatchet -tAc \
+  "SELECT date_trunc('hour', r.inserted_at), w.name, count(*)
+   FROM v1_runs_olap r JOIN \"Workflow\" w ON w.id = r.workflow_id
+   WHERE r.inserted_at > now() - interval '2 days'
+     AND w.name IN ('tombstone-purge', 'drift-detect')
+   GROUP BY 1, 2 ORDER BY 1"
+```
+
+Every row must read `1`: one `tombstone-purge` in the 03:00 bucket and one `drift-detect` in the
+04:00 bucket per day. A `2` on either row means that cron double-fired: roll the engine back to one
+replica immediately (section E) and check the `tombstone-purge` audit rows in the `artemis` database
+for duplicate deletes before doing anything else. `gc-site` is excluded on purpose; its event-driven
+runs vary with deploys and would hide a double fire in a daily total.
+
 ## D. Wire artemis (separate release)
 
 1. Extract the token, add `HATCHET_CLIENT_TOKEN` to `management/artemis.env.enc` + the artemis values overlay (`artemis.values.yaml.enc` → `secretEnv.HATCHET_CLIENT_TOKEN`). sops is operator-only.
@@ -84,5 +126,7 @@ kubectl -n artemis get secret hatchet-client-config -o jsonpath='{.data.HATCHET_
 ## E. Rollback
 
 - artemis side: unset `HATCHET_ADDR` → worker + relay gate off at next boot; deploys/registry unaffected (stage-1 posture).
-- engine side: `helm -n artemis uninstall hatchet` removes engine + netpols. Secrets `hatchet-config`/`hatchet-client-config` are cluster-side artifacts created by the jobs (not helm-owned) and survive uninstall — keep them unless keyset rotation is intended. The hook resources (bootstrap SA/Role/RoleBinding, `hatchet-env-secret`) also survive uninstall (helm never garbage-collects hooks) — delete manually for full teardown.
+- engine side, immediate: `kubectl -n artemis scale deploy hatchet-engine --replicas=1` takes effect at once. Persist it with the line below, or the next `just release` restores two.
+- engine side, replica count only: `helm -n artemis upgrade hatchet <chart> --reuse-values --set engine.replicas=1` returns the single-replica posture without a teardown. Note this re-opens the ADR-022 §Prerequisite gate, so any destructive artemis write path depending on the engine is back to a single point of failure — tell the artemis owner.
+- engine side, full: `helm -n artemis uninstall hatchet` removes engine + netpols. Secrets `hatchet-config`/`hatchet-client-config` are cluster-side artifacts created by the jobs (not helm-owned) and survive uninstall — keep them unless keyset rotation is intended. The hook resources (bootstrap SA/Role/RoleBinding, `hatchet-env-secret`) also survive uninstall (helm never garbage-collects hooks) — delete manually for full teardown.
 - GC has been live in gxy-management since the SHIP7 cutover (`CLEANUP_DRY_RUN: "false"`, `CLEANUP_BLAST_CAP: "10"`, `values.production.yaml`) — the worker moves real bytes, capped at 10 tombstoned deploys per run. Rolling back (unset `HATCHET_ADDR`) stops new GC runs immediately but does not undo ones already completed; those stay recoverable under `_trash/` for `CLEANUP_RECOVERY_DAYS` (7d default) before the purge cron hard-deletes.
@@ -12,7 +12,7 @@ The `artemis` namespace holds three workloads with three different disruption po
 | --- | --- | --- | --- | --- | --- |
 | `artemis` (deploy proxy) | 3 | all three | `minAvailable: 2` | 1 | Drains cleanly, one node at a time |
 | `artemis-postgresql` | 1 | k3s-2 | `minAvailable: 1` | **0** | **Blocks indefinitely** |
-| `hatchet-engine` | 1 | k3s-2 | `minAvailable: 1` | **0** | **Blocks indefinitely** (since 2026-08-23) |
+| `hatchet-engine` | 2 | two of three | `minAvailable: 1` | 1 | Drains cleanly, one node at a time (since 2026-08-31) |
 
 Confirm the live numbers before trusting the table:
 
@@ -30,7 +30,7 @@ Pod placement is not pinned, so re-check which node holds Postgres and the engin
 
 **`artemis-postgresql`** — a single replica carrying both tenant databases (`artemis` and `hatchet`). Its PDB permits zero voluntary evictions, so the drain hangs rather than proceeding. This is deliberate: the alternative is an unscheduled control-plane outage. Serving is unaffected while it is down — `/readyz` returns `200 {"ready":true,"degraded":true}` and deploys still write to R2 — but GC, the index and the audit log stop. See [11-artemis-pg-outage-drill.md](11-artemis-pg-outage-drill.md) for the rehearsed boundary.
 
-**`hatchet-engine`** — the durable-execution substrate. Its PDB permits zero voluntary evictions, so a drain hangs. Before 2026-08-23 it had no PDB and was evicted instantly. The impact of losing it is narrower than it looks:
+**`hatchet-engine`** — the durable-execution substrate. At two replicas with `minAvailable: 1` it permits one voluntary eviction, so a drain rolls rather than hangs. Required anti-affinity keeps the two pods on different nodes, so a single drain never takes both. Losing one replica is a scheduling pause, not an outage: a clean shutdown hands the tenant over at once, an unclean loss waits up to 2 minutes (partition heartbeat 20 s, inactive at 60 s, rebalance every 60 s). The impact below applies only if both go, which needs two nodes down at once:
 
 - **Not affected:** deploy init, upload, finalize, promote, rollback, and serving. Those paths write to R2 and Postgres directly and never touch Hatchet.
 - **Affected:** the scheduled jobs — `tombstone-purge` (03:00 UTC), `drift-detect` (04:00 UTC), and the event-triggered `gc-site`.
@@ -39,13 +39,13 @@ Pod placement is not pinned, so re-check which node holds Postgres and the engin
 
 The engine keeps no local state; its run history lives in the `hatchet` database on `artemis-postgresql-0`, so eviction risks scheduling continuity, not data.
 
-## Closed gap — the hatchet PDB shipped 2026-08-23
+## History — one blocking workload remains
 
-`just release gxy-management hatchet` took the release to revision 2 and applied `hatchet-engine` at `minAvailable: 1`. **`just release gxy-management artemis` does not release the hatchet chart** — the two charts are separate releases in one namespace. That is why the template sat unapplied from 2026-06-06.
+**2026-08-23.** `just release gxy-management hatchet` took the release to revision 2 and applied `hatchet-engine` at `minAvailable: 1`. **`just release gxy-management artemis` does not release the hatchet chart** — the two charts are separate releases in one namespace. That is why the template sat unapplied from 2026-06-06. At one replica that PDB blocked a drain, which was the intended trade at the time: an outage the operator times beats one the scheduler picks.
 
-The engine now blocks a drain instead of being evicted without warning, which is the intended trade: an outage the operator times beats one the scheduler picks.
+**2026-08-31.** The engine went to two replicas with required anti-affinity, per ADR-022 §Prerequisite. That reverses the trade rather than refining it: the engine no longer blocks a drain and no longer needs the manual scale-to-zero step, because one replica always survives. `artemis-postgresql` is now the ONLY workload in this namespace that blocks a drain.
 
-Both blocking workloads presently sit on the same node, so today one node needs the manual step and two drain cleanly. Placement is not pinned. Re-check before every drain — if the engine and Postgres land on different nodes, two nodes need it.
+Placement is not pinned. Re-check before every drain.
 
 ## Procedure
 
@@ -58,14 +58,14 @@ Both blocking workloads presently sit on the same node, so today one node needs
 3. Drain, do the maintenance, uncordon.
 4. Scale back to 1 and verify per [11-artemis-pg-outage-drill.md](11-artemis-pg-outage-drill.md) — `postgres.connected` in the artemis logs, `/readyz` no longer `degraded`, and the outbox backlog draining.
 
-**Node holding `hatchet-engine`** — the drain will hang, as with Postgres. Scale the deployment to zero, drain, uncordon, then scale back to 1. Prefer a window outside 03:00–04:30 UTC so the nightly check-ins are not recorded as missed. Confirm the workers re-attach afterwards: the engine log reports `listing actions for workers` with a non-zero count.
+**Node holding one `hatchet-engine` replica** — no special handling since 2026-08-31; `minAvailable: 1` at two replicas permits the eviction and the drain completes. Do NOT scale to zero: that step belonged to the single-replica posture and now causes an outage the PDB was about to prevent. Two cautions remain. Required anti-affinity means the evicted pod cannot reschedule until a node with no engine pod is free, so it stays `Pending` while the drained node is cordoned — expected, not a fault. And prefer a window outside 03:00–04:30 UTC so the nightly check-ins are not recorded as missed. Confirm the workers re-attach afterwards: the engine log reports `listing actions for workers` with a non-zero count.
 
 ## Verify after any drain
 
 ```sh
 kubectl -n artemis get pods -o wide          # all Running, spread across the surviving nodes
-kubectl -n artemis get pdb                   # artemis disruptionsAllowed back to 1
-kubectl -n artemis get pods -o wide          # confirm which node now holds PG + engine
+kubectl -n artemis get pdb                   # artemis AND hatchet-engine disruptionsAllowed back to 1
+kubectl -n artemis get pods -o wide          # confirm PG's node, and that the two engine pods are on different nodes
 curl -sS https://uploads.freecode.camp/healthz
 ```
 

@@ -7,5 +7,5 @@ description: >-
   Engine image tag MUST stay in lockstep with the artemis go.mod
   hatchet module version (monorepo single-tag releases).
 type: application
-version: 0.4.0
+version: 0.5.0
 appVersion: "0.88.6"
@@ -8,6 +8,16 @@ metadata:
     app.kubernetes.io/component: engine
 spec:
   replicas: {{ .Values.engine.replicas }}
+  # Required anti-affinity means a surge pod needs a node holding no
+  # engine pod. The k8s defaults at 2 replicas are maxSurge 1 and
+  # maxUnavailable 0, so a rollout would demand a third free node and
+  # stall for progressDeadlineSeconds when one is cordoned. Replace
+  # rather than surge: the engine is off the serving path.
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxUnavailable: 1
+      maxSurge: 0
   selector:
     matchLabels:
       {{- include "hatchet.engineSelectorLabels" . | nindent 6 }}
@@ -16,6 +26,27 @@ spec:
       labels:
         {{- include "hatchet.engineSelectorLabels" . | nindent 8 }}
     spec:
+      {{- if eq .Values.engine.antiAffinity "required" }}
+      affinity:
+        podAntiAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            - labelSelector:
+                matchLabels:
+                  {{- include "hatchet.engineSelectorLabels" . | nindent 18 }}
+              topologyKey: kubernetes.io/hostname
+      {{- else if eq .Values.engine.antiAffinity "preferred" }}
+      affinity:
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                labelSelector:
+                  matchLabels:
+                    {{- include "hatchet.engineSelectorLabels" . | nindent 20 }}
+                topologyKey: kubernetes.io/hostname
+      {{- else }}
+      {{- fail (printf "engine.antiAffinity must be \"required\" or \"preferred\", got %q" .Values.engine.antiAffinity) }}
+      {{- end }}
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       containers:
@@ -59,9 +90,12 @@ spec:
             # egress CNP drops it (PG+DNS only); disable at source.
             - name: SERVER_SECURITY_CHECK_ENABLED
               value: "false"
-            # "health" must be listed explicitly: HasService() is an
-            # exact match, "all" does NOT enable the /live + /ready
-            # server (cmd/hatchet-engine/engine/run.go:161).
+            # "health" is INERT on v0.88.6 and kept only so a downgrade
+            # to the V0 path still starts the probes. "all" routes to
+            # runV1Config, where /live + /ready come from
+            # Runtime.Healthcheck (SERVER_HEALTHCHECK, default true) and
+            # not from this list. The HasService() exact-match rule that
+            # once required the token applies to runV0Config only.
             - name: SERVER_SERVICES
               value: "all health"
             - name: SERVER_AUTH_COOKIE_DOMAIN

@@ -8,9 +8,10 @@ metadata:
     {{- include "hatchet.labels" . | nindent 4 }}
     app.kubernetes.io/component: engine
 spec:
-  # Single-replica engine: minAvailable 1 blocks voluntary evictions
-  # (node drains) so the dispatch outage window is operator-timed; run
-  # state itself is safe in Postgres.
+  # At two replicas minAvailable 1 permits one voluntary eviction, so a
+  # node drain rolls rather than blocks; run state is safe in Postgres.
+  # For a drain that lands during a rollout, see the strategy block in
+  # deployment.yaml — maxSurge 0 is what stops the two interacting.
   minAvailable: {{ .Values.pdb.minAvailable }}
   selector:
     matchLabels: