feat(hatchet): two engine replicas with anti-affinity

[raisedadead]

No description provided.

[github-actions]

Run Details - tfws-ops-test

Terraform Cloud Plan Output

Plan: 6 to add, 0 to change, 0 to destroy.

Details : https://app.terraform.io/app/freecodecamp/workspaces/tfws-ops-test/runs/run-EBopLNhHGFAR7ifr

Warning

Please note that the plan output provided may not accurately reflect the impact on the Terraform project you are currently working on in this Pull Request. The CI checks are merely a sanity test to verify that the versions in the lock file are valid and functional. Confirm the actual Terraform plan by running the corresponding project on your machine or on TFC.

[ShaunSHamilton]

I do not know if you care for my LLM-helped review, but here is something possibly worth a double-check, and some doc update:

rolling-update deadlock

Kubernetes defaults results in no space for failure, which could lead to 600s stall:

• maxSurge: 25% -> ceil(0.25 × 2) = 1
• maxUnavailable: 25% -> floor(0.25 × 2) = 0

Required anti-affinity means that surge pod needs a node holding no engine pod.

Suggested fix is to add maxUnavailable: 1 config to rolling update strategy:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 0

node-drain-maintenance doc outdated

docs/runbooks/12-node-drain-maintenance.md suggests that hatchet blocks drains. This PR invalidates it and does not touch it.

Stale content:

• The table: hatchet-engine | 1 | k3s-2 | minAvailable: 1 | 0 | Blocks indefinitely (since 2026-08-23) - every column except the PDB value is now wrong.
• "Blast radius" - hatchet-engine no longer implies a scheduling outage on eviction of one pod.
• "Closed gap - the hatchet PDB shipped 2026-08-23" - "the intended trade: an outage the operator times beats one the scheduler picks" is explicitly reversed by this PR.
• The procedure: "Node holding hatchet-engine - the drain will hang... Scale the deployment to zero, drain, uncordon, then scale back to 1." An operator following this today would take an unnecessary
  full outage.
• "Both blocking workloads presently sit on the same node, so today one node needs the manual step and two drain cleanly" - only artemis-postgresql blocks now.

antiAffinity stringly-typed enum that fails silently

  {{- if eq .Values.engine.antiAffinity "required" }}

One implemented branch, no values.schema.json for this chart, no else. antiAffinity: Required, antiAffinity: preferred, or a typo all render no affinity block at all - the pods silently lose node-spread while the values file reads as though they have it. That is the exact failure this PR exists to prevent, made invisible.

Either drop the toggle and hardcode the block (only one value is ever used), or make the enum total:

  {{- if eq .Values.engine.antiAffinity "required" }}
        affinity: ...
  {{- else if eq .Values.engine.antiAffinity "preferred" }}
        affinity: ...
  {{- else if .Values.engine.antiAffinity }}
  {{- fail (printf "engine.antiAffinity must be required|preferred, got %q" .Values.engine.antiAffinity) }}
  {{- end }