fix(replicated-deploy): Make the Replicated deploy job re-runnable and its failures more legible

[dylan-openhands]

Description

The unstable deploy job failed with Error: deploy rejected: <empty response> and there was no way to act on it — the script never inspects HTTP status, so a bare 4xx from KOTS and a dropped connection produce byte-identical output. Re-running didn't help either: a cursor only appears under /updates while it's still an upstream update, so any re-run of a deploy that already got somewhere failed with cursor N never became an available update, which is not what happened.

Two changes, plus first test coverage for this script.

scripts/replicated_deploy.sh

Resolve the cursor's state once, up front, from a single /apps call and give each state a path:

Cursor is…	Before	Now
currently deployed	never became an available update	verify health, exit 0
pending (downloaded)	hard fail, "deploy from the console"	resume via sequence/$SEQ/deploy
an upstream update	upgrade-service path	unchanged
a past version	never became an available update	refused as a rollback, names the console

So the job is now idempotent for a given cursor — run it fifty times and it converges instead of going red.

api() now captures the response code via curl -w '%{stderr}…', so why() always has something to report: .error → raw body → HTTP 404 / curl: (7) … HTTP 000. <empty response> can't recur, and "KOTS refused" is now distinguishable from "never got a reply".

Also guarded the new up-front /apps read — jq '.apps[0].downstream' returns null with exit 0 on an error body, so without it a transient 401 would skip all three branches and spin out the select loop for three minutes before reporting the wrong thing.

Last, preflight_gate no longer prints preflights ok when KOTS recorded no results. It says preflights: none recorded, because on this fleet .preflightResult.result is an empty string on every sequence, healthy ones included — the old message was claiming a check that never reported.

scripts/test_replicated_deploy.py — new. Eight cases, each running the real script against a stub KOTS on localhost, so they cover the shell's control flow rather than a reimplementation of it. Runs in ~5s; test-scripts.yml already triggers on scripts/**.

Helm Chart Checklist

N/A — no charts touched.

Additional Notes

Verified read-only against unstable rather than reasoned about: the branching was replayed against the live /apps payload (489 → deployed, 491 → upstream, 480 → past), and the downstream sequence/$SEQ/preflight/result path was confirmed to exist with the shape the script expects.

Two things worth knowing:

• This does not explain the original rejection. It makes the next one legible. My first theory — that the empty preflight result meant the script was deploying mid-run — died when the probe showed that empty result is normal on every sequence here, so the root cause is still open.
• The POST sequence/$SEQ/deploy response shape is the one thing I couldn't verify; it needs a real pending version to exercise and there wasn't one. If it answers 204 rather than {"success":true}, the run fails with deploy of sequence N rejected: HTTP 204, naming the exact fix.