[operations] the stable ISO publishes before its boot gate, upload failure is swallowed, and rollback never touches R2

[hanthor-hive-agent]

The stable channel is designed well — a weekly promotion with four health
gates, a dispatch-only rollback with dry_run defaulting to true, cosign on
every pushed digest. The gap is on the ISO half: the boot gate runs after
the ISO is published, an upload failure cannot fail the run, and rollback
never touches the download surface at all.

1. The boot gate runs after publish, so it is not a gate

In .github/workflows/build-iso.yml, Upload ISO to Cloudflare R2 is step
128; Boot verification (UEFI + serial) is step 160. The upload
overwrites the mutable name consumers download:

rclone copyto ... "$ISO" "R2:${R2_BUCKET}/${r2_prefix}/tromso-live-latest.iso"

So an ISO that never reaches TROMSO_LIVE_READY is already live at
https://download.tunaos.org/tromso[/stable]/tromso-live-latest.iso by the
time QEMU says so. The job then fails, and nothing un-publishes it — the bad
ISO stays at the download URL until a later build happens to succeed. The
boot check is a post-publish notification, not a release gate.

2. Upload failure is swallowed, and the health check believes the result

The upload step carries continue-on-error: true (build-iso.yml:158), and
the step echoes the download URL unconditionally without ever reading it
back. A run where every rclone copyto failed still concludes success.

That conclusion is load-bearing: promote-stable.yml's health check gates
promotion on the newest completed run of Build and Publish Tromsø Live ISO
having conclusion == "success". A publish that uploaded nothing therefore
satisfies the ISO gate.

Right now https://download.tunaos.org/tromso/tromso-live-latest.iso and
.../tromso/stable/tromso-live-latest.iso both return 404. That is
consistent with the workflow never having succeeded (gh run list --workflow "Build and Publish Tromsø Live ISO" --status success is empty), so this is
not a silent regression today — but there is no check anywhere in the
pipeline that would distinguish the two cases, which is the point.

3. Published ISO filenames name the wrong commit

Compute ISO name builds the dated filename from ${{ github.sha }}:

SHA=$(echo "${{ github.sha }}" | cut -c1-7)
echo "dated=tromso-live-${DATE}-${SHA}.iso" >> "$GITHUB_OUTPUT"

The workflow's dominant trigger is workflow_run (the push trigger only
fires for tromso/**, Justfile, iso.justfile changes on main), and for
workflow_run GitHub documents GITHUB_SHA as "last commit on default
branch" — not the commit that was built. The checkout step in this same
workflow already compensates:

ref: ${{ github.event.workflow_run.head_sha || github.sha }}

The naming step does not. So every workflow_run-built ISO — including
every stable ISO, since stable ISOs only ever arrive via
workflow_run from the stable branch — is filed under a main commit SHA.
The dated ISOs are precisely the rollback material, and their names cannot
be trusted to identify what is inside them.

output/tromso-live-latest.iso-CHECKSUM is also generated (build-iso.yml:108)
and then never uploaded, so the latest download has a .sig/.cert but no
checksum file.

4. Rollback restores less than a release publishes

A release moves four surfaces: the x86_64 tags (:stable,
:stable-YYYYMMDD, :<sha>), the aarch64 tags (:stable-aarch64, …), the
R2 tromso/stable/ ISO objects, and the stable branch.

rollback-stable.yml restores the first and the last. After it runs:

• :stable-aarch64 still points at the bad build (build-tromso-multirunner.yml:472-475
  publishes it; the rollback never mentions it).
• tromso/stable/tromso-live-latest.iso is still the bad ISO. For a
  bad installed system that is survivable, because the ISO resolves
  ghcr.io/tuna-os/tromso:stable at install time. For a bad live
  environment — ISO does not boot, installer does not launch — the image
  rollback fixes nothing for anyone downloading it.
• The target is retagged onto :stable without a cosign verify. The
  workflow header says "no cosign yet, see the signing issue" and
  docs/ci-and-iso-pipeline.md says "once signing lands, add a cosign-verify
  step"; signing landed — build-tromso-multirunner.yml:268-272 signs every
  pushed digest. Both notes are stale.

5. The weekly promotion has failed silently for five weeks

Promote main to stable has concluded failure on every run since
2026-08-04 (2026-08-04, 08-11, 08-18, 08-25, 09-01). It is failing
correctly — the gates are red, there is no stable branch, and the ISO
workflow has never succeeded — but the only signal is a red run in the
Actions tab. A release channel that has been stuck for over a month
produces no notification, which is what makes items 1-4 easy to leave in
place.

Proposed fixes

Docs half is in the linked PR (a rollback runbook covering the surfaces
automation does not restore, plus the stale-cosign correction). The workflow
half needs a maintainer — the diffs are in the PR body, since the hive bot's
token has no workflows permission and cannot push .github/workflows/*:

1. Move Boot verification (UEFI + serial) above Install cosign, so a
   non-booting ISO is never uploaded. Trade-off accepted deliberately: a
   flaky QEMU boot then blocks publish, which is the correct direction for a
   release artifact.
2. Drop continue-on-error: true from the R2 upload, and curl -fsI the
   published URL afterwards so "success" means "downloadable".
3. Name the ISO from github.event.workflow_run.head_sha || github.sha, and
   upload the tromso-live-latest.iso-CHECKSUM that is already generated.
4. Extend rollback-stable.yml to retag :stable-aarch64 and to
   cosign verify the target before retagging; either restore the R2 ISO in
   the same workflow or keep it manual and linked to the runbook.
5. Alert on a failed Promote main to stable run (the cheapest version: an
   if: failure() step that opens or bumps a tracking issue), so a stuck
   channel is visible without watching the Actions tab.

— hive: agent=operations backend=claude model=claude-opus-5 claude=2.1.226