fix: install bootupd and unblock the common E2E suite (#492)

[hanthor]

Problem

The common post-merge E2E suite fails against bluefin:lts-testing on every run (#492) with three distinct errors:

1. error: Installing to disk: bootupd is required for ostree-based installs
2. bootupctl[1019]: error: Failed adopt and update: opening EFI dir: No such file or directory (→ bootloader-update.service fails at boot)
3. error: Cannot operate; \LockLayering=true` in configuration`

Changes

build_scripts/packages/base.toml — add bootupd to the [install] list. The LTS image never installed bootupd itself and relied on the quay.io/centos-bootc/centos-bootc:c10s base shipping it; installing it explicitly guarantees bootc install to-disk has it regardless of base-image churn. This addresses failure 1. (Upstream projectbluefin/bluefin does the same — bootupd is in its build_files/packages/base.toml install list.)

build_scripts/26-packages-post.sh — populate bootupd's update payload at build time. The Containerfile mounts a tmpfs over /boot during the build, which hides the base image's /boot/efi content, so bootupd has no EFI files to work from. The script now reinstalls the arch-appropriate EFI bootloader packages (grub2-efi-x64/shim-x64, or -aa64 on aarch64) to materialize their payload, then runs bootupctl backend generate-update-metadata, which copies it into /usr/lib/bootupd/updates — under /usr, so it is preserved in the final image. This addresses failure 2.

tests/unit/packages_test.bats — regression test asserting bootupd stays in the [install] list.

What was NOT changed, and why

LockLayering=true (failure 3) is left as-is. It is set by this repo in system_files/etc/rpm-ostreed.conf, but the comment there marks it as deliberate Bluefin policy: local layering with rpm-ostree install is an unsupported configuration that can lead to upgrade issues. The E2E harness hits it because it uses rpm-ostree install to inject its test tooling into the VM; the right fix is on the harness side (e.g. flip LockLayering=false in the VM before installing test tooling, or deliver tooling another way), not weakening the shipped image's policy.

Fixes #492

---

🤖 Generated with Claude Code

https://claude.ai/code/session_01KSjBPVDcnK71WveusaK7Nc

---

Generated by Claude Code

[castrojo]

The approach matches the #492 diagnosis and the base.toml addition plus bats regression test look right — but the new build step breaks the image build itself. From the x86_64 build log (run 31239447088, job 93057856610), after the reinstall succeeds:

+ rm -rf /usr/lib/bootupd/updates
+ bootupctl backend generate-update-metadata
Generated update layout for BIOS: grub2-tools-1:2.12-54.el10.x86_64
Moving usr/lib/ostree-boot/efi/EFI to usr/lib/bootupd/updates/EFI
error: generating metadata failed: Failed to find "/usr/lib/ostree-boot/efi/EFI"
Error: building at STEP "RUN --mount=type=tmpfs,dst=/opt ...": while running runtime: exit status 1

Reinstalling grub2-efi-x64/shim-x64 materializes the EFI payload, but not at /usr/lib/ostree-boot/efi/EFI where bootupctl's generate-update-metadata looks for it. Both arch image builds (x86_64 and aarch64) and the nvidia build fail the same way, so this can't merge as-is — the fix needs to place the EFI tree where bootupctl expects it (or point bootupctl at the right source) before generate-update-metadata runs.

Two smaller notes: (1) Lint & syntax also fails here, but that's the pre-existing hadolint DL3063 on Containerfile:13 (fixed by #501), not this change. (2) The deliberate decision to leave LockLayering=true is well-reasoned in the PR body — agreed that belongs on the harness side.

[hanthor]

@castrojo you were right that this can't merge, and digging into why turned up something worse: the build step this PR adds is not just broken, it's destructive — and the problem it's solving doesn't exist.

1. The base image already ships a complete bootupd payload. From quay.io/centos-bootc/centos-bootc:c10s, untouched:

$ rpm -q bootupd
bootupd-0.2.35-1.el10.x86_64

$ find /usr/lib/bootupd/updates -maxdepth 3
/usr/lib/bootupd/updates/EFI/BOOT/BOOTX64.EFI
/usr/lib/bootupd/updates/EFI/BOOT/fbx64.efi
/usr/lib/bootupd/updates/EFI/centos/{shim,shimx64,shimx64-centos,mmx64,grubx64}.efi
/usr/lib/bootupd/updates/EFI/centos/BOOTX64.CSV
/usr/lib/bootupd/updates/EFI.json
/usr/lib/bootupd/updates/BIOS.json

2. And so does the image we actually ship. From ghcr.io/projectbluefin/bluefin-lts:testing:

$ rpm -q bootupd
bootupd-0.2.35-1.el10.x86_64
$ cat /usr/lib/bootupd/updates/EFI.json
{"timestamp":"2026-07-28T16:03:32Z","version":"grub2-efi-x64-1:2.12-54.el10.x86_64,shim-x64-16.1-1.el10.x86_64", ...}

So both the bootupd addition to base.toml and the payload-generation step are solving a problem the image doesn't have.

3. The rm -rf is the dangerous part. The step does:

rm -rf /usr/lib/bootupd/updates          # deletes the CORRECT payload the base image ships
bootupctl backend generate-update-metadata   # then fails, aborting the build

Today the build fails, which is loud and safe. If someone "fixed" this by making the last line non-fatal, the image would ship with no bootupd payload at all — strictly worse than doing nothing, and a silent boot-update failure rather than a build failure.

4. generate-update-metadata cannot succeed in a container at all. It's not our build environment or the tmpfs. Reproduced in the pristine base image with no modifications:

$ bootupctl backend generate-update-metadata
Generated update layout for BIOS: grub2-tools-1:2.12-54.el10.x86_64
Moving usr/lib/ostree-boot/efi/EFI to usr/lib/bootupd/updates/EFI
error: generating metadata failed: Failed to find "/usr/lib/ostree-boot/efi/EFI"

/usr/lib/ostree-boot/efi/ exists but is empty — upstream populates it during compose, generates the payload, and the source tree ends up drained. The command is a compose-time one-shot, not something a derived build can re-run. And the reinstall doesn't help because those RPMs install to /boot/efi/EFI/..., not /usr/lib/ostree-boot/efi/EFI:

$ rpm -ql grub2-efi-x64 | grep EFI
/boot/efi/EFI/centos/grub.cfg
/boot/efi/EFI/centos/grubx64.efi
$ rpm -ql shim-x64 | grep EFI
/boot/efi/EFI/BOOT/BOOTX64.EFI
...

Which is exactly the mismatch in your review — and it's unfixable by relocating files, because bootupd wants the ostree-boot staging tree that only exists mid-compose.

---

Where this leaves #492. The stated cause — "bootupd missing, breaking bootc install to-disk" — is not true of the current image. Whatever the E2E suite is actually hitting, it isn't a missing bootupd or a missing update payload. I'd rather reopen that investigation from the real E2E failure than keep pushing this diff.

Moving to draft. The right end state for this branch is almost certainly delete the whole build step and the base.toml line, leaving at most the bats regression test — and even that should assert the payload exists rather than that bootupd is in our install list, since it comes from the base image.

I'll take #492 back to the E2E logs and re-diagnose from there.

(The Lint & syntax red here was always the pre-existing Containerfile:13 DL3063, since fixed by #501 on testing — unrelated to this diff, as you noted.)

[castrojo]

Deep-dive assessment: I believe this PR's premise is invalid against the current image, and the right fix for #492 is elsewhere.

bootupd and its EFI payload are already present in current builds. Verified by inspecting the images directly (x86_64):

• Base quay.io/centos-bootc/centos-bootc:c10s (both latest and the pinned sha256-feea845d…): bootupd rpm installed AND /usr/lib/bootupd/updates/ fully populated (EFI/BOOT/BOOTX64.EFI, EFI/centos/shimx64.efi, grubx64.efi, etc.). Nothing in this repo's build removes it ([remove] in base.toml is only setroubleshoot).
• Fresh ghcr.io/projectbluefin/bluefin-lts:testing (created 2026-08-09, run 31289693969): bootupd-0.2.35-1.el10 installed, /usr/bin/bootupctl present, and the full /usr/lib/bootupd/updates/EFI payload intact. No reinstall/generate-update-metadata step is needed.

The E2E failures in #492 were measured against a stale tag. ghcr.io/projectbluefin/bluefin:lts-testing (what common's e2e/promotion-candidate workflows point at) hasn't been published since 2026-05-30 (latest dated tag: lts-testing.20260530). That May image indeed has the broken state: bootupd present but /usr/lib/bootupd/updates/ contains only BIOS.json/EFI.json with the EFI/ payload missing — which exactly produces "Failed adopt and update: opening EFI dir". The current pipeline publishes to ghcr.io/projectbluefin/bluefin-lts:testing, which is healthy. Per common's docs/skills/image-registry.md, the canonical LTS image is projectbluefin/bluefin-lts, so bluefin:lts-testing appears to be a legacy alias that stopped being updated.

Proposed direction instead of this PR:

1. In common, repoint promotion-candidate-e2e.yml (and the disabled e2e.yml matrix entry) from bluefin:lts-testing to bluefin-lts:testing, or republish the legacy alias. That likely resolves failures 1–2.
2. Failure 3 (LockLayering=true) is harness-side, as this PR's body already argued.
3. Note the testsuite harness deliberately passes --bootloader systemd and tolerates "bootupd is required for ostree-based installs" (bootc install exited 1 (bootupd expected — continuing)), so that log line is expected noise for every image, not LTS-specific breakage.

Because the image is already correct, the generate-update-metadata step added here is unnecessary — and as the review showed, it breaks the build because bootupctl sources from /usr/lib/ostree-boot/efi/EFI (empty in both the base image and at build time under the /boot tmpfs; RPM payload lands in /boot/efi/EFI, a different path). Recommend closing this PR unmerged and filing the tag/harness fix against common. Happy to be proven wrong if a current bluefin-lts:testing install still reproduces the bootupd errors.