fix(bootc-build): use kernel overlayfs in reusable-build - #443
Conversation
fuse-overlayfs 1.16 corrupts the rpmdb SQLite file across buildah's stage commit/restore path: a quick_check-clean 95 MB database at the end of a stage inherits byte-different and structurally damaged in the next stage, killing every dnf transaction in later stages. This has frozen bluefin :testing builds since 2026-08-13 (bluefin#995). Bisected 2026-08-31 with a two-stage probe build on the production base image. Native overlay passes byte-identical with the dnf install completing; fuse fails deterministically with the same damage signature in every leg. Full evidence in the PR. Assisted-by: GLM via GitHub Copilot
hanthor
left a comment
There was a problem hiding this comment.
One-line change (native-overlay: "true" on the setup-runner call in reusable-build.yml), but backed by a rigorous bisection: a two-stage probe Containerfile that fingerprints the rpmdb (sha256, sqlite quick_check/integrity_check, rpm -qa) at the end of Stage 1 and again as inherited by Stage 2, run across a 2x2 matrix of storage driver (fuse-overlayfs vs. kernel/native overlay) x checkpoint script (on/off). Results are unambiguous: fuse-overlayfs corrupts the rpmdb across the stage-commit/restore boundary regardless of the checkpoint script, while native overlay produces byte-identical images in both cases. That correctly identifies the storage driver's commit/restore path as the actual corruption channel, not WAL journal mode.
This is an important correction to the earlier root-cause theory: the WAL-checkpoint fix merged in bluefin#1153 (which I approved as "the root-cause fix" in an earlier pass) is shown here to be irrelevant to the actual bug — the checkpoint-on and checkpoint-off legs both fail identically under fuse-overlayfs. The PR body says the revert of that workaround is prepared separately once this merges and @v1 advances, which is the right order of operations (pulling the workaround before the real fix lands would leave builds broken on fuse-overlayfs).
Verified: the consumer-validation PR (bluefin#1177) is linked and its CI run is green, satisfying the org's out-of-org-consumer and consumer-CI checklist items (aurora/bazzite don't consume this workflow via @v1, so blast radius is scoped to bluefin/bluefin-lts/dakota). No status checks are configured on the actions repo itself for workflow-only PRs — validation correctly happens via the consumer PR instead, which is documented and satisfied.
Flag for the user: this supersedes the root-cause understanding behind bluefin#1153 and (by extension) the parallel single-checkpoint fix in bluefin-lts#538 — worth reviewing the prepared revert PR when it lands, and re-checking whether bluefin-lts needs the same native-overlay change rather than keeping its own WAL-checkpoint workaround.
Generated by Claude Code
What does this change?
Sets
native-overlay: "true"on thesetup-runnercall inreusable-build.yml. Rootful podman switches from fuse-overlayfs 1.16 to kernel overlayfs for image builds.Why
Every bluefin
:testingbuild since 2026-08-13 fails in theextension-builderstage withSELECT hnum, blob FROM 'Packages': 11: database disk image is malformed(bluefin#995, bluefin#1167). Bisected 2026-08-31 with a two-stage probe build (Containerfile.probeon bluefin branchci/rpmdb-probe) that replicates production Stage 1 verbatim, then fingerprints the rpmdb (sha256, sqlite quick_check/integrity_check, rpm -qa) at the end of Stage 1 and again as inherited by the next stage.Findings, all on the production base image digest and the resolute builder stack:
The rpmdb is clean (
quick_check: ok, 1768 packages readable) at the end of Stage 1 and inherits byte-different and structurally damaged (freelist mismatch, btreeInitPage errors,damaged header #1809) after the stage commit. Same byte size, different bytes. The corruption channel is the fuse-overlayfs commit/restore path, not WAL journal mode, not dnf, not the cache. The checkpoint workaround merged in bluefin#1153 is irrelevant to the bug and its revert is prepared separately.Probe runs: native+checkpoint, native no-checkpoint, fuse parity.
Note: runners currently skip the BTRFS loopback (/mnt absent, bluefin#1120); the passing probe legs ran in exactly that configuration, so the native-overlay + skipped-loopback combination is validated by the probe itself.
Consumer validation
Consumer PR: projectbluefin/bluefin#1177
Consumer CI run: https://github.com/projectbluefin/bluefin/actions/runs/33406727723
Out-of-org consumer impact: N/A — aurora and bazzite do not consume reusable-build.yml via @v1; the input change affects only workflows that call this reusable workflow (bluefin, bluefin-lts, dakota). The Justfile contract is unchanged.
Checklist
feat:,fix:,chore:, etc.)