Skip to content

fix(bootc-build): use kernel overlayfs in reusable-build - #443

Open
KiKaraage wants to merge 1 commit into
projectbluefin:mainfrom
KiKaraage:fix/reusable-build-native-overlay
Open

fix(bootc-build): use kernel overlayfs in reusable-build#443
KiKaraage wants to merge 1 commit into
projectbluefin:mainfrom
KiKaraage:fix/reusable-build-native-overlay

Conversation

@KiKaraage

@KiKaraage KiKaraage commented Aug 31, 2026

Copy link
Copy Markdown

What does this change?

Sets native-overlay: "true" on the setup-runner call in reusable-build.yml. Rootful podman switches from fuse-overlayfs 1.16 to kernel overlayfs for image builds.

Why

Every bluefin :testing build since 2026-08-13 fails in the extension-builder stage with SELECT hnum, blob FROM 'Packages': 11: database disk image is malformed (bluefin#995, bluefin#1167). Bisected 2026-08-31 with a two-stage probe build (Containerfile.probe on bluefin branch ci/rpmdb-probe) that replicates production Stage 1 verbatim, then fingerprints the rpmdb (sha256, sqlite quick_check/integrity_check, rpm -qa) at the end of Stage 1 and again as inherited by the next stage.

Findings, all on the production base image digest and the resolute builder stack:

checkpoint script fuse-overlayfs (current) native-overlay
off corrupt, sha differs across stage commit byte-identical, dnf install passes
on corrupt, sha differs across stage commit byte-identical, dnf install passes

The rpmdb is clean (quick_check: ok, 1768 packages readable) at the end of Stage 1 and inherits byte-different and structurally damaged (freelist mismatch, btreeInitPage errors, damaged header #1809) after the stage commit. Same byte size, different bytes. The corruption channel is the fuse-overlayfs commit/restore path, not WAL journal mode, not dnf, not the cache. The checkpoint workaround merged in bluefin#1153 is irrelevant to the bug and its revert is prepared separately.

Probe runs: native+checkpoint, native no-checkpoint, fuse parity.

Note: runners currently skip the BTRFS loopback (/mnt absent, bluefin#1120); the passing probe legs ran in exactly that configuration, so the native-overlay + skipped-loopback combination is validated by the probe itself.

Consumer validation

Consumer PR: projectbluefin/bluefin#1177
Consumer CI run: https://github.com/projectbluefin/bluefin/actions/runs/33406727723
Out-of-org consumer impact: N/A — aurora and bazzite do not consume reusable-build.yml via @v1; the input change affects only workflows that call this reusable workflow (bluefin, bluefin-lts, dakota). The Justfile contract is unchanged.

  • Opened a draft consumer PR pinned to this branch SHA
  • Linked a passing consumer CI run that exercised this change
  • Evaluated out-of-org consumers (ublue-os/aurora, ublue-os/bazzite) and documented the impact above

Checklist

  • I am using an agent and I take responsibility for this PR (if AI-assisted)
  • Conventional commit message (feat:, fix:, chore:, etc.)
  • No hardcoded secrets or credentials

fuse-overlayfs 1.16 corrupts the rpmdb SQLite file across buildah's
stage commit/restore path: a quick_check-clean 95 MB database at the
end of a stage inherits byte-different and structurally damaged in the
next stage, killing every dnf transaction in later stages. This has
frozen bluefin :testing builds since 2026-08-13 (bluefin#995).

Bisected 2026-08-31 with a two-stage probe build on the production
base image. Native overlay passes byte-identical with the dnf install
completing; fuse fails deterministically with the same damage
signature in every leg. Full evidence in the PR.

Assisted-by: GLM via GitHub Copilot

@hanthor hanthor left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One-line change (native-overlay: "true" on the setup-runner call in reusable-build.yml), but backed by a rigorous bisection: a two-stage probe Containerfile that fingerprints the rpmdb (sha256, sqlite quick_check/integrity_check, rpm -qa) at the end of Stage 1 and again as inherited by Stage 2, run across a 2x2 matrix of storage driver (fuse-overlayfs vs. kernel/native overlay) x checkpoint script (on/off). Results are unambiguous: fuse-overlayfs corrupts the rpmdb across the stage-commit/restore boundary regardless of the checkpoint script, while native overlay produces byte-identical images in both cases. That correctly identifies the storage driver's commit/restore path as the actual corruption channel, not WAL journal mode.

This is an important correction to the earlier root-cause theory: the WAL-checkpoint fix merged in bluefin#1153 (which I approved as "the root-cause fix" in an earlier pass) is shown here to be irrelevant to the actual bug — the checkpoint-on and checkpoint-off legs both fail identically under fuse-overlayfs. The PR body says the revert of that workaround is prepared separately once this merges and @v1 advances, which is the right order of operations (pulling the workaround before the real fix lands would leave builds broken on fuse-overlayfs).

Verified: the consumer-validation PR (bluefin#1177) is linked and its CI run is green, satisfying the org's out-of-org-consumer and consumer-CI checklist items (aurora/bazzite don't consume this workflow via @v1, so blast radius is scoped to bluefin/bluefin-lts/dakota). No status checks are configured on the actions repo itself for workflow-only PRs — validation correctly happens via the consumer PR instead, which is documented and satisfied.

Flag for the user: this supersedes the root-cause understanding behind bluefin#1153 and (by extension) the parallel single-checkpoint fix in bluefin-lts#538 — worth reviewing the prepared revert PR when it lands, and re-checking whether bluefin-lts needs the same native-overlay change rather than keeping its own WAL-checkpoint workaround.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants