Skip to content

buildah actively breaks binfmt_misc registrations from an outer namespace #6611

@whitequark

Description

@whitequark

Issue Description

If buildah detects /etc/binfmt.d, /usr/lib/binfmt.d, etc it will break certain binfmt_misc configurations.

Consider the following container configuration:

  • Bare metal system: Debian, x86_64 architecture, registers interpreter /usr/libexec/qemu-binfmt/aarch64-binfmt-P with the POF flags (per default qemu-user-binfmt settings). The F flag causes the kernel to open the interpreter image and hold onto the fd. This means that the interpreter image doesn't need to exist inside nested containers.
    • CI job container: Debian, x86_64 architecture, does not install qemu-user-binfmt. This container can run ARM64 binaries directly. It provides /usr/lib/binfmt.d installed by default, which includes just Python.
      • Buildah container with BUILDAH_ISOLATION=chroot: Alpine, ARM64 architecture, does not touch binfmt_misc. This container would be able to run ARM64 binaries directly, except that Buildah detected that the CI job container has a /etc/binfmt.d directory, mounted binfmt_misc, and added one entry here. Since mounting binfmt_misc overrides all registrations, this container cannot run ARM64 binaries as a result.

There doesn't seem to be a way to prevent Buildah from doing this (besides mv /etc/binfmt.d{,.bak}; mv /usr/lib/binfmt.d{,.bak} which I'm not a fan of), and it broke my CI workflows when migrating hosts due to some environment change I haven't been able to track down (I think on the pre-migration host Buildah couldn't mount binfmt_misc?..), so I consider this behavior a bug.

Steps to reproduce the issue

Steps to reproduce the issue:

  1. Provision a system with Podman and binfmt_misc helpers for a foreign architecture installed (this is broadly available, e.g. on a Debian VM you can run sudo apt install podman qemu-user-binfmt)
  2. Run podman run --rm -it docker.io/library/debian:latest and start using the new shell.
  3. Run apt update -y; apt install -y buildah
  4. Run buildah from --arch arm64 --pull docker.io/library/alpine:latest
  5. Run buildah run --isolation chroot alpine-working-container /bin/echo works
  6. Observe process failed to start with error: fork/exec /bin/echo: exec format error
  7. Run mv /etc/binfmt.d{,.bak}; mv /usr/lib/binfmt.d{,.bak}
  8. Run buildah run --isolation chroot alpine-working-container /bin/echo works
  9. Observe works

Describe the results you received

Buildah breaks binfmt_misc defined in an outer namespace

Describe the results you expected

Buildah does not break binfmt_misc defined in an outer namespace

buildah version output

Version:         1.39.3
Go Version:      go1.24.4
Image Spec:      1.1.0
Runtime Spec:    1.2.0
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.34.2
Git Commit:      
Built:           Thu Jan  1 00:00:00 1970
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

buildah info output

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "debian",
            "version": "13"
        },
        "MemFree": 7335829504,
        "MemTotal": 67140960256,
        "OCIRuntime": "crun",
        "SwapFree": 67742584832,
        "SwapTotal": 68719472640,
        "arch": "amd64",
        "cpus": 12,
        "hostname": "777659f15a04",
        "kernel": "6.12.43+deb13-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "839h 25m 3.06s (Approximately 34.96 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 1
        },
        "GraphDriverName": "overlay",
        "GraphOptions": null,
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "overlayfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Supports shifting": "true",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 1
        },
        "RunRoot": "/run/containers/storage"
    }
}

Provide your storage.conf

n/a

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

I tracked down the behavior around binfmt_misc in the source code and I'm reasonably confident it's present in the latest commit as of filing this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions