Can't run amd64 apptainer in rosetta-mode apple silicon mac

[ermaolaoye]

I am currently working on setting up a Lima VM using the Apptainer template and encountered an issue that I need assistance with. Below is the command I used to create the VM:

limactl start template://apptainer --name=default --cpus=4 --memory=8 --disk=400 --vm-type=vz --rosetta --mount-writable --set=.ssh.forwardX11=true

My objective is to build an Apptainer image capable of running x86_64 Linux applications on my Apple Silicon MacBook. Here is the configuration .def file for my Apptainer setup:

Bootstrap: docker
From: amd64/ubuntu:22.04

......

However, when I attempt to execute the apptainer build command within the Lima VM, it seems the x86 Docker environments are not functioning as expected. Despite enabling the Rosetta environment, the setup appears to lack support for x86 Docker environments.

INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: while fetching image: initializing source oci:/root/.apptainer/cache/blob:faa055eb6d54466728420d33d6b3d8b877a2ee63c6f012641bdacab4957cbd3c: choosing an image from manifest list docker://amd64/ubuntu:22.04: no image found in image index for architecture arm64, variant "v8", OS linux

I am aware of the slower QEMU-based approach for running the VM, but I would prefer to maintain performance by continuing to use Rosetta and VZ. If anyone could guide me on correctly configuring Lima or modifying my Apptainer definition file to address this issue, it would be greatly appreciated. Thank you for your time and assistance in advance!

[afbjorklund]

You probably need to tell apptainer the build arch, somehow. If all else fails, by running an arm64 version of the application*.

* Normally there is a parameter like --arch or --platform, but some tools fall back to the compile time GOOS and GOARCH...

[afbjorklund]

There is a parameter for apptainer pull, like so:

$ apptainer pull --arch amd64 alpine.sif docker://alpine
INFO:    Using cached SIF image
$ apptainer run alpine.sif arch
x86_64

But there doesn't seem to be one for apptainer build?

internal/pkg/build/metadata.go: // Local builds currently always use the host architecture.

[afbjorklund]

A workaround seems to be to start from a sif instead of from docker, and then pull the sif with --arch amd64.

Bootstrap: localimage
From: ubuntu-22.04.sif

https://sylabs.io/2022/07/cross-architecture-containers-with-singularityce-pro/

---

EDIT: For some reason it is not always working with rosetta, even if it is supposed to be (with binfmt_misc)

the image's architecture (amd64) could not run on the host's (arm64)

But that's probably a bug in apptainer, if so.

[ermaolaoye]

A workaround seems to be to start from a sif instead of from docker, and then pull the sif with --arch amd64.

Bootstrap: localimage
From: ubuntu-22.04.sif

https://sylabs.io/2022/07/cross-architecture-containers-with-singularityce-pro/

Thank you so much for taking the time to help me with my issue. I tried the workaround you suggested, but unfortunately, it didn’t seem to work for my app's environment. However, I did explore using QEMU instead, and the performance was acceptable. I truly appreciate the effort and time you put into assisting me. Wishing you a wonderful day bro!

[afbjorklund]

It does work with qemu-user on Linux, and that is quicker than running the whole VM with qemu-system.

EDIT: The bug is something with sandbox: open /proc/sys/fs/binfmt_misc/status: no such file or directory

[afbjorklund]

• The detection of emulation does not work with rootless fakeroot apptainer/apptainer#2730

The workaround, until fixed, is to comment out in /etc/subuid (then it will use fakeroot, rather than rootless)