coreos-assembler breaks on ppc64le

[manojnkumar]

This is the error I see when trying to run this on the ppc64le architecture.

[root@rhel-ocpapp1 coreos]# coreos-assembler init https://github.com/coreos/fedora-coreos-config
Trying to pull quay.io/cgwalters/coreos-assembler...Getting image source signatures
Copying blob sha256:e69e955c514f72cef9e9e7db408266a32ba48133f0e2a27d5e4b52ed1995d864
 85.99 MB / 85.99 MB [=====================================================] 10s
Copying blob sha256:a8a4d821e15658a276717ed65d865469e12dabbde7cf1871c755025118b58f3b
 160 B / 160 B [============================================================] 0s
Copying blob sha256:99a83b419d3cb4c515d32e0af4ca354f48dd4fdc40872f7bd0e4b939a04afee8
 2.53 KB / 2.53 KB [========================================================] 0s
Copying blob sha256:8faffdba319aeddefcd0a7598774221bf6ac3e7ba136bc147e0231a74e6166fc
 1.16 KB / 1.16 KB [========================================================] 0s
Copying blob sha256:a417b5e014d6bfb6b81ce057971d82f67739a0c9ebd47dd615939a63255cd7a7
 561.86 MB / 561.86 MB [===================================================] 56s
Copying blob sha256:acf8382988a511a2e77d66b6db5d6ab926198625b1d451a21040458b984184ba
 3.51 MB / 3.51 MB [========================================================] 0s
Copying blob sha256:b4118eadddddafb8e15b73ac929c6cb0e40a53036d49d87c611982c9c66adf96
 169.87 MB / 169.87 MB [===================================================] 19s
Copying blob sha256:134fd40227f49150593822289e2fd1efdf359ec2730aaf6ab01488001eab3b65
 3.32 KB / 3.32 KB [========================================================] 0s
Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
 32 B / 32 B [==============================================================] 0s
Copying blob sha256:710caf03ad410b4ffd7ae487eb56e51e6a3b58f16a9711f551f03cbed8a40973
 137 B / 137 B [============================================================] 0s
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Skipping fetch of repeat blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
Writing manifest to image destination
Storing signatures
standard_init_linux.go:190: exec user process caused "exec format error"

[dustymabe]

hi @manojnkumar. Thanks for trying out coreos-assembler!

This is probably not going to work out of the box right now because we haven't done any enablement work to get it to happen yet. One example would be the container runtime is pulling an x86_64 container from quay.io (we haven't built it for non x86 architectures yet).

In order to get this to work you'll need to build the container locally on ppc64le hardware first and then use the container you just built as the assembler container instead of quay.io/cgwalters/coreos-assembler. Does that make sense?

You'll probably hit other issues along the way, but let's document them and we can try to get it working.

[manojnkumar]

I cloned this project and ran the coreos-assembler script directly and running into the following pre-requisites:

/home/cloud-user//coreos-assembler/coreos-assembler init https://github.com/coreos/fedora-coreos-config
...
error: Failed to find expected dependencies: rpm-ostree dnf-utils fedpkg rpmdistro-gitoverlay distribution-gpg-keys python2-gobject-base python3-gobject-base podman buildah jq awscli ignition python3-dateutil

@dustymabe : Could you provide me instructions on building a ppc64le container?

[manojnkumar]

Should I be building on RHEL, or somewhere else (CentOS etc.)?

[dustymabe]

I cloned this project and ran the coreos-assembler script directly

the tool primarily expects to be run from within a container. So you'd need to clone this repo and then do something like podman build -t coreos-assembler . (you can use docker there if you prefer). Then set up your alias to use coreos-assembler rather than quay.io/cgwalters/coreos-assembler and run it again.

Does that make sense?

Should I be building on RHEL, or somewhere else (CentOS etc.)?

I run on Fedora usually, but it shouldn't matter that much I don't think because the environment is in the container.

[manojnkumar]

docker build fails at this stage:

=False

• read line
• mv /etc/yum.repos.d/fedora.repo.new /etc/yum.repos.d/fedora.repo
• cat
• cat
  ---> fe56827f4074
  Removing intermediate container ca31db715dba
  Step 5/12 : RUN ./build.sh install_rpms
  ---> Running in 72e39808c1a3

++ pwd

• srcdir=/root/containerbuild
• install_rpms
• dnf -y distro-sync
  Repository 'fahc' is missing name in configuration, using id.
  Error: Failed to synchronize cache for repo 'updates'
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 1

[dustymabe]

* Error: Failed to synchronize cache for repo 'updates'

That's typically a network intermittent failure. If you run it multiple times does it continue to die in the same place?

cc @sinnykumari who might be able to try this on some ppc64le hardware she has.

[manojnkumar]

I tried this on several places, where I kept seeing similar errors. Finally I hit a server where I was able to go further:

Installed:
findutils.ppc64le 1:4.6.0-19.fc28

Complete!
++ grep -v '^#' /root/containerbuild/build-deps.txt

• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Failed to synchronize cache for repo 'dustymabe-ignition', disabling.
  Failed to synchronize cache for repo 'walters-buildtools-fedora', disabling.
  Last metadata expiration check: 0:00:00 ago on Wed Nov 7 19:05:26 2018.
  No match for argument: rpmdistro-gitoverlay
  Package python3-gobject-base-3.28.3-1.fc28.ppc64le is already installed, skipping.
  Error: Unable to find a match
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123

[dustymabe]

findutils.ppc64le 1:4.6.0-19.fc28

hmm. this seems odd. Are you trying to build an f28 image? We switched to f29 some time ago.

* Failed to synchronize cache for repo 'dustymabe-ignition', disabling.

There aren't any ppc64le builds for this for f28, there is for f29

  Failed to synchronize cache for repo 'walters-buildtools-fedora', disabling.

@cgwalters can you enable ppc64le for your copr repo? https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/

[dustymabe]

findutils.ppc64le 1:4.6.0-19.fc28

hmm. this seems odd. Are you trying to build an f28 image? We switched to f29 some time ago.

actually. i was mistaken.. this is the container build itself. I have opened #195 for us to switch to f29 for the assembler container.

[dustymabe]

For now, once colin has enabled ppc64le builds for his copr (wait to hear back from him here in this issue), you can switch to f29 in the Dockerfile and try again.

[manojnkumar]

Switching to f29 in the Dockerfile, hit the same issue:

Complete!

• dnf -y install /usr/bin/xargs
  Repository 'fahc' is missing name in configuration, using id.
  fahc 4.5 kB/s | 1.5 kB 00:00
  Copr repo for buildtools-fedora owned by walter 980 B/s | 341 B 00:00
  Failed to synchronize cache for repo 'walters-buildtools-fedora', ignoring this repo.
  Package findutils-1:4.6.0-21.fc29.ppc64le is already installed.
  Dependencies resolved.
  Nothing to do.
  Complete!
  ++ grep -v '^#' /root/containerbuild/build-deps.txt
• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Copr repo for buildtools-fedora owned by walter 1.1 kB/s | 341 B 00:00
  Failed to synchronize cache for repo 'walters-buildtools-fedora', ignoring this repo.
  Last metadata expiration check: 0:00:04 ago on Wed Nov 7 21:22:02 2018.
  No match for argument: rpmdistro-gitoverlay
  Error: Unable to find a match
  The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123

[cgwalters]

This is the same issue as #30 (comment) which was for aarch64.

Although, since COPR does support ppc64le I added that to the buildroots and did the "update revision" dance and started a new build: https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/build/820725/

Like I said in that ticket though...it may be simplest to drop rdgo from this container for now.

[dustymabe]

now that colin has enabled ppc64le and the build succeeded, you can try it again now and see where it breaks this time :)

[manojnkumar]

Thanks @cgwalters @dustymabe . It now fails here:

Package findutils-1:4.6.0-21.fc29.ppc64le is already installed.
Dependencies resolved.
Nothing to do.
Complete!
++ grep -v '^#' /root/containerbuild/build-deps.txt

• self_builddeps=golang
• echo golang
• grep -v '^#' /root/containerbuild/deps.txt
• xargs dnf -y install
  Repository 'fahc' is missing name in configuration, using id.
  Last metadata expiration check: 0:00:03 ago on Wed Nov 7 22:22:28 2018.
  Error:
  Problem 1: conflicting requests
  • package rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64 does not have a compatible architecture
  • nothing provides libpthread.so.0(GLIBC_2.2.5)(64bit) needed by rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64
  • nothing provides rpm-ostree-libs(x86-64) = 2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28 needed by rpm-ostree-2018.9.38-2c4231b3769a99af36fdd68d03c1b24468f3f5b2.d8a5bf5d7acae1bdd57001f2a39aa8e890f79c0f.fc28.x86_64
    Problem 2: package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package skopeo-1:0.1.32-2.dev.gite814f96.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • conflicting requests
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    Problem 3: package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package buildah-1.4-1.dev.git0a7389c.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • conflicting requests
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    Problem 4: conflicting requests
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1.3-4.gitdb08685.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1()(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.3)(64bit), but none of the providers can be installed
  • package podman-1:0.10.1-1.gite4a1553.fc29.ppc64le requires libostree-1.so.1(LIBOSTREE_2016.8)(64bit), but none of the providers can be installed
  • package ostree-libs-2018.8-1.fc29.ppc64le is excluded
    The command '/bin/sh -c ./build.sh install_rpms' returned a non-zero code: 123