coreos-assembler breaks on ppc64le #194
Comments
manojnkumar
changed the title
|
hi @manojnkumar. Thanks for trying out coreos-assembler! This is probably not going to work out of the box right now because we haven't done any enablement work to get it to happen yet. One example would be the container runtime is pulling an x86_64 container from quay.io (we haven't built it for non x86 architectures yet). In order to get this to work you'll need to build the container locally on ppc64le hardware first and then use the container you just built as the assembler container instead of You'll probably hit other issues along the way, but let's document them and we can try to get it working. |
|
I cloned this project and ran the coreos-assembler script directly and running into the following pre-requisites: /home/cloud-user//coreos-assembler/coreos-assembler init https://github.com/coreos/fedora-coreos-config @dustymabe : Could you provide me instructions on building a ppc64le container? |
|
Should I be building on RHEL, or somewhere else (CentOS etc.)? |
the tool primarily expects to be run from within a container. So you'd need to clone this repo and then do something like Does that make sense?
I run on Fedora usually, but it shouldn't matter that much I don't think because the environment is in the container. |
|
docker build fails at this stage: =False
++ pwd
|
That's typically a network intermittent failure. If you run it multiple times does it continue to die in the same place? cc @sinnykumari who might be able to try this on some ppc64le hardware she has. |
|
I tried this on several places, where I kept seeing similar errors. Finally I hit a server where I was able to go further: Installed: Complete!
|
hmm. this seems odd. Are you trying to build an f28 image? We switched to f29 some time ago.
There aren't any ppc64le builds for this for f28, there is for f29
@cgwalters can you enable ppc64le for your copr repo? https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/ |
actually. i was mistaken.. this is the container build itself. I have opened #195 for us to switch to f29 for the assembler container. |
|
For now, once colin has enabled ppc64le builds for his copr (wait to hear back from him here in this issue), you can switch to f29 in the Dockerfile and try again. |
|
Switching to f29 in the Dockerfile, hit the same issue: Complete!
|
|
This is the same issue as #30 (comment) which was for aarch64. Although, since COPR does support ppc64le I added that to the buildroots and did the "update revision" dance and started a new build: https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/build/820725/ Like I said in that ticket though...it may be simplest to drop rdgo from this container for now. |
|
now that colin has enabled ppc64le and the build succeeded, you can try it again now and see where it breaks this time :) |
|
Thanks @cgwalters @dustymabe . It now fails here: Package findutils-1:4.6.0-21.fc29.ppc64le is already installed.
|
|
The It's tricky since today c-a tends to rely on rpm-ostree built from git master. This all loops back to sadly Koji (and to a lesser degree COPR) "owning" the multi-arch hardware; makes it harder for other projects that want to do CI or builds differently to use it too. This is https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/reworking-fedora-releng.md#blend-upstream-testing-and-downstream-testing Anyways so...you could probably remove the bits at the top that enable fahc, or we could switch to COPR I guess, although that takes us back to manual integration... |
|
@cgwalters: Commenting out fahc does get the build much further. It now fails at: |
|
OK, finally I got the docker build to complete. I had to add the [root@rhel-ocpinfra1 coreos-assembler]# git diff
diff --git a/Dockerfile b/Dockerfile
index 0599f8c..be5c830 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,4 +1,4 @@
-FROM registry.fedoraproject.org/fedora:28
+FROM registry.fedoraproject.org/fedora:29
WORKDIR /root/containerbuild
# Only need a few of our scripts for the first few steps
@@ -8,6 +8,7 @@ RUN ./build.sh install_rpms
# Ok copy in the rest of them for the next few steps
COPY ./ /root/containerbuild/
+RUN git submodule update --init
RUN ./build.sh make_and_makeinstall
RUN ./build.sh configure_user
diff --git a/build.sh b/build.sh
index 64f0703..a0d0299 100755
--- a/build.sh
+++ b/build.sh
@@ -5,19 +5,6 @@ srcdir=$(pwd)
configure_yum_repos() {
- # Enable FAHC https://pagure.io/fedora-atomic-host-continuous
- # so we have ostree/rpm-ostree git master for our :latest
- # NOTE: The canonical copy of this code lives in rpm-ostree's CI:
- # https://github.com/projectatomic/rpm-ostree/blob/d2b0e42bfce972406ac69f8e2136c98f22b85fb2/ci/build.sh#L13
- # Please edit there first
- echo -e '[fahc]\nmetadata_expire=1m\nbaseurl=https://ci.centos.org/artifacts/sig-atomic/fahc/rdgo/build/\ngpgcheck=0\n' > /etc/yum.repo
- # Until we fix https://github.com/rpm-software-management/libdnf/pull/149
- excludes='exclude=ostree ostree-libs ostree-grub2 rpm-ostree'
- for repo in /etc/yum.repos.d/fedora*.repo; do
- cat ${repo} | (while read line; do if echo "$line" | grep -qE -e '^enabled=1'; then echo "${excludes}"; fi; echo $line; done) > ${r
- mv ${repo}.new ${repo}
- done
-
# enable `walters/buildtools-fedora` copr
# pulled from https://copr.fedorainfracloud.org/coprs/walters/buildtools-fedora/repo/fedora-28/walters-buildtools-fedora-fedora-28.r
cat > /etc/yum.repos.d/walters-buildtools-fedora-fedora-28.repo <<'EOF' |
|
Also looking at the logs I need to add something here for ppc64le?
|
|
hey @manojnkumar - I edited your earlier comment to make it easier to read. Here is a guide on markdown I've been using for github: https://guides.github.com/features/mastering-markdown/
I don't know if we've looked at building the mantle codebase on other architectures. @bgilbert, @arithx, can you comment on that? @manojnkumar, for now you can comment out the |
|
Thanks @dustymabe. I proceeded with the coreos-assembler init and build steps with my built container for ppc64le. The build steps fails with:
Any clues @cgwalters ? |
|
Did you set up an alias for |
|
What's your host system? Is it RHEL7? You need to enable user namespaces. See... Specifically |
|
Yes RHEL 7.5.
|
|
@manojnkumar Please try the It's quite safe in general, most particularly if you don't have any potentially hostile users with |
|
Doesn't seem to help @cgwalters. Have been trying to do the rest of the steps in the link you provided, and I get into a circular issue with the userns setting in /etc/sysconfig/docker and the lines in /etc/subuid
|
|
I'm not sure if ore and kola currently build on non-x86, but if not, the fixes should be minor. And yes, we'd need to add a kolet build for ppc64le. |
|
@cgwalters : When I try to pull a container built on another system, to a system that has userns enabled I see this error. I seem to be stuck, with now way to proceed to the build phase:
|
|
I haven't played much with userns in Docker...can you try turning it off? To clarify, are you using the upstream Docker or the one in RHEL Extras? |
I have tried this after cleaning up /srv/ and then was getting the error. Note that I am using rpm-ostree-2018.9-3.fc28.1 on ppc64le. If I switch back to rpm-ostree-2018.9-3.fc28.1 on x6_64, I get same error here as well. That's why I believe that this issue should disappear on ppc64le when we start using latest rpm-ostree (coreos/rpm-ostree#1657 looks to me as one of the related change) |
|
For the purposes of the exercise for now, I'd recommend just building rpm-ostree from source in the container. I can make a ppc64le scratch build in Koji if you'd like? Let me do that. |
|
Ahh heh, it needs a newer OSTree as well. Hmm unless... yes I can patch out that dependency. New build: https://koji.fedoraproject.org/koji/taskinfo?taskID=31022867 |
Thanks a lot! |
|
I have created repo with the latest rpm-ostree scratch build rpms which @jlebon built and is available at https://sinnykumari.fedorapeople.org/custom-build/rpm-ostree/f28/ . Will update it in future with more recent builds if needed (until we have some better solution) |
|
I'm elaborating on this point here mainly because I want to be able to link to this thread later as a rationale elsewhere. To build on/rephrase #194 (comment) a bit more: Loosely coupled components in general are good. I think it's worked out well to have rpm-ostree separate from libostree, which is in turn separate from e.g. grub, and on a higher level an operator/ansible/gnome-software or whatever. However: sometimes we really do want to - for a period of time - tightly couple selected components and iterate on them quickly together to solve a problem. This definitely has happened a lot with rpm-ostree/libostree, and the fact that rpm-ostree can (if we need to) depend on libostree git master in CI has been absolutely critical. On the flip side...for example, I think the libostree/grub border is too strong, and if we had the capability to e.g. fork grub for a time and tightly couple it with libostree it would have helped solve some problems. Same thing will happen with coreos-assembler + [orchestrating pipeline]. Ideally they are loosely coupled, but some times you just need to make coordinated changes to both. What I'm saying here is this isn't just about coreos-assembler + rpm-ostree - it's about how we think about software delivery, packaging, and the Koji model of "components can only use other's releases" is too limiting and also discourages people from hacking on higher level or lower level parts. |
In coreos/fedora-coreos-tracker#84 I'm advocating for builds of upstream dev branches, not just releases. I think we are agreeing with one another. Am I mistaken? |
|
Another issue found during coreos-asembler build during From PR coreos/ignition-dracut#25 , it seems install of qemu_fw_cfg module was made explicit. qemu_fw_cfg module is provided by kernel-core sub-package which is available for x86_64 but not for ppc64le . Is this module essential or we can make it on demand loading depending upon availability? |
It's currently needed for Ignition to read userdata from QEMU; see coreos/ignition#656. |
does that mean the functionality just doesn't exist on ppc64le ?
Might be worth investigating if SMBIOS OEM string would work on ppc64le |
May have to check with Fedora kernel folks. https://github.com/torvalds/linux/blob/master/drivers/firmware/qemu_fw_cfg.c#L9 talks about x86_64 and arm but mention nothing about Power.
|
|
Well, we don't really need
Note this logic was tweaked in coreos/ignition-dracut#28 to support the feature baked in. See also specifically coreos/ignition-dracut#28 (comment) re. not necessarily needing it.
Ahh that'd be neat. Otherwise I guess we could recommend config drives or |
|
It turns out that SMBIOS is only available on x86 and ARM.
At present, if we don't have it, there's no reasonable way to pass an Ignition config to a ppc64 QEMU VM (coreos/ignition#666). (It'd have to be done by netbooting the guest or modifying the disk image.) That would also affect production virtualization.
By policy, Ignition doesn't support config drives. |
|
I hit the same error last week trying to build |
This is cheap with qcow2 backing files (or reflinks if available). The main thing is to make it not an ergonomic hit, but I see no issue with unilaterally changing e.g. We may end up needing to ship a script to inject ignition configs which would be used by |
|
We can ship a wrapper script (as we do in Container Linux as a convenience) but that doesn't really help users who want to run VMs directly in existing systems, e.g. libvirt. In CL we've generally felt that the right approach is to introduce proper userdata support into the various virtualization systems, rather than forcing users to go through an extra mangling step between "download VM image" and "run VM in the usual way". In this instance, see coreos/ignition#656 (comment). |
|
@sinnykumari : How far do you get with building for ppc64le? And what changes did you have to make? I am still not getting any further with the changes I had staged. Are any of your changes worth pushing? |
arch-specific changes which can be pushed upstream has already been done:
Other than that, few changes are there which I did locally to see how far we can go with build on ppc64le, but these changes are not meant to get in officially:
After doing above said changes coreos-assembler build fails during virt-install, full logs is available here . Please note that my last attempt was around 1 week back and I didn't get time to try further(hoping to get back to it soon). |
|
@sinnykumari : I am also hitting build failures. I could not find your pastebin to compare. Also your rpm-ostree comment indicates you are trying f28, whereas the ignition comment indicates f29. I am having issues with both. |
The QEMU Ignition config is a very nice to have for development but not currently a targeted production path. Production would be bare metal (PXE HTTP fetch of Ignition config or mounting the And the other production path would be OpenStack, fetching Ignition over user data just like happens on x86_64. |
kola support should be a prerequisite for supporting an architecture. Running kola tests in OpenStack meets this requirement, but not in a very satisfactory way; individual developers may not have access to a ppc64le OpenStack cluster. |
|
This is fixed right? |
@jcajka might be able to say |
|
@jcajka Can you confirm? |
|
@cgwalters @sinnykumari I would say as running, init, fetch and build it should be working. I'm currently working on beyond that. Run PR #408 and started to look at oscontainer build atm. |
|
Should be long since fixed. |
This is the error I see when trying to run this on the ppc64le architecture.
The text was updated successfully, but these errors were encountered: