Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build support for arm64-linux #1055

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

kaboing
Copy link

@kaboing kaboing commented Dec 23, 2024

Adds support for arm64-linux.

Uses buildx to build multi-platform image.

Spawned by: #611

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

What this PR does / Why we need it:
Adds support for arm64/linux. Builds multi-platform using buildx.

Which issue(s) this PR fixes:

Closes #611

Special notes for your reviewer:

Adds support for arm64-linux.

Uses buildx to build multi-platform image.

Spawned by: googleforgames#611
@kaboing
Copy link
Author

kaboing commented Jan 2, 2025

/gcbrun

1 similar comment
@markmandel
Copy link
Contributor

/gcbrun

@markmandel
Copy link
Contributor

Was make build working for you locally?

@kaboing
Copy link
Author

kaboing commented Jan 7, 2025

Yeah, it works on my machine ™️ . I build on an apple m2 pro with docker-desktop.

I noticed that the error messages complains about building ring, but I don't know if that is relevant.

Step #5 - "build": Caused by:
Step #5 - "build":   process didn't exit successfully: `/workspace/target/build-image/lto/build/ring-f1972dd2a537ad30/build-script-build` (exit status: 1)
Step #5 - "build":   --- stdout
Step #5 - "build":   cargo:rerun-if-env-changed=RING_PREGENERATE_ASM
Step #5 - "build":   cargo:rustc-env=RING_CORE_PREFIX=ring_core_0_17_8_
Step #5 - "build":   OPT_LEVEL = Some(3)
Step #5 - "build":   OUT_DIR = Some(/workspace/target/build-image/aarch64-unknown-linux-gnu/lto/build/ring-02bad55d8023c288/out)
Step #5 - "build":   TARGET = Some(aarch64-unknown-linux-gnu)
Step #5 - "build":   HOST = Some(x86_64-unknown-linux-gnu)
Step #5 - "build":   cargo:rerun-if-env-changed=CC_aarch64-unknown-linux-gnu
Step #5 - "build":   CC_aarch64-unknown-linux-gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=CC_aarch64_unknown_linux_gnu
Step #5 - "build":   CC_aarch64_unknown_linux_gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=TARGET_CC
Step #5 - "build":   TARGET_CC = None
Step #5 - "build":   cargo:rerun-if-env-changed=CC
Step #5 - "build":   CC = None
Step #5 - "build":   cargo:rerun-if-env-changed=CROSS_COMPILE
Step #5 - "build":   CROSS_COMPILE = None
Step #5 - "build":   RUSTC_LINKER = Some(/usr/bin/aarch64-linux-gnu-gcc)
Step #5 - "build":   cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
Step #5 - "build":   cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `/usr/bin/aarch64-linux-gnu-gcc` installed?
Step #5 - "build":   RUSTC_WRAPPER = None
Step #5 - "build":   cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
Step #5 - "build":   CRATE_CC_NO_DEFAULTS = None
Step #5 - "build":   DEBUG = Some(false)
Step #5 - "build":   CARGO_CFG_TARGET_FEATURE = Some(aes,neon)
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS_aarch64-unknown-linux-gnu
Step #5 - "build":   CFLAGS_aarch64-unknown-linux-gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS_aarch64_unknown_linux_gnu
Step #5 - "build":   CFLAGS_aarch64_unknown_linux_gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=TARGET_CFLAGS
Step #5 - "build":   TARGET_CFLAGS = None
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS
Step #5 - "build":   CFLAGS = None
Step #5 - "build":   cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `/usr/bin/aarch64-linux-gnu-gcc` installed?
Step #5 - "build": 
Step #5 - "build":   --- stderr
Step #5 - "build": 
Step #5 - "build": 
Step #5 - "build":   error occurred: Failed to find tool. Is `aarch64-linux-gnu-gcc` installed?

@kaboing
Copy link
Author

kaboing commented Jan 7, 2025

/gcbrun

@kaboing
Copy link
Author

kaboing commented Jan 7, 2025

Added setting CC explicitly, as suggested by briansmith/ring#2127.
Seems I need someone to do a /gcbrun for the PR

@kaboing
Copy link
Author

kaboing commented Jan 7, 2025

I'm new to the Rust eco-system so I'm mainly searching the internet to find people who's had similar issues.

@XAMPPRocky
Copy link
Collaborator

/gcbrun

@markmandel
Copy link
Contributor

markmandel commented Jan 7, 2025

Yeah, it works on my machine ™️ . I build on an apple m2 pro with docker-desktop.

Ah fun, so you would be building on ARM, and Cloud Build is building on Intel.

My suggestion would be to add the package g++-aarch64-linux-gnu to

RUN set -eux && \
apt-get update && \
apt-get install -y lsb-release jq curl wget zip git build-essential software-properties-common protobuf-compiler \
libssl-dev pkg-config nodejs npm bash-completion g++-x86-64-linux-gnu g++-mingw-w64-x86-64 && \

Which should give the system what it needs, when it says:

Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is aarch64-linux-gnu-gcc installed?

@kaboing
Copy link
Author

kaboing commented Jan 7, 2025

Gotcha. Let's test this version then!

@markmandel
Copy link
Contributor

/gcbrun

@XAMPPRocky
Copy link
Collaborator

XAMPPRocky commented Jan 8, 2025

ERROR: multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

@kaboing
Copy link
Author

kaboing commented Jan 8, 2025

Updated PR to use docker-container as build driver. However, according to docs,

Unlike docker driver, built images will not automatically appear in docker images and build --load needs to be used to achieve that.

Docs for build --load

Shorthand for --output=type=docker. Will automatically load the single-platform build result to docker images.

=> does not seem to work for multi-arch images. So we perhaps need to restructure a bit to push directly, or configure some other reasonable output. Any thoughts on this?

@markmandel
Copy link
Contributor

does not seem to work for multi-arch images

In my experience the multi arch image is stored temporarily in the build cache, so a subsequent build with --push will be almost immediate, because it won't rebuild.

@markmandel
Copy link
Contributor

/gcbrun

@markmandel
Copy link
Contributor

Well it built! But the image could not be found after the fact.

@kaboing
Copy link
Author

kaboing commented Jan 8, 2025

Well, it's progress at least! And I see in the logs that it build for both architectures, so that's nice.
When trying --load locally (latest commit), the entire thing hanged on sending the resulting tarball, but let's at least try it in the gcb environment and see what happens.

@XAMPPRocky
Copy link
Collaborator

/gcbrun

Restructured test-agones to require build-image directly since it's already doing the pushing.
@XAMPPRocky
Copy link
Collaborator

/gcbrun

@XAMPPRocky
Copy link
Collaborator

So it successfully built, but the image uploaded to CI doesn't look multi arch to me.

docker buildx imagetools inspect us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83

Name:      us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:5a56ba3b94663f46b4e0aa673735c662ad474c21dcb6a1ad908000f75b474985

@kaboing
Copy link
Author

kaboing commented Jan 9, 2025

No, I concur. One of the other manifests uploaded to the registry at the same time do have different entries for arm64 and amd64:
https://console.cloud.google.com/artifacts/docker/quilkin/us/ci/quilkin/sha256:d9d655c2d817e95d85b56286cee74a633cb1efb6d32650fdcb145702454485aa;tab=manifest?inv=1&invt=AbmXyg

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:3a22f7b558eb554c10c351db637d438803472ac59a6af5e5fd6c4ba357c3df3c",
         "size": 3895,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:4f2a123d6da1396fd987f71ae63f7597c180c76bde2c78199aec5d017f27cff1",
         "size": 3895,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      ...
}

I'm trying to figure out what more needs to be done.

Just to make sure, I've tried to run the image created on arm64 and it does not work.

@kaboing
Copy link
Author

kaboing commented Jan 9, 2025

It does work to run on arm64 if I explicitly use a different manifest-file, the one I referred to above. So need to figure out why the manifests get messed up.

... 
containers:
        - name: quilkin-arm64
          image: us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83@sha256:d9d655c2d817e95d85b56286cee74a633cb1efb6d32650fdcb145702454485aa
          args:
            ["proxy", "--management-server", "http://quilkin-manage-agones:80"]
...

@XAMPPRocky
Copy link
Collaborator

/gcbrun

@quilkin-bot
Copy link
Collaborator

Build Succeeded 🥳

Build Id: 6b30e4ca-821b-44fe-8f62-d0c7b11d51c9

The following development images have been built, and will exist for the next 30 days:

To build this version:

git fetch [email protected]:googleforgames/quilkin.git pull/1055/head:pr_1055 && git checkout pr_1055
cargo build

@XAMPPRocky
Copy link
Collaborator

Seeing the same result unfortunately

docker buildx imagetools inspect us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-09bbe46
Name:      us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-09bbe46
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:cf0e208da1ecd98500284826b1cfc20a8b04e3d8f857dc36d3d7a13073c550aa

@kaboing
Copy link
Author

kaboing commented Jan 17, 2025

Thanks for running the gcbruns for me on this PR, I'll investigate further where the problem is.

@kaboing
Copy link
Author

kaboing commented Jan 17, 2025

This digest should have the tag, it is the manifest list that points to the other manifests. Weird.
CleanShot 2025-01-17 at 16 14 59@2x

CleanShot 2025-01-17 at 16 16 43@2x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request size/s
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Publish arm64 binaries and images
4 participants