Skip to content

Docker multiplatform build support#1974

Merged
a3957273 merged 6 commits intogchq:masterfrom
PathToLife:feature/docker-multiplatform-build
Apr 5, 2025
Merged

Docker multiplatform build support#1974
a3957273 merged 6 commits intogchq:masterfrom
PathToLife:feature/docker-multiplatform-build

Conversation

@PathToLife
Copy link
Copy Markdown
Contributor

Adds multiplatform docker support for arm64 and armv7 architectures. Power efficient.. hopefully🔋🌲

Github workflow will need a test. However working locally with docker buildx and test image has been published below.

pathtolife/cyberchef:v10.19.4

image

https://hub.docker.com/r/pathtolife/cyberchef/tags

Command to run and remove on stop (--rm)

docker run --rm -p 8080:80 pathtolife/cyberchef:v10.19.4

image

Thanks!

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Feb 13, 2025

CLA assistant check
All committers have signed the CLA.

@PathToLife PathToLife force-pushed the feature/docker-multiplatform-build branch from 050c1c3 to f8b613b Compare February 13, 2025 07:58
@a3957273
Copy link
Copy Markdown
Member

a3957273 commented Feb 15, 2025

This was the initial implementation of the build job, but ended up taking very long. Over an hour to build an ARM build on the AMD build workers. Is there any chance you could test on a local repository how long the GitHub runners take to build the single image before this PR, and how long it would take to build all three?

This does have the benefit of only building on releases, which I'm less worried about having long build times. However, would still enjoy specific timings before merging this in.

@PathToLife
Copy link
Copy Markdown
Contributor Author

Over an hour to build an ARM build on the AMD build workers. Is there any chance you could test on a local repository how long the GitHub runners take to build the single image before this PR, and how long it would take to build all three?

Will report back. I'll add the time consuming build step only runs on one platform (amd64)! (so we won't be building on arm).

The multiplatform build step will be taking the amd64 build files and putting them in crossplatform supported nginx containers.

image

@a3957273
Copy link
Copy Markdown
Member

Oh, that's actually a really cool solution. Never mind then, the previous implementation also built on every other platform, which was incredibly slow. This seems far better. Don't worry about timing it, I'll be happy to test it out when I get time.

@PathToLife
Copy link
Copy Markdown
Contributor Author

PathToLife commented Feb 25, 2025

I had triggered a release last week. However it seems the CI OS image ubuntu-latest has updated the version of chrome. Therefore the UI Tests step fails, reporting the chrome versions are incompatiable with the npm driver.

You might have similar issues on the next release.

https://github.com/PathToLife/CyberChef/actions/runs/13350228747/job/37285773548

value: {
       error: 'session not created',
       message: 'session not created: This version of ChromeDriver only supports Chrome version 130\n' +
         'Current browser version is 133.0.6943.53 with binary path /opt/google/chrome/chrome',
       stacktrace: ''
     }

image

@a3957273
Copy link
Copy Markdown
Member

a3957273 commented Mar 9, 2025

I was hoping this PR had fixed this issue, which made it into mainline around a month ago:

https://github.com/gchq/CyberChef/pull/1972/files

@PathToLife
Copy link
Copy Markdown
Contributor Author

I was hoping this PR had fixed this issue, which made it into mainline around a month ago:

https://github.com/gchq/CyberChef/pull/1972/files

Looks like releases.yml is missing that change. I'll apply it and give it a go.

- name: Install
run: |
npm ci
npm run setheapsize

@PathToLife PathToLife force-pushed the feature/docker-multiplatform-build branch from 22007ef to b85036b Compare March 10, 2025 04:28
@PathToLife
Copy link
Copy Markdown
Contributor Author

Is there a reason we are using buildah? I'd recommend using docker buildx in the CI step as it's more simple. If we want to stick with buildah, it might be better to manually generate tags per arch, and then merge them together to a docker pullable manifest.

Manifest comparison redhat-actions/buildah-build vs docker buildx

// ghcr.io/pathtolife/cyberchef:10.19.5-beta13 is built using Github CI buildah image
// note the arm64v8 tag
$> docker buildx imagetools inspect ghcr.io/pathtolife/cyberchef:10.19.5-beta13
  Name:      ghcr.io/pathtolife/cyberchef:10.19.5-beta13
  MediaType: application/vnd.oci.image.index.v1+json
  Digest:    sha256:23d696642893987c93c0a14559130674b47b5f708870580403f2911710c06c84
  
  Manifests:
    Name:      ghcr.io/pathtolife/cyberchef:10.19.5-beta13@sha256:88f1cf2883e07355d234693114eb7b06e2238d59282330ba850451b040d29c0c
    MediaType: application/vnd.oci.image.manifest.v1+json
    Platform:  linux/amd64
  
    Name:      ghcr.io/pathtolife/cyberchef:10.19.5-beta13@sha256:4fa3f62e9f8268b631ec7f458488958ffad712e0cef66c2f6e6f6a36795b015b
    MediaType: application/vnd.oci.image.manifest.v1+json
    Platform:  linux/arm64v8
    
// docker.io/pathtolife/cyberchef:v10.19.4 is built using buildx
$> docker buildx imagetools inspect pathtolife/cyberchef:v10.19.4
  Name:      docker.io/pathtolife/cyberchef:v10.19.4
  MediaType: application/vnd.oci.image.index.v1+json
  Digest:    sha256:0579538a0091c6501862c4ea071b311d09cdbb03f5fae431d7021178f6b2c804
  
  Manifests:
    Name:        docker.io/pathtolife/cyberchef:v10.19.4@sha256:dbf0a7ede085045bc2e89c8d38510734d475571d2697461d87eebc38bda5dd92
    MediaType:   application/vnd.oci.image.manifest.v1+json
    Platform:    linux/amd64
  
    Name:        docker.io/pathtolife/cyberchef:v10.19.4@sha256:039fe128f6661f0ee3a003c26b7c92f08771163cde21d3b3e6bfd80bdfcf4223
    MediaType:   application/vnd.oci.image.manifest.v1+json
    Platform:    linux/arm64
...(Truncated Output)

User docker pull experience

It's not ideal as standard pull on arm64 server fails

// Standard pull on arm64 server fails as it does not identify listed platform linux/arm64v8
$> docker pull ghcr.io/pathtolife/cyberchef:10.19.5-beta13
  10.19.5-beta13: Pulling from pathtolife/cyberchef
  no matching manifest for linux/arm64/v8 in the manifest list entries
  
// However a pull on same tag via sha256 works
$> ghcr.io/pathtolife/cyberchef:10.19.5-beta13@sha256:4fa3f62e9f8268b631ec7f458488958ffad712e0cef66c2f6e6f6a36795b015b

@a3957273
Copy link
Copy Markdown
Member

a3957273 commented Apr 5, 2025

No reason we're using buildah over docker. If you want to switch to make the user experience better, then that's more than okay with me! This PR seems like an improvement over the current status quo though (even if it does require pulling via the unwieldy sha hash), so I'll get this merged in now. Thanks so much for your contribution!

@a3957273 a3957273 merged commit 305a02c into gchq:master Apr 5, 2025
@PathToLife
Copy link
Copy Markdown
Contributor Author

PathToLife commented Apr 5, 2025

@a3957273 just noticed that NPM publish CI step is set to false as my fork didn’t have NPM release configured.

might want to verify NPM step is active by removing the ‘if’ statement.

Thanks for the reply about buildx - I’ll take a look later when I have the time :)

@0xh3xa
Copy link
Copy Markdown
Contributor

0xh3xa commented Apr 5, 2025

Fix #1912

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants