Skip to content

WIP (do not merge): GH actions multiarch #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

WIP (do not merge): GH actions multiarch #44

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d9c8117
Adding support for multiarch builds.
ndeet Nov 16, 2019
46dff41
Multiarch workflow for GH actions.
ndeet Nov 16, 2019
87f2f94
Change to upstream.
ndeet Nov 20, 2019
df9e7e6
Only trigger on tags.
ndeet Nov 20, 2019
bace0cd
Fix regex for arch replacement in Dockerfile.
ndeet Nov 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
146 changes: 146 additions & 0 deletions .github/workflows/on-tag.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
name: Build & deploy ElectrumX

on:
push:
tags:
- '*'

jobs:
build:
name: Build ElectrumX
runs-on: ubuntu-18.04

# NOTE: qemu v3.1.1 used instead of currently newest v4.1.0, because v4 is **much** slower for aarch64, see:
# https://github.com/meeDamian/docker-berkeleydb/commit/9e87d11314c2522726497f0c6059e61a31298e7f/checks
env:
QEMU_VERSION: v3.1.1
DOCKER_BUILDKIT: 1

strategy:
matrix:
arch:
- arm32v7
- arm64
- amd64

steps:
- uses: actions/[email protected]

- name: Setup environment
run: |
VERSION="$(echo "${GITHUB_REF}" | awk -F/ '{print $NF}' | tr -d v)"
echo ::set-env name=VERSION::"${VERSION}"
echo ::set-env name=DIR::"$(echo "${VERSION}" | cut -d. -f-2)"
- name: Register self-compiled qemu
if: matrix.arch != 'amd64'
run: docker run --rm --privileged "meedamian/simple-qemu:${QEMU_VERSION}-${{matrix.arch}}" -p yes

# Alter `Dockerfile` to reference used architecture/image combos explicitly. Places changed are:
# * all `FROM` statements (ex. `FROM alpine…` -> `FROM arm64v8/alpine…`)
# `sed` `--expression`s change it in the following way:
# Matches all occurrences of `FROM alpine`, and injects arch prefix before `alpine`, ex: `arm64v8/alpine`
- name: Change Dockerfile to use arch-specific base images
if: matrix.arch != 'amd64'
run: |
CPU="${{matrix.arch}}"
if [[ "${CPU}" == "arm64" ]]; then
CPU="arm64v8"
fi
sed -i Dockerfile \
-e "s|^FROM alpine|FROM $CPU/alpine|g"
Copy link

@meeDamian meeDamian Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line does nothing, and this line is essential to getting it to work. You use a different base for your image, and it doesn't match sed's pattern.

Copy link
Author

@ndeet ndeet Nov 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, thank you, very good catch, now as you mention it makes totally sense, so the docker file for each stays with the same alpine arch (amd64) like the original Dockerfile and therefore we built always the same arch (and it doeas also not fail because it is a valid arch) 🤦‍♂

I will check how those python specific arches are named and play with that regex to make it work, thank you!

Copy link

@meeDamian meeDamian Nov 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quite sure it's: {arm32v6,arm32v7,arm64v8,amd64}/python:3.7-alpine3.9

Copy link
Author

@ndeet ndeet Nov 26, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes found that out and adjusted regex to replace everything after FROM and prepend $CPU/ and it works for the arch. "s|^FROM \(.*\)|FROM $CPU/\1|g"

However, new problem, rocksdb-dev (leveldb-dev works though) is not available for arm arches on alpinelinux it seems, so it fails there now:
https://github.com/ndeet/docker-electrumx-workflow/runs/322018407#step:6:99

https://pkgs.alpinelinux.org/packages?name=rocksdb-dev&branch=edge

Next step, figure out how to build multi arch packages for Alpinelinux? 😬 https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package

I tried also install rocksdb-dev via pip3 on rpi4 on Debian Buster, while it finds the package it errors on building it, but OT here, just wanted to check if it is generally available elsewhere.

 rocksdb/_rocksdb.cpp:622:10: fatal error: rocksdb/slice.h: No such file or directory
     #include "rocksdb/slice.h"
              ^~~~~~~~~~~~~~~~~
    compilation terminated.
    error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

- name: Build ElectrumX
run: docker build . --build-arg "VERSION=${VERSION}" --tag electrumx
- name: Print OS info
run: docker run --rm --entrypoint=uname electrumx -a

- name: Print ElectrumX version
run: docker run --rm electrumx echo $SERVER_SUBVERSION

- name: Save built image into a .tgz file
run: |
mkdir -p images/
docker tag electrumx "electrumx:${{matrix.arch}}"
docker save "electrumx:${{matrix.arch}}" | gzip > "images/electrumx-${{matrix.arch}}.tgz"
- name: Print sha256sum of built image
run: sha256sum images/*

- name: Upload built image
uses: actions/[email protected]
with:
name: images
path: images/


docker-hub-push:
name: Tag & deploy to Docker Hub. Only after successful build and on a git-tag push

runs-on: ubuntu-18.04
needs: build
steps:
- uses: actions/[email protected]

- name: Setup environment
run: |
echo ::set-env name=DOCKER_USER::"${GITHUB_ACTOR,,}"
echo ::set-env name=SLUG::"$(echo ${GITHUB_REPOSITORY,,} | sed 's/docker-//')"
echo ::set-env name=VERSION::"$(echo "${GITHUB_REF}" | awk -F/ '{print $NF}')"
- name: Enable manifests
run: |
mkdir -p ~/.docker
echo '{ "experimental": "enabled" }' > ~/.docker/config.json
sudo systemctl restart docker
docker version
- name: Login to Docker Hub
run: |
echo "Logging in as ${DOCKER_USER}…"
echo "${{ secrets.DOCKER_TOKEN }}" | docker login -u="${DOCKER_USER}" --password-stdin
- name: Download all build artifacts
uses: actions/[email protected]
with:
name: images

- name: Print sha256sum of all images
run: sha256sum images/*

- name: Load images locally
run: ls images/ | xargs -I % docker load -i "images/%"

# No short tags. IMO electrumx version should always be specified exactly, and explicitly.
- name: Version-tag all images
run: docker images electrumx --format "{{.Tag}}" | xargs -I % docker tag "electrumx:%" "${SLUG}:${VERSION}-%"

- name: List all tagged images
run: docker images "${SLUG}"

- name: Push all images
run: docker images "${SLUG}" --format "{{.Repository}}:{{.Tag}}" | xargs -I % docker push %

- name: Create manifest
run: >
docker -D manifest create "${SLUG}:${VERSION}" \
"${SLUG}:${VERSION}-amd64" \
"${SLUG}:${VERSION}-arm64" \
"${SLUG}:${VERSION}-arm32v7"
- name: Annotate arm32v7
run: docker manifest annotate "${SLUG}:${VERSION}" "${SLUG}:${VERSION}-arm32v7" --os linux --arch arm --variant v7

- name: Annotate arm64v8
run: docker manifest annotate "${SLUG}:${VERSION}" "${SLUG}:${VERSION}-arm64" --os linux --arch arm64 --variant v8

- name: Push manifest
run: docker manifest push "${SLUG}:${VERSION}"

- name: Sync README.md and Description to Docker Hub
uses: meeDamian/[email protected]
with:
pass: ${{secrets.DOCKER_TOKEN}}
slug: lukechilds/electrumx
# currently hardcoded description as fetching from GH fails
description: Run an Electrum server with one command

- name: Upload images to Github Release
uses: meeDamian/[email protected]
with:
token: ${{secrets.GITHUB_TOKEN}}
gzip: false
files: images/*