|
| 1 | +# Singularity Builder GitHub CI |
| 2 | + |
| 3 | + |
| 4 | + |
| 5 | +This is a simple example of how you can achieve: |
| 6 | + |
| 7 | + - version control of your recipes |
| 8 | + - versioning to include image hash *and* commit id |
| 9 | + - build of associated container and |
| 10 | + - (optional) push to a storage endpoint |
| 11 | + |
| 12 | +for a reproducible build workflow. This recipe on master is intended to build |
| 13 | +Singularity 3.x (with GoLang). |
| 14 | + |
| 15 | +**Why should this be managed via Github?** |
| 16 | + |
| 17 | +Github, by way of easy integration with **native** continuous integration, is an easy way |
| 18 | +to have a workflow set up where multiple people can collaborate on a container recipe, |
| 19 | +the recipe can be tested (with whatever testing you need), discussed in pull requests, |
| 20 | +and tested on merge to master. If you add additional steps in the [build workflow](.github/workflows/go.yml) |
| 21 | +you can also use [Singularity Registry Client](http://singularityhub.github.io/sregistry-cli) to push your container to a |
| 22 | +[Singularity Registry Server](https://singularityhub.github.io/sregistry) or other |
| 23 | +cloud storage. |
| 24 | + |
| 25 | +**Why should I use this instead of a service?** |
| 26 | + |
| 27 | +You could use a remote builder, but if you do the build in a continuous integration |
| 28 | +service you get complete control over it. This means everything from the version of |
| 29 | +Singularity to use, to the tests that you run for your container. You have a lot more |
| 30 | +freedom in the rate of building, and organization of your repository, because it's you |
| 31 | +that writes the configuration. |
| 32 | + |
| 33 | +## Quick Start |
| 34 | + |
| 35 | +### 1. Add Your Recipes |
| 36 | + |
| 37 | +Add your Singularity recipes to this repository, and edit the [build workflow](.github/workflows/go.yml) |
| 38 | +section where the container is built. The default will look for a recipe file called |
| 39 | +"Singularity" in the base of the respository, [as we have here](Singularity). |
| 40 | +For example, here is the default: |
| 41 | + |
| 42 | +```yaml |
| 43 | + - name: Build Container |
| 44 | + env: |
| 45 | + SINGULARITY_RECIPE: Singularity |
| 46 | + OUTPUT_CONTAINER: container.sif |
| 47 | + run: | |
| 48 | + ls |
| 49 | + if [ -f "${SINGULARITY_RECIPE}" ]; then |
| 50 | + sudo -E singularity build ${OUTPUT_CONTAINER} ${SINGULARITY_RECIPE} |
| 51 | + else |
| 52 | + echo "${SINGULARITY_RECIPE} is not found." |
| 53 | + echo "Present working directory: $PWD" |
| 54 | + ls |
| 55 | + fi |
| 56 | +``` |
| 57 | +
|
| 58 | +And I could easily change that to build as many recipes as I like, and |
| 59 | +even disregard the environment variable. |
| 60 | +
|
| 61 | +```yaml |
| 62 | + - name: Build Container |
| 63 | + run: | |
| 64 | + sudo -E singularity build smokey.sif Singularity.smokey |
| 65 | + sudo -E singularity build toasty.sif marshmallow/Singularity.toasty |
| 66 | +``` |
| 67 | +
|
| 68 | +### 2. Test your Container |
| 69 | +
|
| 70 | +Importantly, then you should test your container! Whether that's running it, |
| 71 | +exec'ing a custom command, or invoking the test command, there is more than |
| 72 | +one way to eat a reeses: |
| 73 | +
|
| 74 | +```yaml |
| 75 | + - name: Test Container |
| 76 | + run: | |
| 77 | + singularity exec smokey.sif python run_tests.py |
| 78 | + singularity test smokey.sif |
| 79 | + singularity run toasty.sif |
| 80 | +``` |
| 81 | +
|
| 82 | +### 3. Push to a registry |
| 83 | +
|
| 84 | +You might be done there. But if not, you can install [Singularity Registry Client](http://singularityhub.github.io/sregistry-cli) and push to your cloud storage of choice! You will want to add python and python-dev to the dependency |
| 85 | +install: |
| 86 | +
|
| 87 | +```yaml |
| 88 | + - name: Install Dependencies |
| 89 | + run: | |
| 90 | + sudo apt-get update && sudo apt-get install -y \ |
| 91 | + build-essential \ |
| 92 | + libssl-dev \ |
| 93 | + uuid-dev \ |
| 94 | + libgpgme11-dev \ |
| 95 | + squashfs-tools \ |
| 96 | + libseccomp-dev \ |
| 97 | + pkg-config \ |
| 98 | + python-dev python python3-pip |
| 99 | +``` |
| 100 | +
|
| 101 | +And then install and use sregistry client. Here are many examples: |
| 102 | +
|
| 103 | +```yaml |
| 104 | + - name: Deploy Container |
| 105 | + run: | |
| 106 | + sudo pip3 install sregistry |
| 107 | + SREGISTRY_CLIENT=google-storage sregistry push --name username/reponame smokey.sif |
| 108 | + SREGISTRY_CLIENT=s3 sregistry push --name username/reponame smokey.sif |
| 109 | + SREGISTRY_CLIENT=registry sregistry push --name username/reponame smokey.sif |
| 110 | + SREGISTRY_CLIENT=dropbox sregistry push --name username/reponame smokey.sif |
| 111 | +``` |
| 112 | +
|
| 113 | +See the [clients page](https://singularityhub.github.io/sregistry-cli/clients) for all the options. |
| 114 | +Remember that the example workflow is intended to run on push to master, so you might want to have |
| 115 | +a similar one (without deployment) that runs on pull_request, or other events. |
| 116 | +See [here](https://help.github.com/en/articles/about-github-actions#core-concepts-for-github-actions) |
| 117 | +for getting started with GitHub actions, and [please open an issue](https://www.github.com/singularityhub/github-ci/issues) |
| 118 | +if you need any help. |
| 119 | +
|
| 120 | +
|
| 121 | +## Other Options |
| 122 | +
|
| 123 | +You can customize this base recipe in so many ways! For example: |
| 124 | +
|
| 125 | + - If you are building a Docker container, you can start with the docker base, build the container, and then pull it down into Singularity and test it. Successful builds can be pushed to Docker Hub, and then you know they will pull okay to a Singularity container. |
| 126 | + - The action can be configured with a Matrix to run builds on multiple platforms. |
| 127 | + - You can also do the same, but test multiple versions of Singularity. |
| 128 | +
|
| 129 | +Have fun! |
0 commit comments