Some integration tests are running in the Nordix infrastructure. Nordix provides a Jenkins instance and cloud resources on CityCloud for the Airship project. We use those resources to run integration tests for Metal3.
In Github the Jenkins bot uses metal3-jenkins username. It will post comments on Pull Requests in the metal3-dev-env, baremetal-operator and cluster-api-provider-metal3 repositories.
All members of the metal3-io organization that set their membership to be publicly visible will get admin rights on the CI jobs. This means :
- They can start the jobs on their PR directly
- They can start the jobs for PR of authors that are not in the organization
- They can add authors to whitelist so that the authors can start jobs on any further PR on their own, by commenting add to whitelist on the PR
We have multiple jobs that run some integration tests. The jobs can be triggered on PR from metal3-dev-env, baremetal-operator and cluster-api-provider-metal3 repositories by commenting the commands below. The job result will be posted as a comment.
- /test-integration run integration tests for V1alpha3 on Ubuntu
- /test-centos-integration run integration tests for V1alpha3 on CentOS
- /test-v1a4-integration run integration tests for V1alpha4 on Ubuntu
- /test-v1a4-centos-integration run integration tests for V1alpha4 on CentOS
It is also possible to prevent any job run by adding /skip-test in the PR description.
If the author is not in the whitelist but should be trusted then by adding a comment add to whitelist on the PR, the author will then be able to run the jobs on its own.
There is a Jenkins master job that every 6 hours cleans up all the leftover VMs from CityCloud which fail to be deleted at the end of v1alphaX integration test.
For all the PRs from authors that are not whitelisted, the bot will add a comment "Can one of the admins verify this patch?". This means that the author is not in the whitelist and that someone from the metal3-io organization should review the PR and run the tests (with /test-integration) or add the author to whitelist if trusted.
The jenkins configuration is stored in two places. The Nordix gerrit instance contains the jenkins job configuration and the Github metal3-io/project-infra repository contains the jobs pipeline.
The job configuration is stored here. Please announce if some reviews are needed on #cluster-api-baremetal in Kubernetes slack or send an email to estjorvas [at] est.tech mailing list.
The pipeline for the jobs is in the jobs folder. The scripts running
the tests are in the scripts folder.
We use pre-baked images to run the tests. The images are based on Ubuntu or CentOS.
The image building scripts can be found here for Ubuntu.
We collect all the container logs at the end of each Jenkins job run and archive them so that they can be later used for debugging purposes. You can find the archived logs under the "Build artifacts" section of the job. Please note that, depending on which one happens first, logs will be wiped out after either 30 days of its life or 100 job runs.
In case of issues or question on the Jenkins CI, please contact the maintainers by email to estjorvas [at] est.tech or by posting your message on the #cluster-api-baremetal channel on Kubernetes Slack.