How to deal with CI

How to deal with CI

What the CI (Continuous Integration) does

The CI we run on the opam-repository checks that newly published packages and package updates can integrate correctly with the existing ecosystem of opam packages.

To this end, we run the following checks:

Lint checking

On every opam file in the PR, we check for an extensive list of linting errors.

Build and test

For each packages added or modified, we run the following tests:

Platform

An up to date list of our build matrix is available in Platforms Tested on the Opam Repository CI. The axes of matrix are as follows:

OCaml compilers

We check that the package can be installed will all "recent" OCaml compilers consistent with its package dependency constraints, as defined by Ocaml_version.recent

Operating systems

We build and test the package on most popular Linux distros, FreeBSD, MacOS, and have experimental support for Windows.

Architectures

We build and test the package on a variety of different architectures, including arm64, ppc64le, s390x, x86_32, and arm32

Opam versions

We build and test with all "live" versions of opam (e.g., 2.0, 2.1, 2.2, 2.3, dev), where "live" means the version is in development, or is still available in the official package repositories of supported distros.

Compiler variants

We build and test with different options enabled in the compilert; e.g., flambda, frame-pointer, bytecode-only, naked-pointers, etc.

Dependency lower bounds

We build and test the package using the oldest possible versions of its dependencies, as determined by the opam solver, and limited by the explicit constraints in the opam file. See How do i test lower-bounds locally?.

Reverse dependencies

Finally, we check that all of a package's "reverse dependencies" (i.e., the set of packages that depend on the updated or newly published package) can build and pass tests.

Failure indicates either an unintended breaking API change or the need to add upper bounds to the dependencies. See Revdeps.

Finding the CI results

The CI results can be seen at the bottom of each PR by clicking to the opam-ci check or from a commit by clicking on the status icon (:x: or :heavy_check_mark:) and then selecting Details on the opam-ci check.

Interpreting the CI results

Each bullet point is a different log that you can read. The error(s) encountered are usually around the end of the log.

Reproducing CI builds locally

Reproduction instructions follow the line To reproduce locally: at the beginning of the logs. These commands require docker for Linux.

Note: for FreeBSD and macOS the Dockerfile is only an approximation as those platform can't run as a guest on Docker.

Dealing with CI errors

When adding new packages or updating packages, ideally the goal should be to have everything succeed.

Known errors that can be ignored

However there are some known errors that we currently ignore:

• missing system binaries from a distribution or missing system configuration. Using x-ci-accept-failures (see annex for more details) can help ignore those failures if the targetted platforms require something that can be dealt with manually but not in CI. Using the available field can also help if a certain platform will never work (e.g. uses x86_64 assembly, …).

• (failed: Sorry, resolution of the request timed out.) errors on older opam versions (2.0, 2.1, and 2.2). See https://github.com/ocurrent/opam-repo-ci/issues/354

• When modifying older packages, errors that existed before the PR can be ignored if the cause of the error stays the same.

• When conf-* packages are added or updated, if the required system dependencies cannot be installed from distro package managers, then the CI for the affected platforms will fail. However, if it is still possible for a user to install the package by other means (e.g., via a non-standard package manager, or by building from source) we still want to allow the package to be published. In those cases, we can try to use x-ci-accept-failure to reduce noise in the CI results and post-message to help inform users.

  If, however the package cannot be installed on a platform in principle then we should mark the package as having limited availability to the supported platforms, via the available field.

Unexpected errors

Unexpected errors can occur. Such errors are inherently unpredictible but a couple of examples can be:

• temporary network failures or broken build machine. If this happens, please open a ticket at ocaml/infrastructure
• packages that were broken before and went unnoticed. Contributions are most welcome for this type of errors.

opam-repository CI results vs. your development CI results

If your development repository has CI enabled, you are probably only checking that your software can build and run its test on a few select platforms. The matrix of platforms that we are testing on for the opam-repository CI is generally much larger and the focus of the tests narrower: we are checking to ensure that the package can be installed with opam, on the most common supported platforms and combinations of the tool essential toll chain.

If you're noticing errors that look genuine but never noticed before on your own CI it could be because:

• your CI doesn't test the oldest versions for the dependencies (aka. lower-bounds) of your packages (check the FAQ to know how to check this locally)
• your CI doesn't test building only one packages at a time. This is common for dune users who have several packages in one repository. The typical dune workflow is to use dune build. However testing dune build -p usually reveals issues in the assignment of dune rules (by default rules belong to every packages unless (package ...) is given). Typical error will occur in tests where a test will be ran in all packages but dependencies become cycles.
• we're testing more platforms (freebsd, macos, s390x, …). If you do not have access to that platform or access to a service providing this platform, don't hesitate to ask the opam-repository maintainers for help.
• we have a different opam setup: opam-repo-ci has the sandbox unable for both build and tests, which might not be the case in your own CI (e.g. if you’re using docker). Errors are typical here if you’re using the network (e.g. during the tests)

If you have a feature request for the CI, feel free to open a ticket or a PR at ocurrent/opam-repo-ci

---

Annex

• x-ci-accept-failures is a custom extra field used in opam-repository. The syntax for this field is the following: x-ci-accept-failures: [ "<platform>" ... ], where <platform> is a certain platform tested by opam-repo-ci. The list of platforms is found under "Operating Systems" in the generated documentation for tested platforms. You can also find a platform you want to ignore from the logs of a CI job, by looking for lines similar to if opam show -f x-ci-accept-failures: "$pkg" | grep -qF "\"opensuse-tumbleweed\"". In this case, opensuse-tumbleweed is the platform.
• sometimes one needs to see more verbose logs of what is going on. This can be partially achieved adding flags: verbose to the package (note that this line should be removed before merging it!)