ocurrent/ocaml-ci

A CI for OCaml projects

OCaml-CI

This is an OCurrent pipeline that provides CI for OCaml projects hosted on GitHub. It uses metadata from the project’s opam and dune files to work out what to build, and uses caching to make builds fast. It automatically tests projects against multiple OCaml versions and OS platforms.

The pipeline is defined in pipeline.ml. It:

1. Gets the list of installations of its GitHub app.
2. For each installation, gets the list of repositories to check.
3. For each repository, gets the branches and PRs to check.
4. For each target, it fetches the head commit, generates a Dockerfile and builds it.

The generated Dockerfile first adds all the *.opam files found in the project, then uses opam to install all the dependencies, then adds the rest of the source files. This means that rebuilds are often very fast, because Docker will reuse the previously cached build step as long as the opam files don't change.

To add the CI to your own project:

1. Go to https://github.com/apps/ocaml-ci and install the app for your GitHub user.
2. Configure just the repositories you want to test (start with one!). If you select All Repositories we won't build anything.
3. Ask us to add you to the alpha-testers list by submitting a PR against this repository adding yourself to deploy-data/github-organisations.txt.
4. Add a status badge from the OCaml-CI endpoint with:

   [![OCaml-CI Build Status](https://img.shields.io/endpoint?url=https://ocaml.ci.dev/badge/<user>/<repo>/<branch>&logo=ocaml)](https://ocaml.ci.dev/github/<user>/<repo>)
   

5. Report bugs :-)

Installation

Get the code with:

git clone --recursive https://github.com/ocurrent/ocaml-ci.git
cd ocaml-ci
opam install --deps-only ./ocaml-dockerfile ./ocluster ./ocurrent ./solver-service .

Note: you need to clone with --recursive because this project uses submodules (it depends on some packages that aren't released yet). If you forget, git submodule update --init --recursive will fetch them.

To test the CI on a local Git clone, you need to first install the dependencies and then use dune exec as shown below:

opam update
opam install . --deps-only --with-test --yes
dune exec -- ocaml-ci-local /path/to/project

This will build the project as the real CI would, but it only monitors the default branch and does not push the results anywhere. It runs a web interface at http://localhost:8080. This is useful if you want to try out changes to the pipeline.

If you want to build the whole system, the easiest way is using Docker:

docker build -f Dockerfile -t ocaml-ci-service .
docker build -f Dockerfile.gitlab -t ocaml-ci-gitlab .
docker build -f Dockerfile.web -t ocaml-ci-web .

The stack.yml contains the configuration used on the live system. You'll have to register your own GitHub app to be able to test the services locally.

If you want it to update to changes in opam-repository automatically you'll also need to register a webhook there sending push events to the CI's /webhooks/github path.

This document takes you through the steps necessary to deploy the Docker images.

Remote API

The service provides a Cap'n Proto endpoint and a command-line client that uses it. You will need to be given the ocaml-ci.cap file, which grants access to the API. The client can be built and run using dune exec -- ocaml-ci --ci-cap=ocaml-ci.cap ..., or installed as ocaml-ci.

To see the branches and PRs that ocaml-ci is monitoring in a repository:

$ ocaml-ci mirage/irmin
615364620f4233cb82a96144824eb6ad5d1104f0 refs/heads/1.4 (passed)
e0fcf0d336544650ca5237b356cfce4a48378245 refs/heads/master (passed)
6c46d1de5e67a3f504fc55af1d644d852c946533 refs/heads/mirage-dev (passed)
28421a152e8e19b3fb5048670629e7e01d0fbea6 refs/pull/523/head (passed)
acfbee7e82fcaaa5a0dad900068dc67f22021f2e refs/pull/678/head (passed)
3fc04e9f6e7574c0f61eacb3187b412b3bababe4 refs/pull/728/head (passed)
32f6c9f303616880994998881ee75c8d1fe0df91 refs/pull/771/head (passed)
b2d4b06f94d13384ae08eb06439ce9c6066419cd refs/pull/815/head (failed)
d8161e6cbf06c3005a080d4df209f7de67d6fa5c refs/pull/851/head (passed)
5e36237d7ce6279878578cf48d8b63937c499e5a refs/pull/858/head (failed)
04a368ecd52ea436bfcd252ed94772f55b5159d5 refs/pull/866/head (passed)
2e838b491a4c0b21750f7a2e6dee88eee1c7d94e refs/pull/867/head (passed)

You can pass either the reference (e.g. refs/heads/master) or the commit hash to choose one of them.

$ ocaml-ci mirage/irmin refs/heads/main
alpine-3.10-ocaml-4.08

To view the log (following it if incomplete):

$ ocaml-ci mirage/irmin refs/heads/master alpine-3.10-ocaml-4.08 log
[...]
- Test Successful in 17.643s. 99 tests run.
-> compiled  irmin-unix.dev
-> installed irmin-unix.dev
Done.
# Run eval $(opam env) to update the current shell environment
Removing intermediate container 4c85cdc76ddc
 ---> c8e34c3b5eee
Successfully built c8e34c3b5eee
2019-09-25 14:55.57: Job succeeded

Instead of log, you can also use cancel, rebuild or status.

For convenience, you can omit the leading refs/ when specifying a reference, and for PRs you can omit the trailing /head. For commits, you must give at least the first 6 characters. e.g.

$ ocaml-ci mirage/irmin pull/867 alpine-3.10-ocaml-4.08 cancel

Deployment

ocaml-ci is deployed as three docker images built from Dockerfile, Dockerfile.gitlab and Dockerfile.web, with the live service following live-engine for the backend and live-www for the frontend. An ocurrent-deployer pipeline watches these branches, performing a docker build and deploy whenever it sees a new commit. The live branches should typically contain commits from master plus potentially short lived commits for testing changes that are later merged into master.

To deploy code changes either from master or a branch:

• check that you've rebased the changes onto master
• git push -u upstream HEAD:live-engine or
• git push -u upstream HEAD:live-www

To deploy changes to stack.yml run (assuming a docker context with sufficient access):

make deploy-stack

Opam repository updates

When it is updated opam-repository sends a webhook to Ocaml-ci triggering its pipelines. This mechanism allows builds to remain up to date with changes in the opam package ecosystem. For further details of this webhook, please contact a maintainer of opam-repository.

Local development

See this document for set up and running the server and web components locally.

ocaml-ci's Issues

monorepo: does not support compiler versions not in base images

Hi,

We're having the following situation in RWO:

• the version for ocaml and ocaml-base-compiler is set to 4.10.2 in the lock file
• in the analysis phase, it considers only 4.10 platforms and queries for (ocaml=4.10.2, opam-monorepo=0.1.0, dune=2.7.1)
• this fails because the platform has 4.10.1 installed:

[INFO] Can't find all required versions.
Selected: base-bigarray.base base-threads.base base-unix.base ocaml-config.2
          ocaml-base-compiler&opam-monorepo-deps ocaml-base-compiler
          ocaml-variants
- ocaml -> ocaml.4.10.1
    ocaml-base-compiler 4.10.1 requires = 4.10.1
- ocaml-base-compiler -> ocaml-base-compiler.4.10.1
    User requested = 4.10.1
- ocaml-variants -> (problem)
    Rejected candidates:
      ocaml-variants.4.13.0+trunk+no-flat-float-array: In same conflict class (ocaml-core-compiler) as ocaml-base-compiler
      ocaml-variants.4.13.0+trunk+nnp: In same conflict class (ocaml-core-compiler) as ocaml-base-compiler
      ocaml-variants.4.13.0+trunk+fp+flambda: In same conflict class (ocaml-core-compiler) as ocaml-base-compiler
      ocaml-variants.4.13.0+trunk+fp: In same conflict class (ocaml-core-compiler) as ocaml-base-compiler
      ocaml-variants.4.13.0+trunk+flambda: In same conflict class (ocaml-core-compiler) as ocaml-base-compiler
      ...
- opam-monorepo-deps -> (problem)
    Rejected candidates:
      opam-monorepo-deps.dev: Requires ocaml = 4.10.2

I'm not sure what to do in that case; it would be fine to create a new switch where dune and opam-monorepo would be installed.

A similar problem would appear if the project locks a compiler variant (e.g. the application needs flambda).

big duniverses exceed layer count in depext calculation

The COPY and opam pin strategy for depext calculation leads to a large number of layers in the resulting Dockerfile. There is a limit (127) of these in the engine, which leads to build failures.

I think we need a more efficient (but still cache-friendly) way to do the opam pin / depext dance that doesn't add a lot of layers. Some options:

• copy all the opam files into a single directory and then pin them all there. This would let it work in a single COPY command.
• we could create a new layer for all the opam pins, and subsequently roll that back.

I suspect the first option is best...

Don't cache the final build/test step

If you re-build a successful run, you probably want to try the tests again (to see if they're flaky). We should probably remove the last line from the generated Dockerfile and run it using docker run instead, so there's no caching.

A side-effect of this is that we'd run the build step twice for PRs (once in the upstream repository and once in the fork). However, the final step is usually fairly quick, and it would mean that you get to see the test output in both places.

Lint step specifes fixed OCaml version

Currently the linting stage uses a statically-configured Docker image with a fixed OCaml version (4.08). This is problematic for projects which require OCamlformat versions that are incompatible with 4.08 (such as mirage/wodan, which currently uses OCamlformat.0.9).

do not generate dockerfiles with ENV DEPS being empty

This results in an error in dockerfile parsing, e.g. from https://ci.ocamllabs.io:8100/job/2020-07-24/004716-ci-ocluster-build-f8ddc7

git clone --recursive "https://github.com/favonia/yuujinchou.git" && cd "yuujinchou" && git fetch origin "refs/pull/34/head" && git reset --hard 62d123e5
cat > Dockerfile <<'END-OF-DOCKERFILE'
FROM ocurrent/opam@sha256:84a5debe280b7aded6828f9be21bd0eb82d03c850faa6935ef762d4665e6ead1
# debian-10-ocaml-4.03
WORKDIR /src
RUN sudo chown opam /src
RUN cd ~/opam-repository && (git reset --hard 1a576beaaa094013d2935556a76bf48f89ae6098 || (git pull origin master && git reset --hard 1a576beaaa094013d2935556a76bf48f89ae6098)) && opam update -u
RUN mkdir -p "./"
COPY yuujinchou.opam ./
RUN opam pin add -yn yuujinchou.dev "./"
ENV DEPS 
RUN opam depext --update -y yuujinchou.dev $DEPS
RUN opam install $DEPS
COPY --chown=opam . /src/
RUN opam exec -- dune build @install @runtest && rm -rf _build
END-OF-DOCKERFILE
docker build .

2020-07-24 00:47.16: Using cache hint "favonia/yuujinchou-ocurrent/opam@sha256:84a5debe280b7aded6828f9be21bd0eb82d03c850faa6935ef762d4665e6ead1-debian-10-ocaml-4.03-b7c2a91365c1f6898231aa5a71eac69a"
2020-07-24 00:47.16: Waiting for resource in pool OCluster
2020-07-24 00:47.16: Waiting for worker...
2020-07-24 00:47.16: Got resource from pool OCluster
Using BuildKit Dockerfile:
# syntax = docker/dockerfile:experimental@sha256:ee85655c57140bd20a5ebc3bb802e7410ee9ac47ca92b193ed0ab17485024fe5
FROM ocurrent/opam@sha256:84a5debe280b7aded6828f9be21bd0eb82d03c850faa6935ef762d4665e6ead1
# debian-10-ocaml-4.03
WORKDIR /src
RUN sudo chown opam /src
RUN cd ~/opam-repository && (git reset --hard 1a576beaaa094013d2935556a76bf48f89ae6098 || (git pull origin master && git reset --hard 1a576beaaa094013d2935556a76bf48f89ae6098)) && opam update -u
RUN mkdir -p "./"
COPY yuujinchou.opam ./
RUN opam pin add -yn yuujinchou.dev "./"
ENV DEPS 
RUN --mount=type=cache,target=/home/opam/.opam/download-cache,uid=1000 opam depext --update -y yuujinchou.dev $DEPS
RUN --mount=type=cache,target=/home/opam/.opam/download-cache,uid=1000 opam install $DEPS
COPY --chown=opam . /src/
RUN opam exec -- dune build @install @runtest && rm -rf _build
Building on x86-build-5.ocamllabs.io
All commits already cached
HEAD is now at 62d123e Add the efficient Trie.to_seq_values.
#2 [internal] load .dockerignore
#2 sha256:014d103dfe70ba646e0c923b243b14440f2a099d89a55012ddc9dc66266508d7
#2 transferring context: 0.0s
#2 transferring context: 2B 0.0s done
#2 ...

#1 [internal] load build definition from Dockerfile
#1 sha256:3e3bf8d7e84687460518e505107ec466b05eca59241bcef40fb7c66a2b54bca0
#1 transferring dockerfile: 887B 0.0s done
#1 DONE 0.4s

#2 [internal] load .dockerignore
#2 sha256:014d103dfe70ba646e0c923b243b14440f2a099d89a55012ddc9dc66266508d7
#2 DONE 0.5s

#4 docker-image://docker.io/docker/dockerfile:experimental@sha256:ee85655c57140bd20a5ebc3bb802e7410ee9ac47ca92b193ed0ab17485024fe5
#4 sha256:71a594c684f98692da7baadc89226dfbe7a87a365660bd1251c8db6c1d7f4620
#4 CACHED

#3 resolve image config for docker.io/docker/dockerfile:experimental@sha256:ee85655c57140bd20a5ebc3bb802e7410ee9ac47ca92b193ed0ab17485024fe5
#3 sha256:8db1d80318d235996b816ce713bcf8909e6fb35d7d54a0c31710b9c3b442271a
#3 DONE 0.0s
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = ENV must have two arguments
docker-build failed with exit-code 1
2020-07-24 00:47.34: Job failed: Failed: Build failed

Enable `lint` stage even if `.ocamlformat` is missing

For some projects, we might want to use dune formatting feature without ocamlformat, e.g. to format dune files only.
This is currently not supported since the .ocamlformat file is the trigger for the lint stage and is not present in that scenario.

While you could parse the dune-project file to decide whether formatting should be enabled, an even simpler solution would just be to trigger it every time.
It is always possible to disable the dune formatting:

• With formatting disabled for dune-lang >= 2.0
• By not having the (using fmt ...) stanza for dune-lang < 2.0

In both cases, dune build @fmt does not fail if dune formatting is disabled, it just does nothing.

Keep log files from projects after they disable ocaml-ci

When a repository enables ocaml-ci, then disable it after some commit have been pushed the log files from those commits don't seem to be available anymore. For instance: https://ci.ocamllabs.io/github/ocaml-ppx/ppx/commit/40e5a35a4386d969effaf428078c900bd03b78ec/variant/(lint-fmt) previously showed some content but the ocaml-ci has been disabled for this project since and the page is now showing Error: Failed: Invalid GitHub repo "ocaml-ppx"/"ppx".

Docker does not cache past COPY

in opam-ci I noticed builds not taking advantage of the cache. Here is a reproduction of what ocaml-ci does.
Let's start with a simple setup:

$ cat Dockerfile
# syntax = docker/dockerfile:1.1.5-experimental
FROM ocurrent/opam@sha256:5bb6268b1fb7a8dfceffd92383cb4c72d0c3f995231ff916fa6aaf0b29bff4df
COPY --chown=opam . /src/
RUN echo hehehe
$ git clone --recursive "https://github.com/kit-ty-kate/opam-repository.git" repo
[...]

Let's copy and unpack the git directory to the right commit the same way ocaml-ci does and then build:

$ mkdir test20
$ cp -a -- repo/.git test20/.git
$ git -C test20 reset -q --hard 49560cde56610f9fd03c0af5ef852e9b873ecf6c
$ DOCKER_BUILDKIT=0 docker build -f Dockerfile test20
Sending build context to Docker daemon  129.7MB
Step 1/3 : FROM ocurrent/opam@sha256:5bb6268b1fb7a8dfceffd92383cb4c72d0c3f995231ff916fa6aaf0b29bff4df
 ---> 9ad67388a87c
Step 2/3 : COPY --chown=opam . /src/
 ---> e39d2567cbf2
Step 3/3 : RUN echo hehehe
 ---> Running in d358af5eec78
hehehe
Removing intermediate container d358af5eec78
 ---> 9478116f67b2
Successfully built 9478116f67b2

Now let's try again but in another directory:

$ mkdir test21
$ cp -a -- repo/.git test21/.git
$ git -C test21 reset -q --hard 49560cde56610f9fd03c0af5ef852e9b873ecf6c
$ DOCKER_BUILDKIT=0 docker build -f Dockerfile test21
Sending build context to Docker daemon  129.7MB
Step 1/3 : FROM ocurrent/opam@sha256:5bb6268b1fb7a8dfceffd92383cb4c72d0c3f995231ff916fa6aaf0b29bff4df
 ---> 9ad67388a87c
Step 2/3 : COPY --chown=opam . /src/
 ---> 9647c5da5b2b
Step 3/3 : RUN echo hehehe
 ---> Running in 62826c30a4b9
hehehe
Removing intermediate container 62826c30a4b9
 ---> 1a5fb44c6a03
Successfully built 1a5fb44c6a03

We can see here that even though the two directory should be the same, docker doesn't think so and builds the image twice.
Now let's see what's actually different:

$ diff -ru test20 test21
Binary files test20/.git/index and test21/.git/index differ
diff -ru test20/.git/logs/HEAD test21/.git/logs/HEAD
--- test20/.git/logs/HEAD	2020-04-24 09:16:30.400177079 +0100
+++ test21/.git/logs/HEAD	2020-04-24 09:19:21.788589883 +0100
@@ -1,2 +1,2 @@
 0000000000000000000000000000000000000000 a1f2c8222b2a755968b2212471fd1732bdaf6f3d Kate <[email protected]> 1587716122 +0100	clone: from https://github.com/kit-ty-kate/opam-repository.git
-a1f2c8222b2a755968b2212471fd1732bdaf6f3d 49560cde56610f9fd03c0af5ef852e9b873ecf6c Kate <[email protected]> 1587716190 +0100	reset: moving to 49560cde56610f9fd03c0af5ef852e9b873ecf6c
+a1f2c8222b2a755968b2212471fd1732bdaf6f3d 49560cde56610f9fd03c0af5ef852e9b873ecf6c Kate <[email protected]> 1587716361 +0100	reset: moving to 49560cde56610f9fd03c0af5ef852e9b873ecf6c
diff -ru test20/.git/logs/refs/heads/master test21/.git/logs/refs/heads/master
--- test20/.git/logs/refs/heads/master	2020-04-24 09:16:30.400177079 +0100
+++ test21/.git/logs/refs/heads/master	2020-04-24 09:19:21.788589883 +0100
@@ -1,2 +1,2 @@
 0000000000000000000000000000000000000000 a1f2c8222b2a755968b2212471fd1732bdaf6f3d Kate <[email protected]> 1587716122 +0100	clone: from https://github.com/kit-ty-kate/opam-repository.git
-a1f2c8222b2a755968b2212471fd1732bdaf6f3d 49560cde56610f9fd03c0af5ef852e9b873ecf6c Kate <[email protected]> 1587716190 +0100	reset: moving to 49560cde56610f9fd03c0af5ef852e9b873ecf6c
+a1f2c8222b2a755968b2212471fd1732bdaf6f3d 49560cde56610f9fd03c0af5ef852e9b873ecf6c Kate <[email protected]> 1587716361 +0100	reset: moving to 49560cde56610f9fd03c0af5ef852e9b873ecf6c

It seems that git stores a timestamp when calling git reset --hard <hash> which discard caching from docker's point of view. I believe that if we manage to avoid the use of git reset we should be able to lift this issue

Jobs started because opam-repository has been updated should have a low priority

Feel free to close this issue if I'm wrong but it seems that the jobs started by ocaml-ci when a dependency has been updated in opam-repository are started with a hight priority.

However I would argue that this should not be the case as, for instance a new release of dune would start (and has started, today) more than 8k jobs at the same time, which blocks regular commits and PRs from being tested for several hours.

Documentation linting job does not include the source code

It appears that the lint-doc Dockerfile does not include the source code of the project when running dune build @fmt when triggered from the GitHub pipeline.

Running dune build @doc inside a directory that is not a Dune project (i.e. contains no dune-project files in the ancestry of the root nor dune files in its descendents) will succeed, leaving an empty _build directory:

❭mkdir /tmp/doc && cd /tmp/doc && dune build @doc
Done: 0/0 (jobs: 0) •

This has the effect of creating false-positives on all project with odoc dependencies and a false-negative for the Odoc repository itself (since the analysis phase correctly determines that odoc does not need to be installed from Opam, then tries to build the documentation inside an empty tree.)

The fix is to copy the project source tree into the current directory before performing dune build @doc. Thanks to @jonludlam for the report.

ocaml-ci assumes default branch is "master"

Hi,

When the default branch is not named "master", it does not display the status correctly.

For example mirage/ca-certs uses "main", and it shows a gray icon for that repository on https://ci.ocamllabs.io/github/mirage.

No matching version of ocamlformat, last ocamlformat release makes lint-fmt fail

Hi,
It seems the opam cache is not updated, as it does not find the last release of ocamlformat 0.17.0 (see https://ci.ocamllabs.io/github/realworldocaml/mdx/commit/3841447381115fdb4bf60b6f3f8f5fef5fa53326/variant/(lint-fmt)), I was wondering if the update was automatic but could take some time, or if I something needs to be in the project source to indicate ocaml-ci to update the opam packages. Thanks for your great work on ocaml-ci!

Test against beta versions of OCaml

Related to #95, we could test against beta / release-candidate versions of the OCaml compiler, and would likely want any failures to be non-fatal to CI.

Support for the GitHub Checks API would allow us to report this information back to GitHub without failing the CI, but even just showing the result on the web API is a step in the right direction.

Lint documentation comments in the CI

It might be nice to have a linting stage that checks that the documentation comments in the project are well-formed.

ODoc now supports a 'fatal warnings' flag that could be used for this purpose. This feature is scheduled to be released in ODoc 1.5 (see ocaml/odoc#405).

Support the GitHub Checks API

This would allow us to report rich information back to GitHub PRs directly, potentially including:

• success/failures in the compiler matrix (including 'non-fatal' beta compilers);
• linting errors, for source code and documentation (#94);
• impact of changes on rev-deps.

very cool!

come here from https://discuss.ocaml.org/t/ann-capnp-rpc-0-4-0/4524. The doc is one of the best among ocaml lib.

Amazing that whole project is using ocaml. Would like to know how this happen. Especially, how to call capnp-rpc service in the frontend. Can I call the backend service using Javascript from web browser?

opam update step should use --unlock-base

Context: https://ci.ocamllabs.io/github/ocurrent/ocaml-version/commit/61b6842fbee2d750d0af8489f76036d56db8338c/variant/debian-10-4.12

The recently merged ocaml/opam-repository#17541 causes an update to the ocaml-variants.4.12.0+trunk package and an upgrade to the ocaml-config package from version 1 to version 2. However, ocaml-ci does a normal opam update -u which only recompiles root packages in the base, so only picks up the recompilation:

opam@3f6ac4439947:~/opam-repository$ opam upgrade
The following actions will be performed:
  - recompile ocaml-variants 4.12.0+trunk* [upstream changes]
  - recompile ocaml-config   1             [uses ocaml-variants]
  - recompile ocaml          4.12.0        [uses ocaml-variants]
  - recompile opam-depext    1.1.4         [uses ocaml]
===== 4 to recompile =====
Do you want to continue? [Y/n] n

the compiler package is pinned:

opam@3f6ac4439947:~/opam-repository$ opam pin list
ocaml-variants.4.12.0+trunk    version  4.12.0+trunk

so it is safe to run opam update && opam upgrade --unlock-base:

opam@3f6ac4439947:~/opam-repository$ opam upgrade --unlock-base
The following actions will be performed:
  - recompile ocaml-variants 4.12.0+trunk* [upstream changes]
  - upgrade   ocaml-config   1 to 2
  - recompile ocaml          4.12.0        [uses ocaml-config, ocaml-variants]
  - recompile opam-depext    1.1.4         [uses ocaml]
===== 3 to recompile | 1 to upgrade =====

At present the job fails later because opam-depext ocaml-config.2 will try to install the package, but can't because it's in the base.

Test packages on the full range of supported OCaml versions

To my understanding, the analysis phase for Opam projects works roughly as follows:

• determine the set of OCaml versions at which all packages are co-installable (by attempting to solve the constraints);
• for each such version, emit the set of opam packages to install (as returned by the constraint solver).

This has the consequence that only OCaml versions at which all packages are co-installable are tested in CI, even if certain packages have support for a larger set. This is problematic for repositories containing many packages, since it can severely limit the test matrix. For instance, Irmin is currently only tested on OCaml 4.11 despite almost all of the packages supporting back to at least 4.08.

It would be nice to have a mechanism that tests each package on the full range of OCaml versions that it (allegedly) supports. An obvious solution is to branch the pipeline for each package and install dependencies separately – as was previously done with ocaml-ci-scripts – but this is inefficient and doesn't provide any information about co-installability. Here's an initial proposal for how we might do better:

• for each OCaml version in the test matrix, determine the largest co-installable subset of opam packages p_i;
• for each non-empty p_i, emit the set of opam dependencies to install and the set of Dune packages d_i corresponding to p_i. The test phase can then use dune runtest --only-packages [... d_i] to ensure that only these packages are tested at that version.

The magic here is in calculating the p_is and d_is. I suspect p_i can be approximated by initially assuming total co-installability and then using counter-examples to iterate downwards, but perhaps there is better built-in solver magic for this. A complication is that bizarre sets of constraints can result in multiple p_is, in which case it seems sufficient to just pick one deterministically. Once that's done, d_i might be computable from p_i using opam-dune-lint?

depexts from local packages are ignored

If one of the opam packages in the repository you're testing using the current pipeline has custom depexts, those depexts are not going to be installed as, currently, opam depext is only invoked on the dependencies and not on the local opam packages themselves.

lint-fmt: cannot find a solution for ocamlformat itself

Hi,

Context:
The next version of ocamlformat will use ppxlib >= 0.18.0. In the PR where I'm adding this dependency, the lint-fmt job fails:

https://ci.ocamllabs.io/github/ocaml-ppx/ocamlformat/commit/8b1ed80e8b77be4b4c4ba52daf2d8447c3fc6602/variant/(lint-fmt)

# Detecting depexts using vars: arch=x86_64, os=linux, os-distribution=debian, os-family=debian
[ERROR] No solution for ocamlformat: The following dependencies couldn't be met:
          - ocamlformat -> ppxlib >= 0.18.0
              no matching version

I think there's no matching version because all released versions conflict with that ppxlib version (all released versions depend on omp 1.x, this ppxlib version on omp 2.x).

But I thought that there was special handling so that ocamlformat could use its dev version to format itself, so I'm not sure why it can't find a solution.

Thanks!

Support pin-depends

Say I have two packages:

• A
• B, which depends on the master branch of A in opam's pin-depends field.

Now I have the following scenario:

• Commit to B, CI runs normally.
• Commit to A, adding a depext D.
• Commit to B.
  When the CI runs for that last commit, it fails during A install because of missing system dependency (it did not install D).

It seems that the pin is cached by the CI during the depext stage, and can't detect upstream changes, since the pin is against a branch name instead of a commit, and the pinned package is not pulled at this moment.

/cc @Julow

Docker error "No such image"

The lint variant for this commit (https://ci.ocamllabs.io/github/ocaml-ppx/ocamlformat/commit/fcf69afb6e5f93e697ad36c43e7028ed53081f4d/variant/lint) fails with a weird error:

2020-02-24 15:05.04: New job: "docker" "--context" "default" "run" "--rm"
                     "-i"
                     "sha256:b718ec4bdbd1e07569f45a3ec4b52d92be760844d934c4c3e733a36730bf62fe"
                     "sh" "-c"
                     "dune build @fmt || (echo "dune build @fmt failed"; exit 2)"
2020-02-24 15:05.04: Exec: "docker" "--context" "default" "run" "--rm" 
                           "-i" "sha256:b718ec4bdbd1e07569f45a3ec4b52d92be760844d934c4c3e733a36730bf62fe" 
                           "sh" "-c" "dune build @fmt || (echo "dune build @fmt failed"; exit 2)"
docker: Error response from daemon: No such image: sha256:b718ec4bdbd1e07569f45a3ec4b52d92be760844d934c4c3e733a36730bf62fe.
See 'docker run --help'.
2020-02-24 15:05.07: Job failed: Command "docker" "--context" "default" "run" "--rm" "-i" "sha256:b718ec4bdbd1e07569f45a3ec4b52d92be760844d934c4c3e733a36730bf62fe" 
"sh" "-c" "dune build @fmt || (echo "dune build @fmt failed"; exit 2)" exited with status 125

The error is still here after a few rebuilds.

Current_web graph render fails poorly if Graphviz is not installed

In particular, no helpful error message is printed (just the alt text for the image):

Thanks to @gs0510 for the report.

CI check suggestion: dune-project and opam files are in sync

some projects generate their opam file from dune-project, but keep the opam files in the git repository for technical reasons (to keep opam pin --dev working).

the generated opam file contain a comment "please do not edit, generated by ..." - that is fine. but already more than once it happened to me that I modify (out of habit) the opam file, get the PR merged, and discover only at a later point that the dune-project and opam are now out of sync.

it would be great if the ocaml CI could check that after a dune build the git diff is empty (i.e. there are no modified files). thanks.

Support build status badges

Would be nice to have something colourful to put in READMEs. Likely requires exposing new per-repository endpoints which check the build status and redirect to an appropriate badge. For example (courtesy of @emillon):

• https://img.shields.io/badge/ocaml--ci-builds-informational?logo=ocaml&style=for-the-badge
• https://img.shields.io/badge/ocaml--ci-passing-success?logo=ocaml&style=for-the-badge
• https://img.shields.io/badge/ocaml--ci-failing-critical?logo=ocaml&style=for-the-badge

Add OCaml+32bit

In some case, a test with OCaml +32bit is useful, specially for:

• optint which the compilation differs between 32bit/64bit
• mirage-crypto which use some specials instructions (eg. mirage/mirage-crypto#60

More globally, you should be able to test our libraries to ensure a support for arm32/esp32. So, I don't have any clues to add it.

OCaml micro releases cause problems

When OCaml 4.10.1 was released, the PPC base image got updated before the x86 one. However, the CI assumes that the compiler version opam var from the x86 image will be the same for all arches. This meant that it tried to get the PPC 4.10.1 base image to use OCaml 4.10.0, which didn't work.

I manually redid the x86 pull, but we should probably find another way to get the variables for non-x86 platforms. One option would be to use the cluster for the query. Another might be to store the vars as labels on the image.

Continuously support new OCaml versions

We've been statically tracking support for the 4.0x compiler versions in this repository (#47, #53, #66, #90).

This information should probably be derived from ocaml-version, so that we don't have to manually update it each time.

Client failure on flaky connexions

I'm not sure if that should be an issue here or in cohttp, I can redo it elsewhere if so.

When our current internet connexion is a slow and very flaky one (through train wifi for instance), the client terminates on any command involving connecting to the server with the following error message:

ocaml-ci: internal error, uncaught exception:
          (Failure
            "Failed: TLS connection failed: attempted to write to a closed flow")

If I switch to a more stable connexion (though a shared connexion from my 4G phone for example, it works again)

Should we skip linting when ocamlformat_source is None?

I have a question about this part of the code:

ocaml-ci/lib/lint.ml

Lines 22 to 26 in a94d6da

	@@ (match ocamlformat_source with
	| Some src -> install_ocamlformat src
	| None -> empty)
	@@ copy ~chown:"opam" ~src:["./"] ~dst:"./" ()
	@@ run "opam exec -- dune build @fmt || (echo \"dune build @fmt failed\"; exit 2)"

It seems when ocamlformat_source is None, it is impossible to have a working ocamlformat around. If so, should we skip dune build @fmt completely?

Opam files are not checked during lint stage

The CI currently does not fail or notice when an invalid opam file (w.r.t. opam lint) is used in the repo, e.g. when there is no synopsis/description.
Maybe that could be checked during the lint stage?

Pass -q to git reset --hard

Will suppress the many lines of "Checking out files" - errors will still be displayed.

builder running on an old centos (version 7)

Hi,

I noticed that the centos builder uses centos7, while centos8 has been out for several months (according to wikipedia September 2019). Is there a good reason for that, or could the centos builder be upgraded to centos 8?

Thanks.

Integrate with GitHub Actions

Someone posted the GitHub Actions OCaml integration on /r/ocaml reddit. Maybe it makes sense to coordinate?

See the sources of the template here: https://github.com/fangyi-zhou/actions-ocaml

cc @fangyi-zhou

Add a rebuild button for a whole commit

It would be very useful to add a rebuild button on each github/<user>/<repo>/commit/<hash>/ pages, for the whole pipeline used on this commit to be rebuilt.

monorepo: fails when dependencies are locked using a newer version of opam-repository

Hi,

At the moment the following happens for monorepo builds:

• during the analysis phase, the solver is queried for ocaml, dune and opam-monorepo. The oldest opam-repository commit that knows about that solution is selected.
• during the build phase, the lock file is pinned in order to resolve depexts.

The issue is that if the lock file has been produced with an opam-repository state that's more recent than the result of the analysis, it won't be able to compute the depexts.

As an example (inspired by a true story with RWO), querying for ocaml=4.10.1, dune=2.7.1, opam-monorepo=0.1.0 resolves to ocaml/opam-repository@aaf8fed. If the project refers to conf-gmp, today it will get locked to version 3. But this version has been uploaded only after that opam-repository commit.

As a workaround, conflicting with conf-gmp > 2 kind of works, but this is only a short term solution since it will apply to all transitive dependencies released after.

The root issue is that we need access to the repository to determine depexts. That's a known opam-monorepo 0.1.0 limitation which is going to be lifted soon, but I'm wondering if there's something we can do in the meantime on the CI side. Maybe passing the pin-depends to the solver would work, but if done too naively it will try to install them via opam in the build phase.

Show previous builds in web UI

The "admin" UI (which is now readable to the public) allows navigating to previous builds. The public UI should allow something similar (but it should probably also link to previous builds of the same PR, even if the head commit changed).

Update immediately to opam-repository changes

At the moment, we update the base Docker images once a week, and this brings in the latest opam-repository changes. However, it would be good to be able to test against new releases immediately.

The problem with doing opam update before each job is that we would lose the benefits of caching, because the layer hash would be different each time.

Some options:

1. If opam could extract from a repository just the information it needed, we could throw away the repository and just keep that. Then repository changes would invalidate the cache only if something we needed changed. There doesn't seem to be any way to do that, though.

2. We could add a configuration file where the user specifies the mimimum repository commit they need. e.g.

    (ocaml-ci
     (minimum-opam-commit abcd123e))
   

   If the repository doesn't contain that commit then we update to it.

Ubuntu 20.04 jobs blocked on keyboard-configuration dialog

Currently, certain CI jobs on the Ubuntu 20.04 runner are getting blocked indefinitely on a keyboard-configuration dialog triggered by debconf:

#15 229.6   95. Ukrainian
#15 229.6   96. Urdu (Pakistan)
#15 229.6   97. Uzbek
#15 229.6   98. Vietnamese
#15 229.6   99. Wolof
#15 229.6 Country of origin for the keyboard: 2020-07-21 08:59.46: Cancelling: Timeout (60.0 minutes)
Job cancelled

See for instance: https://ci.ocamllabs.io/github/mirage/irmin/commit/66366d36b7b822ffe73b4ebff27892d7626b4949/variant/ubuntu-20.04-ocaml-4.10.

This appears to happen as part of the opam depext call, where passing -yy to apt-get install is not sufficient to make the install run non-interactively. See moby/moby#27988 (comment), which suggests using debconf-set-selections to explicitly tell debconf to be non-interactive.

Test opam constraints of multi-package repositories separately

OCaml-CI's current strategy for running tests of multi-package repositories is to first pin all of the packages and then install them all in the same switch and run dune runtest. This approach has two undesirable consequences:

• missing external dependencies may be ignored by OCaml-CI, if those dependencies are introduced by another package in the same repository (e.g. irmin-pack and irmin-git both depend on alcotest, but this dependency is missing from irmin-git.opam);

• missing internal dependencies will always be ignored by OCaml-CI, since those dependencies are always available in the test switch by default.

It would be nice to have some CI solution that still tests that the opam metadata corresponds to the true project dependencies, as otherwise this becomes an issue that is only discovered at the point of release (see mirage/repr#12). This is the major reason that we still have Travis CI running (very slowly) on certain mirage/ repositories.

CC @avsm.

OCaml-CI doesn't support the `extra-source` field

We're using OCaml-CI for ocaml-gmp but it seems to fail because is uses wget as a fallback option to fetch GMP sources. The GMP sources were expected to be fetched using the extra-source field of the opam definition, but it doesn't look like it's the case.

• CI log: https://ci.ocamllabs.io/github/mirage/ocaml-gmp/commit/801dda44df59702989d28d8abf57b6d7f366bf7e
• Opam file: https://github.com/mirage/ocaml-gmp/blob/master/gmp.opam

lint-doc should install doc dependencies

The lint-doc phase does not currently seem to install the doc dependencies (packages tagged with {with-doc}).
This might be an issue for packages that have custom steps (e.g. building pdfs from latex, in which case they might have "conf-texlive" {with-doc} or something similar).

To test the documentation, ocaml-ci currently installs odoc and dune by hand in the lint-phase

ocaml-ci/lib/lint.ml

Line 42 in 7df11b8

run ~network ~cache "opam depext -i conf-m4 && opam depext -i dune 'odoc>=1.5.0'";

However if ocaml-ci were to install the doc dependencies this step would be redundant.

Furthermore, if odoc were to not be listed in the dependencies it would still be an issue for users down the road but would not be indicated by ocaml-ci. In my opinion ocaml-ci should only check that odoc >= 1.5.0 is installed but not install it. This incidently will also make this phase faster by removing two calls to opam depext (which itself calls opam twice)

opam-lint should warn if the ocaml constraint doesnt match analysis

The analysis phase solves for the versions of OCaml that have a valid solution.

In the opam-lint check, we should also check that the ocaml constraint matches the package set found by analysis. It's often out of sync as dependencies evolve.

Have a way of setting custom installation and testing steps

Currently ocaml-ci uses a simple opam install -t to install and test the packages. However there might be pre-install or pre-tests steps or any kind of custom steps that we'd want the CI to do. For this, some sort of config file might be useful. This configure file could also serve to set the system we would want the packages to be tested on in the future if we want to test more than the current default (alpine)

Spurious errors due to Docker BuildKit

A significant proportion of CI jobs fail with the following spurious error:

#5 [internal] load metadata for docker.io/ocurrent/opam@sha256:8115c13b8c30...
#5 DONE 0.0s

#10 [internal] load build context
#10 transferring context: 453.44kB 0.0s done
#10 DONE 0.1s
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c (opam install ocaml-ci-web ocaml-ci-service ocaml-ci-client ocaml-ci-api --dry-run --deps-only -ty; echo $? > /tmp/exit-status) | tee /tmp/opam-plan; exit $(cat /tmp/exit-status)]: runc did not terminate sucessfully
2020-01-20 13:04.46: Job failed: Docker build exited with status 1
2020-01-20 13:04.46: Log analysis:
2020-01-20 13:04.46: >>> #10 DONE 0.1s
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c (opam install ocaml-ci-web ocaml-ci-service ocaml-ci-client ocaml-ci-api --dry-run --deps-only -ty; echo $? > /tmp/exit-status) | tee /tmp/opam-plan; exit $(cat /tmp/exit-status)]: runc did not terminate sucessfully (score = 10)
2020-01-20 13:04.46: BuildKit missing output

It's not clear to me what's causing this; I know @talex5 has some ideas.

Failures of pre-job pipeline stages are difficult to observe

The ocaml-ci web interface shows only 'jobs', and not all of the pipeline stages taking place. If a dependency of one of these jobs fails, the job appears as 'not started' on the interface, but the user has no feedback as to why. For example, as a result of #76, the build of the OCamlformat image was failing:

which is not well-described by https://ci.ocamllabs.io:

opam-depext does not update system dependencies if necessary

It is possible for system dependencies to fail to be installed because the package manager state in the system docker images is out-of-date. For example:

#15 [stage-0 8/11] RUN --mount=type=cache,target=/home/opam/.opam/download-c...
#15 0.706 # Detecting depexts using vars: arch=x86_64, os=linux, os-distribution=ubuntu, os-family=debian
#15 2.787 # The following system packages are needed:
#15 2.787 debianutils
#15 2.787 gnuplot-x11
#15 2.787 libgmp-dev
#15 2.787 m4
#15 2.787 perl
#15 2.787 pkg-config
#15 7.835 The following command needs to be run through "sudo":
#15 7.835     apt-get install -qq -yy debianutils gnuplot-x11 libgmp-dev m4 perl pkg-config
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisc-export1100_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libdns-export1104_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisc1100_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libdns1104_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisccc161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisccfg163_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libbind9-161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/liblwres161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-host_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/n/nss/libnss3_3.42-1ubuntu2.1_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/snapd/snapd_2.40+19.04_amd64.deb  404  Not Found [IP: 91.189.88.174 80]
#15 15.01 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
#15 15.01 OS package installation failed
#15 ERROR: executor failed running [/bin/sh -c awk < /tmp/opam-plan '/-> installed/{print $3}' | xargs opam depext -iy]: runc did not terminate sucessfully
------
 > [stage-0 8/11] RUN --mount=type=cache,target=/home/opam/.opam/download-cache,uid=1000 awk < /tmp/opam-plan '/-> installed/{print $3}' | xargs opam depext -iy:
------
failed to solve with frontend dockerfile.v0: failed to solve with frontend gateway.v0: rpc error: code = Unknown desc = failed to build LLB: executor failed running [/bin/sh -c awk < /tmp/opam-plan '/-> installed/{print $3}' | xargs opam depext -iy]: runc did not terminate sucessfully
2019-11-29 11:05.36: Job failed: Docker build exited with status 1
2019-11-29 11:05.36: Log analysis:
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisc-export1100_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libdns-export1104_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisc1100_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libdns1104_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisccc161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisccfg163_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libbind9-161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/liblwres161_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/bind9-host_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/n/nss/libnss3_3.42-1ubuntu2.1_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: >>> E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/s/snapd/snapd_2.40+19.04_amd64.deb  404  Not Found [IP: 91.189.88.174 80] (score = 30)
2019-11-29 11:05.36: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/b/bind9/libisc-export1100_9.11.5.P1+dfsg-1ubuntu2.5_amd64.deb  404  Not Found [IP: 91.189.88.174 80]

This could be resolved by using opam depext -u to update the package manager state, ideally only when necessary so as not to slow down the build in the common case.

make ppc64le/ppc64be opt-in

In the analysis phase, we could scan for C files in the opam package and/or a x-test-on-arch: ppc64 flag and turn off that architecture if not found.

The reason is that it would be nice to activate ppc64be testing, but we only have about 8 CPUs available on that arch. So that means we do need to minimise the number of packages using it.

Test dune release mode

There are still some quirks which can be found only when compiling the project in dune's release mode (instead of the default dev mode). For instance, some symlinks are present only in dev mode (and pass the CI steps) but can break in release mode (and be caught by dune-release)

/cc @hannesm who had the issue recently.

main page and github checks status disagree with build page status

Apologies in advance for the reproduction case which is probably going to be stale somewhat soon.
Right now, when I go on https://ci.ocamllabs.io/github/ocaml/merlin everything is red, but if I pick one of the branch actively being developped (e.g. master), I can see a majority of the tests as orange and "(active)".
If indeed I click on one such test, the log ends with: "2021-01-14 14:42.43: Waiting for resource in pool OCluster".

Equally annoyingly, this PR ocaml/merlin#1238 is flagged as failing to pass the CI, however if I click on "Details", I can see no failure, only active builds.