ocaml/ocaml-ci-scripts

Skeletons for CI scripts

CI scripts for OCaml projects

Supported CI:

• stable: Travis CI Ubuntu, Debian and OSX workers.
• experimental: Appveyor Windows Server 2012 R2 (x64) workers.

ocaml-ci-scripts's Issues

Docker-based Travis builds failing with OCaml 4.04

Recently most of our Travis builds started failing with the following error:

opam: "execvp" failed on ci-opam: No such file or directory

For example, this one: https://travis-ci.org/xapi-project/ocaml-netlink/jobs/313427741#L581

The cause is probably this issue: https://caml.inria.fr/mantis/view.php?id=7640

I'd be grateful for any help you can provide 🙂 - It would be really helpful for us if this was fixed or there was a workaround that we could use to make our Travis builds work again.

appveyor-opam.sh references fdopen/ which is 404ing

https://raw.githubusercontent.com/ocaml/ocaml-ci-scripts/master/appveyor-opam.sh has

opam init -a mingw "https://github.com/fdopen/opam-repository-mingw.git" --comp "$SWITCH" --switch "$SWITCH"

Unfortunately https://github.com/fdopen and https://github.com/fdopen/opam-repository-mingw.git are both 404ing for me today.

Destination directory of documentation

The docgen script copies the contents of all *.docdir directories as follows:

cp _build/*.docdir/* $DOCDIR

This just clobbered my gh-pages directory. There are several issues here:

1. If there is more than one .docdir directory containing files of the same name (typically index.html), the copy command can clobber itself.
2. The user might not even want to have the API-documentation in the topmost directory of the gh-pages branch.
3. Documentation files that don't exist anymore will not be removed.

The following may be more closer to what most users want:

rsync -a --delete  _build/*.docdir $DOCDIR

This would keep API-documentation of distinct libraries of a repository separate and allows the user to have their own index.html to refer to the appropriate documentation.

Alternatively, we could introduce new variables to indicate the destination directory or what should happen. At least for my purposes the above would be perfectly sufficient.

How to avoid an optional dependency being built?

I've tried many variations on:

    - PACKAGE=biokepi INSTALL=false TESTS=false PINS="trakeva:https://github.com/smondet/trakeva.git ketrew:https://github.com/hammerlab/ketrew.git" DEPOPTS=postgresql EXTRA_DEPS=postgresql PRE_INSTALL_HOOK='opam install postgresql'

but the build always fails on MacOSX: https://travis-ci.org/hammerlab/biokepi/builds/82633540
(because of Sqlite needing a hack: https://github.com/smondet/trakeva#sqlite3-on-macosx)

Sqlite3 is an optional dependency and postgresql should replace it →
https://github.com/hammerlab/ketrew/blob/master/opam#L30
how do I convince the ocaml-travisci-skeleton to not install it?

.travis-opam.sh doesn't run tests as part of depopts build

At the moment depopts_run and tests_run are handled orthogonally; there's no way to test e.g. a package which is only built if Lwt is installed.

CI no longer works: master/yorick.mli has moved

https://github.com/ocaml/ocaml-ci-scripts/blob/master/.travis-mirage.sh tries to fetch https://raw.githubusercontent.com/ocaml/ocaml-ci-scripts/master/yorick.mli. This file no longer exists.

@samoht : looks like you moved this in #180

Ability to override OPAM_SWITCH

I'd like to leave a matrix testing 4.05.0+rc1, so I added OCAML_VERSION=4.05 OPAM_SWITCH=4.05.0+rc1 to the test matrix. However, this results in this:

+OPAM_SWITCH=4.05.0+rc1
+BASE_REMOTE=git://github.com/ocaml/opam-repository
+UPDATE_GCC_BINUTILS=0
+UBUNTU_TRUSTY=0
+INSTALL_XQUARTZ=true
+install_on_linux
+OCAML_VERSION=4.02
+OPAM_SWITCH=4.05.0+trunk

Would it be OK for install_on_linux to go with OPAM_SWITCH if it's specified?

`make configure` awkward for Mirage unikernels

The docs say:

It assumes the existence of a Makefile at the root of the repo having targets configure and build.

However, Makefile is normally generated by mirage, so adding one to the source repository will cause lots of trouble with git.

CI for multi opam packages

Is there a way to test all packages defined in a single git repository? I'm transitioning cohttp away from depopts, and I'd like to make sure travis is still working.

Yorick uses sh but some pre-Elizabethan scripts used bash features

Should we use bash or try to stay inside the sh feature set? @mor1's port is broken by using bash's string substituion feature # for XENIMG.

Evaluate users of 3.12 tests and remove support for that compiler if impact is minimal

See #81 where 3.12 on OS X fails due to the well-known -fno-defer-pop issue when OS X made gcc an alias for clang.

Whitelist PPA for faster builds

According to http://docs.travis-ci.com/user/apt/ we can ask for the opam PPA and packages to be whitelisted, which would allow building with sudo: false, which is apparently much faster.

document reverse dependency feature

REVDEPS is not mentioned in the README

What is the license for this code?

We're suggesting that people copy these build scripts into their own projects. It would be good to say what the license for these files is.

Add option to inject commands

Hi,

I am using OASIS for my project but as I don't want to have it as a dependency of my project, I pregenerate the setup.ml in release tarballs. Therefore it is not listed in the OPAM file. Could there be an additional option for calling additional commands? I'd need to install oasis via OPAM and then run oasis setup.

If interested, I can prepare a PR.

Add an env variable for adding pinned packages

Could be:

- OPAM_PIN_ADD1=<name:url> OPAM_PIN_ADD2=<name:url>

Directly contribute to Travis?

It is possible to add community supported language to TravisCI, such that one can use language: ocaml in travis.yml. Details are at http://docs.travis-ci.com/user/languages/community-supported-languages/

opam: unknown command `depext'.

All builds have started failing with opam: unknown command depext using the vanilla travis-opam scripts.

allow running test after DEPOPTS are installed

Since this got applied and then reverted would it be acceptable to have an env var in .travis.yml that allows the user to choose whether they want to run the tests after installing DEPOPTS or not?

travis-docker defaults to an old travis-opam version

When using travis-docker, this part determines what version of travis-opam should be used:

ocaml-ci-scripts/.travis-docker.sh

Lines 36 to 42 in 0749cbb

	if [ $fork_user != $default_user -o $fork_branch != $default_branch ]; then
	echo RUN opam remove travis-opam >> Dockerfile
	echo RUN opam pin add -n travis-opam \
	https://github.com/$fork_user/ocaml-ci-scripts.git#$fork_branch \
	>> Dockerfile
	echo RUN opam depext -i travis-opam >> Dockerfile
	fi

In particular, this means that by default, the latest version on opam is used. At this time, it means 1.0.3, even though 1.1.0 is released and includes useful fixes such as #146.

I believe that releasing 1.1.0 on opam would be enough to fix this issue.

As a workaround, it is possible to set FORK_BRANCH=v1.1.0 in the environment.

Thanks!

travis-opam spins forever with an opam parse error

It attempts to spin up nano repeatedly; should mark it non-interactive.

See e.g. https://travis-ci.org/avsm/ocaml-dockerfile/jobs/100559929

New release

There's quite a lot of unreleased code on master (currently 60 commits since the last release, v1.0.2).

It would be useful to have a new release, since some recent changes (e.g. #96) are expected to fix longstanding problems with CI builds such as this.

pass variables in the build matrix to the container

When testing mirage, we'd like to pass the MODE environment variable to the container we start with .travis-docker.sh . It's easy enough to add this to the list of variables copied into env.list and passed to the container, but I wonder whether a more general solution might be widely useful.

appveyor fails because of missing jq

Witness https://ci.appveyor.com/project/ygrek/ocurl/build/1.0.6
Any way to fix it on appveyor side?

Don't try to remove base packages

From https://travis-ci.org/dsheets/ocaml-unix-fcntl/jobs/115832937#L1049:

opam remove lwt base-threads
Your request can't be satisfied:
  - Package base-threads is part of the base for this compiler and can't be changed
No solution found, exiting

This happens (at least) in the case of base packages in depopts.

Name change causes problems with pinning

The script .travis-docker.sh allows pinning ocaml-ci-scripts to enable using features from unreleased versions.

However, the name of the travis-opam binary has been changed in recent versions to ci-opam. Since the ocaml/opam Docker images and the .travis-docker.sh script both still use the old name, pinning recent versions rarely works.

Build on MacOSX consistantly fails while Linux passes

See: https://travis-ci.org/jnavila/plotkicadsch/jobs/185877988

The compilation consistantly passes with Linux, but fails as consistantly on Mac ;-)

The compilation fails with the following error:

- + /Users/travis/.opam/4.04.0/bin/ocamlfind ocamlopt -a -o src/plotkicadsch.cmxa
- ar: no archive members specified
- usage:  ar -d [-TLsv] archive file ...
- 	ar -m [-TLsv] archive file ...
- 	ar -m [-abiTLsv] position archive file ...
- 	ar -p [-TLsv] archive [file ...]
- 	ar -q [-cTLsv] archive file ...
- 	ar -r [-cuTLsv] archive file ...
- 	ar -r [-abciuTLsv] position archive file ...
- 	ar -t [-TLsv] archive [file ...]
- 	ar -x [-ouTLsv] archive [file ...]
- File "_none_", line 1:
- Error: Error while creating the library src/plotkicadsch.a
- Command exited with code 2.
- E: Failure("Command ''/Users/travis/.opam/4.04.0/bin/ocamlbuild' src/plotkicadsch.cma src/plotkicadsch.native src/plotgitsch.native -tag debug' terminated with error code 10")

Digging into similar issues, it seems that the ar utility on MacOSX doesn't accept creating empty archives where as GCC's ar is happy with it.

Use package name from opam if it exists

When the opam file contains a name, it should be used instead of the default my-package.

4.05 support

Hi,

OCaml 4.05+beta2 was released earlier, so I assume that a final version will be out in a couple weeks.
Can you please add support to it so that it is possible to test it on Travis?

Thanks!

remove deployment logic for mirage-www from travis-mirage

The logic for deploying the mirage-www website is specific to that project and is annoying to have in a separate repository. See #196 and #195 and likely more incoming PRs.

Use `depext`

See https://github.com/OCamlPro/opam-depext

Add optional reverse dependency compilation

Not enabled by default, but could be useful to track early reverse dependency constraints which needs to be updated.

DEPOPTS broken?

while I can find some export DEPOPTS=mirage-xen in here, mirage-xen itself is never installed!?!?

Debian Docker images are currently broken

This manifests as a failure to download a .deb which looks superficially like a network glitch but is in fact a permanent error.

From mirage/mirage#837 -- we use unstable stable and testing as names but these are aliases and they've recently been bumped, causing the Docker images to be out of sync.

I think we should

• regenerate the Docker images
• perhaps switch to using the release names (stretch etc)

Update version on brew (system is now 4.04)

Is there a reason that that the .travis-ocaml.sh script hasn't been updated to the new 4.04 system switch?

In the current state asking for 4.03. will switch to system which is now 4.04. I'm just wondering if there is a reason besides the "no one has noticed it yet."

Add an env variable to add alternate repositories

Could be:

- OPAM_REPO_ADD0=<url> OPAM_REPO_ADD1=<url>

By default, OPAM_REPO_ADD0 would be https://opam.ocaml.org/

DEPOPTS variable confusion

DEPOPTS is being used for 2 different purposes:

• as a true/false variable to determine if a run with the optional dependencies should be done
• the names of the optional packages

If you set DEPOPTS=true, you end up also having "true" as the name of the optional package to install.

The README should also be fixed. Right now it mentions the second usage above for DEPOPTS.

CircleCI support

It'd be useful for ocaml-ci-scripts to support CircleCI. Here's the documentation for the configuration.

Update 'latest' OCaml version

https://github.com/ocaml/ocaml-ci-scripts/blob/master/.travis-ocaml.sh#L36 is wrong. See mirage/ocaml-vchan#81 (comment).

OCAML_VERSION=4.01 fails to install on MacOS

My build log is here:
https://travis-ci.org/DIKU-EDU/ott/jobs/235615547

It complains that the XQuartz-2.7.6.dmg which it downloads from http://xquartz.macosforge.org/downloads/SL/XQuartz-2.7.6.dmg is not in a format that hdiutil can recognize.

OPAM env required?

Shouldn't this script evaluate the OPAM init scripts to adjust PATH? Or is it a deliberate decision not to do that? I would make a PR, but would like to know if there is some reason behind this.

Background: I have dependencies that install binaries via OPAM, but these are not in the PATH unless setting the OPAM variables properly.

OCAML_VERSION=4.04 leads to 4.04.0

whereas I'd expect it to be 4.04.2, i.e. the highest minor version. I suspect similar inconsistencies are around with other OCaml versions if you disregard the minor number.

Fix self-CI which uses downloads OCaml source from the master branch

The bootstrap scripts download source from this repository so the self-CI tests don't actually test the PR content. This caused the issue in #81/#83.

Wrong OCaml version for 4.02.3 image

It looks like image ocaml/opam:debian-stable_ocaml-4.02.3 contains OCaml 4.01.0 instead of 4.02.3.

> docker pull ocaml/opam:debian-stable_ocaml-4.02.3
debian-stable_ocaml-4.02.3: Pulling from ocaml/opam
28c297c6ec72: Already exists 
a3ed95caeb02: Already exists 
d3a2cca92131: Pull complete 
38f8b0265afb: Pull complete 
7f5a04423ce8: Pull complete 
Digest: sha256:09cb52431d5034beb5fd407f08d839a5a9185ebb2f4417decaa934a14537586e
Status: Downloaded newer image for ocaml/opam:debian-stable_ocaml-4.02.3
> docker run -ti ocaml/opam:debian-stable_ocaml-4.02.3 bash
opam@b4fa61ef998e:~$ opam config var ocaml-version
4.01.0

.travis-docgen.sh: line 21: KEEP: unbound variable

Travis fails with the error mentioned in the title:
https://travis-ci.org/frenetic-lang/frenetic/builds/144830896

Any ideas what the problem may be?

Support for compiler extensions

Hello!

Should we include support for compiler extensions?

e.g. introduce a variable OPAM_SWITCH_EXT to use for extension names' concatenation (strings like "+BER" or "+modular-implicits-ber") at the init command in .travis-ocaml.sh.

strymonas/staged-streams.ocaml#2

Support for setting package name

Currently, the scripts use the default my-package name. We could add something like:

- PACKAGE=<name>

in the env variable to set-ip correctly. Or we can read it in the opam file ...

supporting OSX backend

A few projects got the right to experiment with new osx boxes on Travis.

Right now this requires working around: ocaml/opam#1853
(brew install opam --HEAD seems to do it for now).

See these scripts using the OSX backend:

• https://github.com/hammerlab/ketrew/blob/use_trakeva/tools/travis_ci_test.sh
• https://github.com/samoht/assemblage/blob/master/.travis-ci.sh

Print the cause of REVDEP failure for most recent version

When attempting to install REVDEPS, it would be nice to print the reason that the latest version of a REVDEP fails to install.

TravisCI Log Folding

Travis' default shell provides the command travis_fold(action, name), where action is either open or close. Its implementation is here and simply involves outputting a string to trigger the fold.

Would it be possible to add these commands/string output into the scripts to make the logs easier to follow?