ocurrent/ocaml-dockerfile

OCaml interface for creating Dockerfiles

dockerfile — Dockerfile eDSL and distribution support

%%VERSION%%

Docker is a container manager that can build images automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands you would normally execute manually in order to build a Docker image. By calling docker build from your terminal, you can have Docker build your image step-by-step, executing the instructions successively. Read more at https://docker.com.

This library provides a typed OCaml interface to generating Dockerfiles programmatically without having to resort to lots of shell scripting and awk/sed-style assembly.

ocaml-dockerfile is distributed under the ISC license.

• HTML Documentation: https://www.ocurrent.org/ocaml-dockerfile/;
• Source: https://github.com/ocurrent/ocaml-dockerfile;
• Issues: https://github.com/ocurrent/ocaml-dockerfile/issues;
• Email: [email protected].

Installation

dockerfile can be installed with opam:

opam install dockerfile
opam install dockerfile-opam
opam install dockerfile-cmd

The dockerfile-opam package includes modules for OPAM- and Linux-specific Dockerfile generation, such as common distributions.

If you don't use opam consult the opam file for build instructions.

ocaml-dockerfile's Issues

Install linux-headers package in Alpine build for bechamel package

The bechamel package CI is failing for Alpine 3.19 as observed in the logs at:
https://ocaml.ci.dev/github/mirage/bechamel/commit/a54a0b47300ed4ae9b95d4b05af701012caa31ca

The fix is to install the linux-header package using apk before proceeding to install the package dependencies.

I see there is dev_packages as an extra argument to module Apk in linux.ml:

 let dev_packages ?extra () =
    install
      "build-base patch tar ca-certificates git rsync curl sudo bash \
       libx11-dev nano coreutils xz ncurses-dev%s"
      (match extra with None -> "" | Some x -> " " ^ x)

See https://github.com/ocurrent/ocaml-dockerfile/blob/master/src-opam/linux.ml#L138

Is there a way to pass these dev_packages list from the respective project sources directory, say from https://github.com/mirage/bechamel?

Linking error OCaml msvc64 4.03 → 4.05

I've disabled the OCaml msvc64 port from OCaml 4.03 to 4.05 included because of a link failure:

#=== ERROR while compiling ocaml-variants.4.05.0+msvc64 =======================#
# context              2.0.8 | win32/x86_64 |  | file://C:/cygwin64/home/opam/opam-repository
# path                 C:/opam/.opam/4.05/.opam-switch/build/ocaml-variants.4.05.0+msvc64
# command              C:\cygwin64\bin\make.exe flexdll world.opt install
# exit-code            2
# env-file             C:/opam/.opam/log/ocaml-variants-2392-c743ac.env
# output-file          C:/opam/.opam/log/ocaml-variants-2392-c743ac.out
### output ###

...

# ../../boot/ocamlrun ../../tools/ocamlmklib -oc unix accept.obj bind.obj channels.obj close.obj close_on.obj connect.obj createprocess.obj dup.obj dup2.obj errmsg.obj getpeername.obj getpid.obj getsockname.obj gettimeofday.obj isatty.obj link.obj listen.obj lockf.obj lseek.obj nonblock.obj mkdir.obj open.obj pipe.obj read.obj readlink.obj rename.obj select.obj sendrecv.obj shutdown.obj sleep.obj socket.obj sockopt.obj startup.obj stat.obj symlink.obj system.obj times.obj unixsupport.obj windir.obj winwait.obj write.obj winlist.obj winworker.obj windbug.obj access.obj addrofstr.obj chdir.obj chmod.obj cst2constr.obj cstringv.obj envir.obj execv.obj execve.obj execvp.obj exit.obj getaddrinfo.obj getcwd.obj gethost.obj gethostname.obj getnameinfo.obj getproto.obj getserv.obj gmtime.obj putenv.obj rmdir.obj socketaddr.obj strofaddr.obj time.obj unlink.obj utimes.obj -ldopt ws2_32.lib -ldopt advapi32.lib
#    Creating library C:\cygwin64\tmp\dyndll_implib83e4b8.lib and object C:\cygwin64\tmp\dyndll_implib83e4b8.exp
# C:\cygwin64\tmp\dyndll_implib83e4b8.lib : fatal error LNK1136: invalid or corrupt file
# ** Fatal error: Error during linking
# 
# make[3]: *** [../Makefile:77: libunix.lib] Error 2
# make[3]: Leaving directory '/cygdrive/c/opam/.opam/4.05/.opam-switch/build/ocaml-variants.4.05.0+msvc64/otherlibs/win32unix'
# make[2]: *** [Makefile:1050: otherlibraries] Error 2
# make[2]: Leaving directory '/cygdrive/c/opam/.opam/4.05/.opam-switch/build/ocaml-variants.4.05.0+msvc64'
# make[1]: *** [Makefile:512: all] Error 2
# make[1]: Leaving directory '/cygdrive/c/opam/.opam/4.05/.opam-switch/build/ocaml-variants.4.05.0+msvc64'
# make: *** [Makefile:531: world.opt] Error 2

The complete log can be found at https://gist.github.com/MisterDA/b1138aaecf1f35a9da37e649a63158b0.

Switch to doas on Alpine

It would be a good testing ground for when ocaml/opam#4991 is fixed

Writeup policy for linux distro support

Currently we support whatever is listed in https://github.com/ocurrent/ocaml-dockerfile/blob/master/src-opam/distro.mli#L93-L169 and is being used by CI applications like ocaml-ci and opam-repo-ci.

However some distributions present challenges like:

• Oracle Linux, not providing packages for many common conf-* packages in opam-repository
• OpenSUSE Tumbleweed, support in opam ocaml/opam#5565

It would be useful to explicitly state what we do support and what we don't (plus why we don't support something).

mount type cache support

TL;DR:

• add ?mounts:(Mount.t list) to Dockerfile.run and support in Dockerfile.crunch
• add a simple Mount.t: mount type, list of options (or string map of options)
• introduce ?mount_cache in dockerfile-opam's package managers (default off), which would mount /var/cache/yum with an appropriately computed cache id (perhaps overridable with ?cache_id)
• provide some helpers for an 'opam' command with appropriate caches mounted (~/.cache for dune, and a symlinked ~/.opam/download-cache/{md5,sha256,sha512} to .cache), usable by both 'opam install' and 'opam monorepo pull' for downstream containers to use
• would this be useful for you in https://github.com/ocurrent/docker-base-images too? Anything in particular I should be aware of to make it useful there?

I've got some very early experimental code to add support for --mount=type=cache on RUN lines that works both with Docker BuildKit and Podman 4.x (see https://docs.docker.com/engine/reference/builder/#run---mounttypecache).

For now it is as a layer on top of dockerfile and dockerfile-opam, but I'd like to contribute at least some of the changes back to this library. In particular Dockerfile.crunch needs to know about the mounts, because the correct way to crunch this:

RUN --mount=type=cache,target=/var/cache sudo yum install -y foo
RUN --mount=type=cache,target=/home/opam/.cache opam install bar

is

RUN --mount=type=cache,target=/var/cache --mount=type=cache,target=/home/opam/.cache sudo yum install -y foo && opam install bar

(i.e. the mounts need to stay grouped together at the beginning)

And once that is in place then the various Package managers in dockerfile-opam could be taught to take a ?use_cache parameter to enable a package download cache (shared among all dockerfiles, not just the current one, with some care to use a proper cache ID per OS/architecture as needed!), and skip the 'clean all' at the end.
Which is beneficial even on fast networks (the package mirrors are sometimes very slow, and especially on CentOS/Fedora just refreshing the mirror/package metadata can take significantly longer than downloading the package).

Also a cache mount type is very useful for downstream container builds that do 'opam install' or 'dune build' or 'opam depext',
or 'opam monorepo pull', all of which can be cached.

(There are other mount types that are useful as well, such as 'tmpfs' but come with various caveats, such as the directory disappearing if you don't consistently specify once you started using it, and 'bind' mounts (as an efficient alternative to copying from another build stage!), but they work slightly differently between Docker and Podman, where the latter requires 'z', and the former doesn't support it)

Caching would be opt-in (who knows what I'd break otherwise).

I'll try to keep the changes minimal, just add the mechanism to support crunching mounts, and the basic mount types that are supported by both podman and docker, and leave the actual management of those caches and cached paths and stages (computing cache ids from OS/arch/etc. checking you don't use overlapping paths) to another library.

However the mount types are likely to evolve (and converge or diverge between podman and docker), and support for that might be best served by the actual application using it, so I'd keep mount types very generic here: (string * string) list or string String.Map.t.

For context: the end goal is a tool that builds development and CI containers for monorepos, but that is a separate project (it sort of "works" on the XAPI project already, but not yet ready for release).

I'm opening this issue to give some background on some PRs that I may open shortly, I'll try to feed the changes in as small chunks.

Proposal: detatch dockerfile-opam to its own repository in the ocurrent org

dockerfile-opam is a fast-changing library needed for ocurrent/docker-base-images.
Given that the rest of the library doesn't change much, and given that the libraries are already split, it seems relatively easy to give dockerfile-opam its own repository in the ocurrent org. This would avoid having to release three packages everytime something has to change (e.g. distributions breaking something, adding new distributions, …) and would make the management of it more centralized and easy to look for.

Feature request: Support for openSUSE Tumbleweed

Hello there,

Currently, it seems this repository supports openSUSE Leap images in many versions:

ocaml-dockerfile/src-opam/distro.ml

Lines 348 to 357 in df1e8f6

	| `OpenSUSE of
	[ `V42_1
	| `V42_2
	| `V42_3
	| `V15_0
	| `V15_1
	| `V15_2
	| `V15_3
	| `V15_4
	| `Latest ]

I would be interested in images for openSUSE Tumbleweed, the rolling release. I would be interested in having it available here (and ultimately in the ocaml/opam images), for CI purposes, just like I like checking things on Debian stable and unstable.

I hope feature requests like this are welcome.
Best,
-- Niols

build error for 3.0.0

#=== ERROR while installing dockerfile.3.0.0 ==================================#
# opam-version         1.2.2 (58ef3b8213100953848d362f7120a30356d7f77d)
# os                   linux
# command              jbuilder build -p dockerfile -j 4
# path                 /home/opam/.opam/4.03.0/build/dockerfile.3.0.0
# compiler             4.03.0
# exit-code            1
# env-file             /home/opam/.opam/4.03.0/build/dockerfile.3.0.0/dockerfile-7-b38251.env
# stdout-file          /home/opam/.opam/4.03.0/build/dockerfile.3.0.0/dockerfile-7-b38251.out
# stderr-file          /home/opam/.opam/4.03.0/build/dockerfile.3.0.0/dockerfile-7-b38251.err
### stderr ###
# Error: The implementation src/dockerfile.pp.ml
# [...]
#        The value `t_of_sexp' is required but not provided
#        The value `sexp_of_t' is required but not provided
#     ocamlopt src/dockerfile.{cmx,o} (exit 2)
# (cd _build/default && /home/opam/.opam/4.03.0/bin/ocamlopt.opt -w -40 -g -I /home/opam/.opam/4.03.0/lib/fmt -I /home/opam/.opam/4.03.0/lib/ocaml -I /home/opam/.opam/4.03.0/lib/result -I /home/opam/.opam/4.03.0/lib/sexplib -I /home/opam/.opam/4.03.0/lib/uchar -no-alias-deps -I src -o src/dockerfile.cmx -c -impl src/dockerfile.pp.ml)
# File "src/dockerfile.ml", line 1:
# Error: The implementation src/dockerfile.pp.ml
#        does not match the interface src/dockerfile.cmi:
#        The value `t_of_sexp' is required but not provided
#        The value `sexp_of_t' is required but not provided

Enabling compression for OCaml 5.1+

Once ocaml/ocaml#12006 is merged, the Dockerfiles for OCaml 5.1+ should install libzstd-dev or equivalent in order to enable compressed compiler artefacts.

Support for "RUN <<EOT" buildkit syntax

https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/reference.md#here-documents

create a Dockerfile parser module

@talex5 points out there are test cases upstream in docker/docker that could be used that have sexp test cases. This would allow more easy bidirectional manipulation of Dockerfiles from datakit-ci (see moby/datakit#440 (comment))

Fedora doesn't use YUM anymore, should migrate to DNF

For many releases DNF is the default package manager, maybe it makes sense to update it from Yum to DNF? It will require dropping some older versions though (seems only Fedora 21 should be removed then).

https://fedoraproject.org/wiki/Releases/22/ChangeSet#Replace_Yum_With_DNF

Default git credentials setup

The Dockerfiles and images generated with this for the ocaml/opam images set up default git credentials:
https://github.com/avsm/ocaml-dockerfile/blob/f184554282a3836bf3f1c34d20e77d0530f8349d/src-opam/dockerfile_linux.ml#L24-L28
Why is this done/desirable?

This creates /home/opam/.gitconfig, which prevents the ocaml/opam images from being used out-of-the-box as devcontainers, because apparently the presence of this file prevents the automatic use of outside git credentials: https://code.visualstudio.com/docs/remote/containers#_sharing-git-credentials-with-your-container.
Thus as a devcontainer base image, inside the container you'd be accidentally committing everything as Docker <[email protected]> without realizing that the usual mechanism of it using outside git credentials hasn't worked. Then one has to rewrite git history to fix the author information of such commits.
Moreover, even without using as part as devcontainer, but for other purposes of doing git commits inside such containers will use those default credentials instead of giving the usual prompt of asking the user to set them up, because they've already been set up by the base image.

Currently I have to work around this by using RUN rm ~/.gitconfig as an extra step on top of these base images. It isn't much, but I fail to see why I'd ever want to commit as some weird default user I haven't explicitly set up myself.

Add support for ARG

https://docs.docker.com/engine/reference/builder/#arg

Presence of spaces in a command is not escaped and results in an incorrect Dockerfile

Example:

from "alpine" @@
run "ls\n/"

results in the incorrect Dockerfile:

FROM alpine
RUN ls
/

instead, each \n characters should be escaped using \ as per https://docs.docker.com/engine/reference/builder/#format

Ubuntu 16.04

Should Ubuntu 16.04 be also added to Deprecated? Its EOL is in 3 days

Originally posted by @kit-ty-kate in ocurrent/docker-base-images#103 (comment)

Inspect winget disable telemetry

This error showed up. Need to fix it.
https://images.ci.ocaml.org/job/2022-09-19/163805-ocluster-build-55d394

Step 30/39 : RUN powershell -Command "winget settings ;         $path=""""${Env:LocalAppData}\Microsoft\WinGet\Settings\settings.json"""" ;         $json=(Get-Content -Encoding ascii $path | Select -SkipLast 1) -Join """"`n"""" ;         $json=($json, '    """"telemetry"""": { """"disable"""": true },', """"}"""") -Join """"`n"""" ;         $json | Set-Content -Encoding ascii -NoNewLine $path ;         winget settings"
 ---> Running in 7db9a76d543a
Get-Content : Cannot find path 'C:\Users\ContainerAdministrator\AppData\Local\M
icrosoft\WinGet\Settings\settings.json' because it does not exist.
At line:1 char:112
+ ... ings.json" ;         $json=(Get-Content -Encoding ascii $path | Selec ...
+                                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\Users\Contai...s\settings.js 
   on:String) [Get-Content], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetCo 
   ntentCommand
 
Removing intermediate container 7db9a76d543a
 ---> 93370368b28a

Use winget to install or modify Visual Studio

Have a look at Use winget to install or modify Visual Studio:

You can use the Windows Package Manager "winget" tool to programmatically install, modify, or update Visual Studio on your machine along with other packages managed by winget.