e4s-project/e4s-alc

E4S à la carte is a tool that allows a user to customize a container image by adding packages to it. These can be system packages and Spack packages.

e4s-alc's Issues

Add `add-repo` functionality

The feature add-repo would allow users to specify a GitHub repository to clone into the image.

Including this feature would include building out the CLI flag --add-repo and the .yaml support for the variable add-repo.

Matrix build

Image I wanted to build trilinos containers for multiple operating systems, with different compilers and different compiler versions. This feature would build out all the Dockerfiles needed.

Given something similar to the following in the .yaml file:

base-image-matrix:
  - rhel8
  - ubuntu
  - centos

spack-compiler-matrix:
  - gcc:
     - 10.2
     - 11.3
     - 12.0

  - clang
     - 9.0.0
     - 12.0.0
     - 16.0.6

  - intel-oneapi-compiler:
     - latest

spack-packages:
  - trilinos

This would produce the directory dockerfiles:

+ dockerfiles:
|-- Dockerfile.rhel8-gcc-10.2-trilinos
|-- Dockerfile.rhel8-gcc-11.3-trilinos
|-- Dockerfile.rhel8-gcc-12.0-trilinos
|-- Dockerfile.rhel8-clang-9.0.0-trilinos
|-- Dockerfile.rhel8-clang-12.0.0-trilinos
|-- Dockerfile.rhel8-clang-16.0.6-trilinos
|-- Dockerfile.rhel8-intel-oneapi-compiler-latest-trilinos
|-- Dockerfile.ubuntu-gcc-10.2-trilinos
|-- Dockerfile.ubuntu-gcc-11.3-trilinos
|-- Dockerfile.ubuntu-gcc-12.0-trilinos
|-- Dockerfile.ubuntu-clang-9.0.0-trilinos
|-- Dockerfile.ubuntu-clang-12.0.0-trilinos
|-- Dockerfile.ubuntu-clang-16.0.6-trilinos
|-- Dockerfile.ubuntu-intel-oneapi-compiler-latest-trilinos
|-- Dockerfile.centos-gcc-10.2-trilinos
|-- Dockerfile.centos-gcc-11.3-trilinos
|-- Dockerfile.centos-gcc-12.0-trilinos
|-- Dockerfile.centos-clang-9.0.0-trilinos
|-- Dockerfile.centos-clang-12.0.0-trilinos
|-- Dockerfile.centos-clang-16.0.6-trilinos
|-- Dockerfile.centos-intel-oneapi-compiler-latest-trilinos

Add json and yaml compatible

For the sake of being compatible with old e4s-alc, a json format would be okay

Add --build flag

The build flag builds the images and the creation of the Dockerfiles.

system-compiler-matrx: add system matrix

I'm still investigating this one.

Add files: download and extract .tgz

add-files:
   - https://tau.uoregon.edu...../file.tgz /opt/file
   - https://tau.uoregon.edu...../file /opt/file

Both of these features should work. Should be easy

Install tarball file from URL in the `-t/--tarball` option

Given the address to a remote tarball, a user should be able to install a tarball into the image without needing the tarball on the host OS.

Here is an example using https://github.com/Exawind/nalu-wind/archive/refs/tags/v1.3.0.tar.gz:

e4s-alc create
   -n new_image
   -i ubuntu:latest
   -t https://github.com/Exawind/nalu-wind/archive/refs/tags/v1.3.0.tar.gz:/opt

Add SUSE Support

This issue will be marked as closed when e4s-alc create -i {SUSE_OS} ..... will pull SUSE images and install the correct packages needed to install spack on SUSE operating systems.

make clean

removes existing build from makefile

Replace modules.yaml file with more appropriate url

Currently, the modules.yaml file is located at: https://www.nic.uoregon.edu/~cfd/e4s-alc/modules.yaml. This should be changed.

Add exception for if image not found

e4s-alc create -n test -i ubu

Returns:

KeyError: 'RepoTags'

Use catch

OS specific packages get disaligned with new OS versions

Example: when pulling the latest ubuntu image and building upon it, the process fails with the following error:

+ apt-get install -y build-essential ca-certificates coreutils curl file environment-modules gfortran git gpg lsb-release vim python3 python3-distutils python3-venv unzip zip cmake
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Package python3-distutils is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'python3-distutils' has no installation candidate
FATAL:   While performing build: while running engine: exit status 100

Building singularity images with software depending on gmake using out of date singularity fails

When building an image that builds a gcc compiler (or any software depending on gmake) with older singularity versions (this issue happens at least up to 3.11.0, but doesn't at 3.11.4) through spack, the process will fail with an error of this type:

==> gcc: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    '/spack/opt/spack/linux-ubuntu22.04-zen3/gcc-11.4.0/gmake-4.4.1-wi55lxbbhsc647bu3c6qsoqf5guyuora/bin/make' '-j16' 'V=1'

1 error found in build log:
     125    checking where to find the target windmc... host tool
     126    checking whether to enable maintainer-specific portions of Makefiles... no
     127    configure: creating ./config.status
     128    config.status: creating Makefile
     129    ==> gcc: Executing phase: 'build'
     130    ==> [2023-09-22-16:28:27.773153] '/spack/opt/spack/linux-ubuntu22.04-zen3/gcc-11.4.0/gmake-4.4.1-wi55lxbbhsc647bu3c6qsoqf5guyuora/bin/make' '-j16' 'V=1'
  >> 131    make: *** cannot open jobserver /tmp/GMfifo279169: No such file or directory.  Stop.

See build log for details:
  /root/.spack/stage/spack-stage-gcc-10.3.0-46nt4agntyurpc5u7wq7bh5odst6scvp/spack-build-out.txt

FATAL:   While performing build: while running engine: exit status 1

This issue can be avoided by updating singularity to it's latest version. It can also be worked around by manually specifying to install the concerned program with the [email protected] dependency, in this case: spack install [email protected] ^[email protected]

Add Podman Support

At the time of writing this, only Docker is supported.

This issue can be marked as closed once e4s-alc init -b podman returns Found Podman! and images are created through the Podman backend.

Creating a specific image version creates two of them with mixed characteristics

When creating a non-latest version image, two images are created: one latest and one of the specified version.
The latest will have all the characteristics passed as parameter (name, installed packages) apart from the specified version.

fdeny@pop-os:~$ e4s-alc list -a
+------+-----+----+---------+------+
| Name | Tag | Id | Created | Size |
+------+-----+----+---------+------+
+------+-----+----+---------+------+

fdeny@pop-os:~$ e4s-alc create -i "ubuntu:18.04" -n dockerpulledubuntu -a neovim -p zlib
[...]

fdeny@pop-os:~$ e4s-alc list -a
+--------------------+--------+--------------+----------------------+------------+
|        Name        |  Tag   |      Id      |       Created        |    Size    |
+--------------------+--------+--------------+----------------------+------------+
| dockerpulledubuntu | latest | d72e18c0e611 | 04/18/2023, 16:30:27 | 599.66 MiB |
|       ubuntu       | 18.04  | 3941d3b032a8 | 03/08/2023, 03:22:44 | 60.22 MiB  |
+--------------------+--------+--------------+----------------------+------------+

fdeny@pop-os:~$ docker run -it ubuntu:latest
root@80a9f359facf:/# which nvim
/usr/bin/nvim
root@80a9f359facf:/# spack install zlib
[+] /spack/opt/spack/linux-ubuntu18.04-haswell/gcc-7.5.0/zlib-1.2.13-y4hravoqkccnt4gezmdmb3y6ybucdqst

fdeny@pop-os:~$ docker run -it ubuntu:18.04
root@0d79f2ca3050:/# which nvim
root@0d79f2ca3050:/# spack install zlib
bash: spack: command not found

Singularity definition file support

This feature would allow the user to produce singularity definition files rather than Dockerfiles.