suse/bci-tests

This repository contains the tests for the SUSE Base Container Images

bci-tests's Issues

Add testing with Dapper

@mvarlese has tested a dapper build of his 'fleet' fork https://github.com/mvarlese/fleet , leveraging our BCI image for golang.

We should do that in CI.

Create tests for the git image

The git container is mostly untested, we only have a very basic smoke test: tests/test_git.py

Potential tests:

• clone a repo over http
• clone a repo over ssh

Fix all CVEs, or document why they don't matter in our case

It's good that we are scanning for CVEs, because any ISVs would do the same.
Yet, if our scans don't look good, we should explain why.

This card is intended to fix or document why some CVE are listed by the scanners we are using in this project.

Test container logos

Some of the BCI container have a container logo specified by an annotation. In case it is set , we should be testing that

• The logo url is fetchable
• the logo is an image (common format supported by browsers)
• the size is roughly quadratic
• the sizes are larger than 64 and smaller than 512px

Update gh actions with dependabot

Test container metadata and cryptographic signature

We should ensure that the containers are properly signed, and their metadata is accurate.

Slow CI

The CI has become slow, but it is weird, the same test runs for 10 min one day and 1 hour the other day. We might be hitting a rate limit or similar.

We need to identify what causes our CI to be slow sometimes and if possible find a solution.

BCI test failure in Ruby 2.5 on 15-SP5

Since switching Ruby 2.5 from 15-SP4 to 15-SP5, on s390x and ppc64le the sqlite3 test fails (see log below). On aarch64 and x86_64

Referenced test failures:

• aarch64 (passing)
• s390x (failing)
• ppc64le (failing)
• x86_64 (passing)

Attached logs:

• bci-ruby25_sle15sp5.txt

See also https://progress.opensuse.org/issues/130901

Add security scanner periodic testing

For our images to look good, we need to ensure the security tests don't report a very bad vulnerability (and ensure those scanners work at all times).

We should add a periodic test in this repo to ensure things are green and okay.

Improve/refactor multistage tests

There are multiple issues with the multistage test module:

• it does not forward the marks from containers and thus will always run all tests
• the amidst project is dead upstream and will eventually stop working (probably), we should find a replacement
• k3sup is not working on non-x86_64 and upstream is not willing to merge the 2 line fix: alexellis/k3sup#345
• the adventureworks test is relying on the eol .Net 5

And we really are atm only testing openjdk 11, go on x86_64 and .Net 5.0. We should extend the test coverage here.

run tests on SLE Base Container (using SLE BCI repository)

In addition to openSUSE BCI image, the testsuite should also be running on SLE Base Image, which is now setup with SLE_BCI repository. No SCC enablement is needed.

Increase test coverage for go, node, java, and python

• node (#5)
• node frontend (i.e. with xvfb)
• go
• java
• python

Create a dedicated FIPS testsuite

Currently we are only testing fips for the base image, but we should at least test also busybox and micro, but preferably all images.

The shrinkwrap option is deprecated

npm ERR! The `shrinkwrap` option is deprecated, and can not be set in this way
npm ERR!     Use the --package-lock setting instead.

Failing test:
https://openqa.suse.de/tests/10700996#step/_root_BCI-tests_node_docker/3

Please add a license

Add testing of BCI system/init container

We have a container with systemd, we should test it.

Add generics to go 1.18 container

Since generics are introduced in go 1.18 we should add a simple test run to test generics there (and there only).

Need to ensure size of containers to match embedded use cases

There are ISVs which are looking for runtime containers which size don't go above 50MB (without software). We should ensure that runtime containers never exceed that size.

Add test to ensure BCI images don't ship systemd

There is no point to have systemd in images OUTSIDE the "system" container. We should ensure its absence for all the rest of the containers.

Need to ensure size of the developer image isn't too big

Golang images currently are weighting close to a GB. We need to make sure this compares well with alpine, ubuntu, and RH equivalents.

Add testing with 'buildpacks.io'

Some ISVs mentioned the fact that they are building with Dockerfiles (like tested in this repo).

Yet, they were interested to see if there is tooling that would make their life easier to maintain containers in their lifecycle.

I think the 'pack' CLI could help those users, and should be tested.

Add test to validate the BCI systemd container image

The BCI systemd container image shall be able to bootstrap docker via "service docker start" and start a container image.

Extend MariaDB Container Entrypoint coverage

We pull the mariadb entrypoint script from the canonical image on dockerhub. That entrypoint script also has support for some kind of healthcheck (that we do not test). It would would be great to extend the test coverage to be able to test: SUSE/BCI-dockerfile-generator#904

Run CI pipelines only for affected changes

We are running the full pipeline list of tests for each new commit/PR. Sometimes, the changes are done in specific test environment and we could limit those runs to only run that environment. This way, we could save time and unnecessary resources, and specially ci failures that have nothing to do with the change under review.

15-SP3 init container fails on RHEL 7.9 - container doesn't start

The tests are failing only on RHEL 7.9:
https://openqa.suse.de/tests/9005682#step/all/1
with

E       AssertionError: Unexpected exit code 126 for CommandResult(command=b'uname -s', exit_status=126, stdout=None, stderr=b'Error: cannot exec into container that is not running: container state improper\n')
E       assert 126 in [0, 1]
E        +  where 126 = CommandResult(command=b'uname -s', exit_status=126, stdout=None, stderr=b'Error: cannot exec into container that is not running: container state improper\n').rc

Environment:

localhost:~/BCI-tests #cat /etc/os-release 
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.9"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.9
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION=7.9
# This is a "SLES Expanded Support platform release 7.9"
# The above "Red Hat Enterprise Linux Server" string is only used to
# keep software compatibility.

localhost:~/BCI-tests #podman -v
podman version 1.6.4

localhost:~/BCI-tests #podman info
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.8-1.el7.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.8, commit: 52580ecb98528a431a28ed597cb453c716bcd70d'
  Distribution:
    distribution: '"rhel"'
    version: "7.9"
  MemFree: 519049216
  MemTotal: 4142096384
  OCIRuntime:
    name: runc
    package: runc-1.0.0-69.rc10.el7_9.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 2
  eventlogger: journald
  hostname: localhost.localdomain
  kernel: 3.10.0-1160.66.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 19m 26.71s
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 8
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

I have tried to run it manually, and the container doesn't keep running as it should be expected:

localhost:~/BCI-tests #podman --log-level=debug run -d registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-init:15.3 
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  runc map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-init:15.3" 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@80c2354d9be93b2f9d54a056b427c8227d786bfbb9cb651b67c1db902c72179d" 
DEBU[0000] exporting opaque data as blob "sha256:80c2354d9be93b2f9d54a056b427c8227d786bfbb9cb651b67c1db902c72179d" 
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Using bridge netmode                         
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 2 for container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage]@80c2354d9be93b2f9d54a056b427c8227d786bfbb9cb651b67c1db902c72179d" 
DEBU[0000] exporting opaque data as blob "sha256:80c2354d9be93b2f9d54a056b427c8227d786bfbb9cb651b67c1db902c72179d" 
DEBU[0000] created container "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" 
DEBU[0000] container "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" has work directory "/var/lib/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata" 
DEBU[0000] container "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" has run directory "/var/run/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata" 
DEBU[0000] New container created "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" 
DEBU[0000] container "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" has CgroupParent "machine.slice/libpod-43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708.scope" 
DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/HH5JEWDIZCPGL73P74ZETBUG4T:/var/lib/containers/storage/overlay/l/ZTIQHNZ57B4OQ25ISM2GCLWIBA,upperdir=/var/lib/containers/storage/overlay/914c899a7100f5e5081d440db4586d6e6e4da82d2beebb8555921fa1e5823b3b/diff,workdir=/var/lib/containers/storage/overlay/914c899a7100f5e5081d440db4586d6e6e4da82d2beebb8555921fa1e5823b3b/work,context="system_u:object_r:svirt_sandbox_file_t:s0:c267,c680" 
DEBU[0000] mounted container "43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708" at "/var/lib/containers/storage/overlay/914c899a7100f5e5081d440db4586d6e6e4da82d2beebb8555921fa1e5823b3b/merged" 
DEBU[0000] Created root filesystem for container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 at /var/lib/containers/storage/overlay/914c899a7100f5e5081d440db4586d6e6e4da82d2beebb8555921fa1e5823b3b/merged 
DEBU[0000] Made network namespace at /var/run/netns/cni-12365a23-e6a6-7215-dc00-a1c9de96976d for container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 
INFO[0000] Got pod network &{Name:goofy_easley Namespace:goofy_easley ID:43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 NetNS:/var/run/netns/cni-12365a23-e6a6-7215-dc00-a1c9de96976d Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network cni-loopback (type=loopback) 
INFO[0000] Got pod network &{Name:goofy_easley Namespace:goofy_easley ID:43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 NetNS:/var/run/netns/cni-12365a23-e6a6-7215-dc00-a1c9de96976d Networks:[] RuntimeConfig:map[podman:{IP: PortMappings:[] Bandwidth:<nil> IpRanges:[]}]} 
INFO[0000] About to add CNI network podman (type=bridge) 
DEBU[0000] [0] CNI result: Interfaces:[{Name:cni-podman0 Mac:92:a1:85:99:60:cb Sandbox:} {Name:veth22f31dac Mac:9a:e6:64:56:a7:fe Sandbox:} {Name:eth0 Mac:1a:ff:bf:3d:97:0d Sandbox:/var/run/netns/cni-12365a23-e6a6-7215-dc00-a1c9de96976d}], IP:[{Version:4 Interface:0xc00014fac8 Address:{IP:10.88.0.9 Mask:ffff0000} Gateway:10.88.0.1}], Routes:[{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}], DNS:{Nameservers:[] Domain: Search:[] Options:[]} 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 to machine.slice:libpod:43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] reading hooks from /etc/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 at /var/lib/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 -u 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata -p /var/run/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level debug --syslog --conmon-pidfile /var/run/containers/storage/overlay-containers/43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708]"
INFO[0000] Running conmon under slice machine.slice and unitName libpod-conmon-43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708.scope 
DEBU[0000] Received: 16797                              
INFO[0000] Got Conmon PID as 16785                      
DEBU[0000] Created container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 in OCI runtime 
DEBU[0000] Starting container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 with command [/usr/lib/systemd/systemd] 
DEBU[0000] Started container 43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708 
43b89d7f70a858c139b6d70912969fdacfe2034acd9d785786a5cc8f6343c708

Seems ok, but looking at the existing containers:

localhost:~/BCI-tests #podman ps --all
CONTAINER ID  IMAGE                                                                       COMMAND               CREATED         STATUS                       PORTS  NAMES
43b89d7f70a8  registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-init:15.3  /usr/lib/systemd/...  5 seconds ago   Exited (255) 5 seconds ago          goofy_easley

podman logs showing:

localhost:~/BCI-tests #podman --log-level=debug logs 43b89d7f70a8
DEBU[0000] Reading configuration file "/usr/share/containers/libpod.conf" 
DEBU[0000] Merged system config "/usr/share/containers/libpod.conf": &{{false false false false false true} 0 {   [] [] []}  docker://  runc map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] [crun runc] [crun] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] systemd   /var/run/libpod -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman []   k8s.gcr.io/pause:3.1 /pause false false  2048 shm    false} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/lib/containers/storage 
DEBU[0000] Using run root /var/run/containers/storage   
DEBU[0000] Using static dir /var/lib/containers/storage/libpod 
DEBU[0000] Using tmp dir /var/run/libpod                
DEBU[0000] Using volume path /var/lib/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] cached value indicated that overlay is supported 
DEBU[0000] cached value indicated that metacopy is not being used 
DEBU[0000] cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument 
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist 

Remove mentions of m8a

It's more appropriate to rename m8a to BCI now that we are moving away from the m8a name.

This needs we need renaming imports, etc...

IMO: Not a priority. Apparently it now is, it's not the first time I hear it :D

First failures in PostgreSQL tests

1. RuntimeError: Container xyz did not become healthy within 45000.0ms
   After looking at all the tests, I have observed that it needs between 45s and 50s on x86_64, but for safety I would go to 60s timeout.
   e.g. https://openqa.suse.de/tests/10880049

2. SyntaxError: invalid syntax
   In other architectures != x86_64 the tox command fails with:

wa/psycopg2_ee724a19d7ac48d894cf733efbd3685e/
  Complete output (6 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-fwlvwwwa/psycopg2_ee724a19d7ac48d894cf733efbd3685e/setup.py", line 225
      except Warning, w:
                    ^
  SyntaxError: invalid syntax
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/2d/d7/496da11d7c81971870ddd36800419c4f84e8f6208aac5eabedf9f7748729/psycopg2-2.0.11.tar.gz#sha256=e6b4e0e41df97441eff34e00065376414da6488e0d55848a45cd77551dbae434 (from https://pypi.org/simple/psycopg2/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached psycopg2-2.0.10.tar.gz (255 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /root/BCI-tests/.tox/postgres/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-fwlvwwwa/psycopg2_f8a4bace0e584dc1a9ae18dfe83aa00f/setup.py'"'"'; __file__='"'"'/tmp/pip-install-fwlvwwwa/psycopg2_f8a4bace0e584dc1a9ae18dfe83aa00f/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-mh15ro96
       cwd: /tmp/pip-install-fwlvwwwa/psycopg2_f8a4bace0e584dc1a9ae18dfe83aa00f/
  Complete output (5 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-install-fwlvwwwa/psycopg2_f8a4bace0e584dc1a9ae18dfe83aa00f/setup.py", line 50, in <module>
      import ConfigParser
  ModuleNotFoundError: No module named 'ConfigParser'
  ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/19/79/35c7596bab4456f3610c12ec542a94d51c6781ced587d1d85127210b879b/psycopg2-2.0.10.tar.gz#sha256=e40cc04b43849085725076ae134bfef9e3b087f6dd7c964aeeb930e2f0bc14ab (from https://pypi.org/simple/psycopg2/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement psycopg2 (from versions: 2.0.10, 2.0.11, 2.0.12, 2.0.13, 2.0.14, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.3.2, 2.4, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.5, 2.5.1, 2.5.2, 2.5.3, 2.5.4, 2.5.5, 2.6, 2.6.1, 2.6.2, 2.7, 2.7.1, 2.7.2, 2.7.3, 2.7.3.1, 2.7.3.2, 2.7.4, 2.7.5, 2.7.6, 2.7.6.1, 2.7.7, 2.8, 2.8.1, 2.8.2, 2.8.3, 2.8.4, 2.8.5, 2.8.6, 2.9, 2.9.1, 2.9.2, 2.9.3, 2.9.4, 2.9.5, 2.9.6)
ERROR: No matching distribution found for psycopg2

=================================== log end ====================================
ERROR: could not install deps [pytest, pytest-testinfra, pytest-xdist ; python_version >= "3.6", dataclasses ; python_version < "3.7", pytest-rerunfailures, typing_extensions, git+https://github.com/dcermak/pytest_container, psycopg2]; v = InvocationError('/root/BCI-tests/.tox/postgres/bin/python -m pip install pytest pytest-testinfra \'pytest-xdist ; python_version >= "3.6"\' \'dataclasses ; python_version < "3.7"\' pytest-rerunfailures typing_extensions git+https://github.com/dcermak/pytest_container psycopg2', 1)
___________________________________ summary ____________________________________
ERROR:   postgres: could not install deps [pytest, pytest-testinfra, pytest-xdist ; python_version >= "3.6", dataclasses ; python_version < "3.7", pytest-rerunfailures, typing_extensions, git+https://github.com/dcermak/pytest_container, psycopg2]; v = InvocationError('/root/BCI-tests/.tox/postgres/bin/python -m pip install pytest pytest-testinfra \'pytest-xdist ; python_version >= "3.6"\' \'dataclasses ; python_version < "3.7"\' pytest-rerunfailures typing_extensions git+https://github.com/dcermak/pytest_container psycopg2', 1)

reference: https://openqa.suse.de/tests/10879980#step/bci_test_podman/45

Write test for distribution-image

We need a new test run for our distribution-image (See distribution - it's basically a container registry).

I'm thinking of a simple push and pull test run, so we would need to create a container, and in one test function push a container to the registry, which is pulled in another test run. Or we do this in one function.

Related ticket: https://progress.opensuse.org/issues/120810

Add logs about which images are being pulled

It would be very useful to include the tested/pulled container images in the logs.

Current logs, like bci_test-result.xml, only contain info about the results with the name of the container, e.g.:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite errors="0" failures="0" hostname="install" name="minimal" skipped="1" tests="6" time="10.771" timestamp="2022-05-02T09:57:48.688025">
    <testcase classname="tests.test_minimal" name="test_minimal_image_size[container1-size1]" time="3.766"/>
    <testcase classname="tests.test_minimal" name="test_minimal_image_size[container0-size0]" time="3.802">
      <skipped message="Temporary size increase due to mozilla cert bundle hack" type="pytest.xfail"/>
    </testcase>
    <testcase classname="tests.test_minimal" name="test_fat_packages_absent[auto_container1]" time="3.145"/>
    <testcase classname="tests.test_minimal" name="test_fat_packages_absent[auto_container0]" time="3.379"/>
    <testcase classname="tests.test_minimal" name="test_rpm_absent_in_micro[container0]" time="2.698"/>
    <testcase classname="tests.test_minimal" name="test_rpm_present_in_micro[container0]" time="2.656"/>
  </testsuite>
</testsuites>

Maybe we could inject it here:

<testcase classname="tests.test_minimal" name="test_fat_packages_absent[auto_container1]" image="registry.suse.de/xyz" time="3.145"/>

or just create a log message saying "pulling image registry.suse.de/xyz"..

Add Code of Conduct

Add blackduck security scanner

We are already testing with security scanners, but some ISVs mentioned the fact that they are using blackduck. It might be a good idea to test our deliverable with that security scanner too.

See also: https://github.com/blackducksoftware/synopsys-detect

Add performance testing

@dirkmueller anything in particular that you'd like to check?

Test container READMEs

Container have to have a pointer to a README.md. we should be checking for that.

Ideally we have a way to fetch the reference and validate its structure with a markdown linter. Spell checking for common errors could be a stretch goal but isn't required

Use random port binding for creating 389ds container

Since we are using always the same port when launching a 390ds container using
extra_launch_args=["-p", "3389:3389"],
https://github.com/SUSE/BCI-tests/blob/main/bci_tester/data.py#L318
The tests that are using this container fail with

Bind for 0.0.0.0:3389 failed: port is already allocated.

It would be better to create some random number for the host side.

A ugly solution would be

port = random.randint(3000, 4000) # or whatever range that fits better
extra_launch_args=["-p", port+":3389"],

with proper syntax, ofc...

Restructure tests for multi-staged builds (if necessary)

Currently the tests are in the root folder. They are easily discoverable, but it's gonna be a mess if we do different kinds of testing.

I am suggesting here to create multiple folders based on test cases:

• tests/single/test_<language>.py can take care of the simple builds, for those not doing multi-staged builds
• tests/multistaged/test_<language>.py to test multi staged builds
• tests/pack/test_<buildpack>.py to ensure paketo/buildpacks.io are working as expected with our base images.

For this, we will have to:

• rewire tox to point to the right tests
• refactor some tests: extract them from tests_ and split them into the right folders
• Create multi staged containers in obs
• Modify data structures in matryoshka_tester/data.py to add the newly built containers. If data.py become more convoluted, we might want to add convenience datastructures. We need to adapt existing tests to new structure.

Tox missing setup.py error

Since 3a04ab7#diff-ef2cef9f88b4fe09ca3082140e67f5ad34fb65fb6e228f119d3812261ae51449L5 removed skipsdist = True from tox.ini, getting this:

# tox -e init 
Traceback (most recent call last):
  File "/usr/bin/tox", line 11, in <module>
    load_entry_point('tox==2.9.1', 'console_scripts', 'tox')()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 40, in main
    retcode = Session(config).runcommand()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 392, in runcommand
    return self.subcommand_test()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 552, in subcommand_test
    path = self.get_installpkg_path()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 528, in get_installpkg_path
    path = self._makesdist()
  File "/usr/lib/python3.6/site-packages/tox/session.py", line 407, in _makesdist
    raise tox.exception.MissingFile(setup)
tox.MissingFile: MissingFile: /root/BCI-tests/setup.py

Add Dockerfile tests

See also multi staged issue.

Some golang tests get stuck after reruns

A bunch of tests in internal openQA running go tests get stuck and timeout.
https://openqa.suse.de/tests/8731168#step/bci_test/25
https://openqa.suse.de/tests/8731347#step/bci_test/25
https://openqa.suse.de/tests/8731787#step/bci_test/25
https://openqa.suse.de/tests/8731789#step/bci_test/25
https://openqa.suse.de/tests/8731793#step/bci_test/25

Apparently, it only happens using docker.

I have tried to reproduce the issue, but it's randomly happening.

So, I have done:

export TOX_PARALLEL_NO_SPINNER=1
export CONTAINER_RUNTIME=docker
export OS_VERSION=15.3 
export TARGET=ibs-cr 

tox -e go -- -n auto -k bci/golang_1.17 --reruns 3 --reruns-delay 10
go installed: attrs==21.4.0,dataclasses==0.8,execnet==1.9.0,filelock==3.4.1,importlib-metadata==4.8.3,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.9,pytest==7.0.1,pytest-container @ git+https://github.com/dcermak/pytest_container@e3381ba64e0ce4ce268e21c68d886dcd7bc0560f,pytest-forked==1.4.0,pytest-rerunfailures==10.2,pytest-testinfra==6.7.0,pytest-xdist==2.5.0,tomli==1.2.3,typing_extensions==4.1.1,zipp==3.6.0
go run-test-pre: PYTHONHASHSEED='700224569'
go run-test: commands[0] | pytest -vv tests/test_go.py --junitxml=/root/BCI-tests/junit_go.xml -n auto -k bci/golang_1.17 --reruns 3 --reruns-delay 10
=========================================================================================================================== test session starts ============================================================================================================================
platform linux -- Python 3.6.12, pytest-7.0.1, pluggy-1.0.0 -- /root/BCI-tests/.tox/go/bin/python
cachedir: .tox/go/.pytest_cache
rootdir: /root/BCI-tests, configfile: pyproject.toml
plugins: testinfra-6.7.0, forked-1.4.0, xdist-2.5.0, rerunfailures-10.2, container-0.0.2
[gw0] linux Python 3.6.12 cwd: /root/BCI-tests
[gw1] linux Python 3.6.12 cwd: /root/BCI-tests
[gw0] Python 3.6.12 (default, Nov 25 2020, 20:33:10) [GCC]
[gw1] Python 3.6.12 (default, Nov 25 2020, 20:33:10) [GCC]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/test_go.py::test_base_PATH_present[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-bci/bci-base:15.3 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base:15.3] 
tests/test_go.py::test_go_size[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
[gw1] [ 20%] PASSED tests/test_go.py::test_go_size[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
tests/test_go.py::test_build_kured[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-kured] 
[gw0] [ 40%] PASSED tests/test_go.py::test_base_PATH_present[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-bci/bci-base:15.3 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base:15.3] 
tests/test_go.py::test_go_version[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
[gw0] [ 60%] PASSED tests/test_go.py::test_go_version[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
[gw1] [ 80%] RERUN tests/test_go.py::test_build_kured[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-kured] 
tests/test_go.py::test_build_kured[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-kured] 

And it gets stuck here.

On another console I ca see:

localhost:~ # docker images
REPOSITORY                                                              TAG                 IMAGE ID            CREATED             SIZE
<none>                                                                  <none>              c6d7611af7f0        10 minutes ago      992MB
<none>                                                                  <none>              0f773570f87d        10 minutes ago      923MB
registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang     1.17                9b58c2d6f790        13 hours ago        986MB
registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang     1.16                d6701832443b        16 hours ago        917MB
registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base   15.3                9752b59e1b92        16 hours ago        116MB

localhost:~ # docker ps --all
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
179f9cdc7d57        9b58c2d6f790        "/bin/bash"         About a minute ago   Up About a minute                       sweet_davinci

but docker logs sweet_davinci doesn't return any output.

In this case it did a re-run of test_build_kured. But after a few tries, it stucks somewhere else, normally always at 80% of execution. Another example:

tox -e go -- -n auto -k bci/golang_1.17 --reruns 3 --reruns-delay 10 --pytest-container-log-level DEBUG
go installed: attrs==21.4.0,dataclasses==0.8,execnet==1.9.0,filelock==3.4.1,importlib-metadata==4.8.3,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pyparsing==3.0.9,pytest==7.0.1,pytest-container @ git+https://github.com/dcermak/pytest_container@e3381ba64e0ce4ce268e21c68d886dcd7bc0560f,pytest-forked==1.4.0,pytest-rerunfailures==10.2,pytest-testinfra==6.7.0,pytest-xdist==2.5.0,tomli==1.2.3,typing_extensions==4.1.1,zipp==3.6.0
go run-test-pre: PYTHONHASHSEED='3998865324'
go run-test: commands[0] | pytest -vv tests/test_go.py --junitxml=/root/BCI-tests/junit_go.xml -n auto -k bci/golang_1.17 --reruns 3 --reruns-delay 10 --pytest-container-log-level DEBUG
=========================================================================================================================== test session starts ============================================================================================================================
platform linux -- Python 3.6.12, pytest-7.0.1, pluggy-1.0.0 -- /root/BCI-tests/.tox/go/bin/python
cachedir: .tox/go/.pytest_cache
rootdir: /root/BCI-tests, configfile: pyproject.toml
plugins: testinfra-6.7.0, forked-1.4.0, xdist-2.5.0, rerunfailures-10.2, container-0.0.2
[gw0] linux Python 3.6.12 cwd: /root/BCI-tests
[gw1] linux Python 3.6.12 cwd: /root/BCI-tests
[gw0] Python 3.6.12 (default, Nov 25 2020, 20:33:10) [GCC]
[gw1] Python 3.6.12 (default, Nov 25 2020, 20:33:10) [GCC]
gw0 [5] / gw1 [5]
scheduling tests via LoadScheduling

tests/test_go.py::test_base_PATH_present[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-bci/bci-base:15.3 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base:15.3] 
tests/test_go.py::test_go_size[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
[gw1] [ 20%] PASSED tests/test_go.py::test_go_size[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
tests/test_go.py::test_build_kured[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-kured] 
[gw0] [ 40%] RERUN tests/test_go.py::test_base_PATH_present[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-bci/bci-base:15.3 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base:15.3] 
tests/test_go.py::test_base_PATH_present[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-bci/bci-base:15.3 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/bci-base:15.3] 
[gw1] [ 60%] PASSED tests/test_go.py::test_build_kured[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17-kured] 
tests/test_go.py::test_go_get_binary_in_path[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 
[gw1] [ 80%] PASSED tests/test_go.py::test_go_get_binary_in_path[bci/golang:1.17 from registry.suse.de/suse/sle-15-sp3/update/cr/totest/images/bci/golang:1.17] 

Switch alpine:latest to some other image otuside docker.io in the distribution tests

    """run registry container with attached volume '/var/lib/docker-registry'"""
    engine = container_runtime.runner_binary
    host_port = auto_container_per_test.forwarded_ports[0].host_port
    content = """
    FROM alpine:latest
    CMD ["echo", "container from my local registry"]
    """

https://github.com/SUSE/BCI-tests/blob/main/tests/test_distribution.py#L19

This is pulling alpine from docker.io, which we know has some pull limits. Therefore, many tests in openQA are failing with the message:

Error: creating build container: initializing source docker://alpine:latest: reading manifest latest in docker.io/library/alpine: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Maybe we could simply use registry.suse.com/bci/bci-micro:latest ...

Reduce parallelization inside each language test

If a language is using an installer which can't run in parallel, we shouldn't try testing things in parallel. For that, we might want to reduce the scope of the pytest containers fixture to function instead of module.

However, that will have a negative impact on other languages' tests. To fix that, we could think of passing the scope to the fixture based on the python module (=language test file).