osinside/kiwi-functional-tests

KIWI openQA functional tests based on the integration tests in OBS

kiwi-functional-tests

KIWI openQA functional tests using the images from the staging project on OBS.

Prerequisites

You need access to an openQA instance, where either the job groups, test suites & products have been configured or where you have operator access so that you can create these yourself. For testing purposes, you can setup a local instance as described in the documentation.

Furthermore, you need to have a valid account in openSUSE's instance of the Open Build Service and osc set up.

Last but not least, you'll need poetry to utilize the scripts shipped in this repository.

Before proceeding further, be sure to install the dependencies and run poetry install.

Setup on openQA

You can apply the necessary settings automatically using the settings.py script as follows:

poetry run ./settings.py --server $URL_TO_MY_INSTANCE

If you are running a local instance, then you will have to add the domain opensuse.org to the download_domains option in /etc/openqa/openqa.ini:

[global]
download_domains = opensuse.org

Schedule a test run

Test runs are scheduled using the script schedule_test_run.py. It will by default schedule all tests for all distributions & versions on o3 and create a state file which contains the started job ids:

$ poetry run ./schedule_test_run.py
Wrote build state into kiwi_build_20210430_1619795290_483278.pickle

The created .pickle file can be fed into the monitoring script monitor.py to query the current state of the jobs:

$ poetry run ./monitor.py -p kiwi_build_20210430_1619795290_483278.pickle
https://openqa.opensuse.org/tests/876: state: passed, result: done
https://openqa.opensuse.org/tests/877: state: passed, result: done
https://openqa.opensuse.org/tests/886: state: failed, result: done
https://openqa.opensuse.org/tests/920: state: failed, result: done
https://openqa.opensuse.org/tests/921: state: skipped, result: cancelled
https://openqa.opensuse.org/tests/922: state: failed, result: done

or to cancel them all:

$ poetry run ./monitor.py -c kiwi_build_20210430_1619795290_483278.pickle

kiwi-functional-tests's Issues

Add initial test

First steps to use the openQA system

Test failures on openqa.opensuse.org

• all download of disk images are currently failing on o3: https://progress.opensuse.org/issues/94997
• kiwi-test-image-disk-simple.x86_64-1.42.1-Build11.19.vmdk fails to boot: https://openqa.opensuse.org/tests/2044724, https://openqa.opensuse.org/tests/2044725 (EFI & non-EFI)
• kiwi-test-image-luks.x86_64-1.15.1-Build11.19.raw: fails to boot but not only on o3, this is a known issue. https://openqa.opensuse.org/tests/2044726, https://openqa.opensuse.org/tests/2044727
• kiwi-test-image-lvm.x86_64-1.42.1-Build12.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044728, https://openqa.opensuse.org/tests/2044729
• kiwi-test-image-overlayroot.x86_64-1.42.1-Build12.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044732, https://openqa.opensuse.org/tests/2044733
• kiwi-test-image-suse-on-dnf.x86_64-1.42.1-Build11.19.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044734, https://openqa.opensuse.org/tests/2044735
• kiwi-test-image-disk-simple.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044784, https://openqa.opensuse.org/tests/2044785
• kiwi-test-image-lvm.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044788, https://openqa.opensuse.org/tests/2044789
• kiwi-test-image-overlayroot.x86_64-1.15.3-Build11.7.vmdk fails to boot in EFI and non-EFI mode: https://openqa.opensuse.org/tests/2044790, https://openqa.opensuse.org/tests/2044791

All items on that list (except the busted LUKS image) are vmdk images, which leads me to believe that something is wrong with vmdk on o3, whereas it works on my box. @okurz @foursixnine @asmorodskyi any ideas?

Implement job launcher

To run the openqa tests we need a simple job launcher that can do the following in a first version:

1. lookup current integration test images file names and build a map
2. create openqa job taking all or a given set of images into account

create a KIWI job group in openqa

To group our integration tests into a namespace it would be good to have a job group in openqa.
Such a group can only be created by an openqa admin. Dan Cermak will be able to create the group once we
reached a stable state with the tests

how to handle EFI/UEFI boot process in openqa

For testing we require that the qemu VM is called in EFI/UEFI mode. From a qemu perspective this means:

qemu-kvm -bios /usr/share/qemu/ovmf-x86_64-ms.bin ...

How can we setup the use of a custom firmware when running in openqa ?

Failing images

I am getting closer to finally getting #3 into decent shape, but there's still a ton of failures. For some of these I am not sure whether these are caused by the image being broken or openQA not launching it properly or something else.

Tumbleweed

• https://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/tumbleweed/images/iso/kiwi-test-image-disk-ramdisk.x86_64-1.42.1-Build9.26.install.iso fails to create a bootable disk image

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-orthos: raw.xz disk image does not display anything in UEFI mode

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-overlayroot: broken, should be fixed by OSInside/kiwi#1864

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-qcow-openstack: does not launch in UEFI mode

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-disk-legacy, https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-disk-ramdisk and https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-custom-partitions do not install properly when booted from the .install.iso: they just reboot again and again until openQA kills the tests. This happens also in a VM created by virt-manager and both in UEFI and non UEFI mode.

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-MicroOS: this is a weird one. When using the installation iso in UEFI mode, then the resulting disk image will display the installation on the second reboot as well! But this does not happen in non UEFI mode… See for the video recording: https://user-images.githubusercontent.com/45594031/123279000-bf8b5900-d507-11eb-8c7b-294f45f7d611.mp4

Leap

• https://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/leap/images/iso/kiwi-test-image-disk-ramdisk.x86_64-1.15.3-Build9.7.install.iso fails to create a bootable disk image
• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:leap/test-image-disk-ramdisk: .install.iso hangs in UEFI and non UEFI mode

Fedora 33

• https://build.opensuse.org/package/show/Virtualization:Appliances:Images:Testing_x86:fedora/test-image-microdnf: displays a non fatal grub error in UEFI and non-UEFI mode:
  

openqa failed to work with .raw.xz images

For still unknown reasons the .raw.xz images from the integration tests do not work when used in a job for openqa. Interesting enough the .qcow2 and also vmdk.xz images works which leads to the assumption it could be a problem in openqa itself.

Dan suggested to get in contact with Oliver to clarify on that end

Additional testing ideas

Currently we perform only very basic tests and do not really check the image's functionality or additional properties.

This is a collection of the various images that we currently produce in the staging area and what their capabilities/properties are:

test-image-MicroOS
  builds a MicroOS image, this comes with a semi read-only
  rootfs which could be tested

test-image-azure
  build image suitable to run in Microsoft Azure,
  there is imho nothing you can test other than booting
  has vhd-fixed disk type in qemu

test-image-bundle-format
  build an image with a custom image name, the test
  can check if the image file name matches the given
  bundle format: https://osinside.github.io/kiwi/image_types_and_results.html?highlight=bundle_format#image-bundle-format

test-image-custom-partitions
  build an image including two custom scripts to change the
  partition table for including custom changes. The image exists
  to demo how extra scripts can be used to modify the disk layout.
  There is not much to test other than the boot. With the new
  feature of the <partitions> element users don't need to
  create custom scripts anymore, but the test is still there

test-image-disk
  build an image with the resize feature enabled, a test could
  check if the disk geometry was really resized

test-image-disk-legacy
  build an image using the legacy oemboot code. This means the
  image builds a custom initrd not based on dracut. There is not
  much to test other than the boot

test-image-disk-ramdisk
  build an image to be deployed onto a ramdisk. The image also builds
  an install.iso image which can be used to dump the system onto a
  ramdisk. A real test would call this .install.iso and see if the
  later system completely runs from a ramdisk

test-image-disk-simple
  build a simple disk image, no resize feature

test-image-docker
  build a docker image

test-image-docker-derived
  build a docker image derived from another image

test-image-ec2
  build an image for the Amazon EC2 cloud, there is nothing you
  can test outside of the cloud

test-image-gce
  build an image for the Google cloud, there is nothing you
  can test outside of the cloud

test-image-live
  build a live system. A test can boot and see if it's writable
  everywhere to check if the overlay technology works as expected

test-image-luks
  build a luks encrypted system. This also includes /boot to be
  encrypted. Boot test would be enough

test-image-lvm
  build an image with LVM volumes, a test can check if they exist
  as expected

test-image-orthos
  build an image for the SUSE orthos system. I think that one should
  be deleted

test-image-overlayroot
  build a disk image using kiwi's overlayroot technology. The root
  partition gets overlayed via overlayfs and all writes are targeted
  into a dedicated write partition on the same disk. A test can
  check the layout and the write capabilities. With overlayroot you
  can build relatively small systems because all of the root is turn
  into a squashfs. It'a also a real immutable system but no transactional
  updates

test-image-partitions-and-volumes
  testing the <partitions> feature, a test can check if the partitions
  matches the setup from the description

test-image-pxe
  build a pxe image for the old netboot concept. Testing is only
  possible in this PXE environment

test-image-qcow-openstack
  build a qcow2 image

test-image-raid
  build a raid image. All kiwi raid images produces raids in degraded
  mode. This is needed because you have no access to other raid disks
  at image build time. A test can check the state of the raid and could
  also attach a disk and see if the raid turns into a non degraded raid

test-image-suse-on-dnf
  build a suse image with dnf

test-image-tbz
  build a tarball from the root

test-image-vagrant
  build a vagrant box, testing requires to run vagrant

test-image-wsl
  build a WSL (Windows Subsystem for Linux) container image. Testing
  can only be done on Windows