The OS-autoinst project aims at providing a means to run fully automated tests. Especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily and safely be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS.
os-autoinst can be executed alone, but is currently designed to be executed together with openQA, the web user interface that allows to run more than one os-autoinst instance at the same time.
More information on os-autoinst and openQA can be found on http://os-autoinst.github.io/openQA/
Under openSUSE the os-autoinst
package can be installed from the official
repository or from our
devel repository.
For further details, have a look at the
openQA documentation.
For building os-autoinst manually checkout the build instructions below.
The main executable isotovideo
can read test parameters from the command
line or read test parameters from a file named vars.json
. This file stores
the values of the different variables that will configure the behavior of the
test execution.
A container is provided and can be pulled and the main execution can be called
in one step, for example using the podman
container engine for tests defined
in the current directory on x86_64 if your environment supports KVM
virtualization acceleration:
podman run --rm -it -v .:/tests registry.opensuse.org/devel/openqa/containers/isotovideo:qemu-kvm casedir=/tests
Use the image variant ending with qemu-x86
on x86_64 if no KVM support is
available.
Take a look on https://registry.opensuse.org/cgi-bin/cooverview?srch_term=project%3Ddevel%3AopenQA for all available container images.
Additional test variables can be supplied on the command line. There are some variables used by os-autoinst itself and other that are used by the tests. A minimal command line can look like this:
isotovideo distri=opensuse casedir=/full/path/for/tests iso=/full/path/for/iso
As alternative or completementary a corresponding vars.json
with additional
parameters could be:
{
"DISTRI" : "opensuse",
"CASEDIR" : "/full/path/for/tests",
"NAME" : "test-name",
"ISO" : "/full/path/for/iso",
"VNC" : "91",
"BACKEND" : "qemu",
"DESKTOP" : "kde"
}
Be advised that the file vars.json
is also modified by os-autoinst
so make
sure to backup handcrafted versions of this file.
For more concrete instructions read on in the "How to run test cases" section below. Find sections about "How to contribute" or "Build instructions" further below.
This following instructions shows how to run test cases. First one needs to clone the test distribution. Checkout os-autoinst-distri-example for an example of a minimal test distribution.
Example for openSUSE’s tests:
mkdir distri && cd distri git clone [email protected]:os-autoinst/os-autoinst-distri-opensuse.git opensuse cd opensuse/products/opensuse git clone [email protected]:os-autoinst/os-autoinst-needles-opensuse.git needles
Example for openQA’s self-tests ("openQA-in-openQA" test):
mkdir distri && cd distri git clone [email protected]:os-autoinst/os-autoinst-distri-openQA.git openqa cd openqa git clone [email protected]:os-autoinst/os-autoinst-needles-openQA.git needles
Then create a working directory for the test execution, e.g.:
mkdir /tmp/os-autoinst-run && cd /tmp/os-autoinst-run
Create a minimal vars.json
config file within that directory, e.g.:
{
"ARCH" : "x86_64",
"BACKEND" : "qemu",
"CASEDIR" : "/path/to/os-autoinst-distri-opensuse",
"DESKTOP" : "gnome",
"DISTRI" : "opensuse",
"ISO" : "/path/to/openSUSE-Tumbleweed-DVD-x86_64-Snapshot20160715-Media.iso",
"PRODUCTDIR" : "/path/to/os-autoinst-distri-opensuse/products/opensuse",
"VNC" : 90,
}
You will need to correct the file paths to point to real locations. Some of the variables you can use are listed here. Test case specific variables are listed in the distri directories e.g. os-autoinst-distri-opensuse/variables.
Then you can run the isotovideo
script within the created working directory. When doing
a manual build, that script can be found at the top-level of the os-autoinst Git checkout.
When using the QEMU backend it is possible to access the system under test via VNC:
vncviewer localhost:91 -ViewOnly -Shared
Run isotovideo
with the environment variable RUN_VNCVIEWER
set to autostart a VNC viewer
on the right port.
Run isotovideo
with the environment variable RUN_DEBUGVIEWER
to start the internal debug
screenshot viewer updated with an always recent screenshot of the test run.
Individual test modules are written with one test module per file using the test API in Perl code. Experimental support for test modules in the Python programming language is provided.
Find more details about how to write tests on http://open.qa/docs/#_how_to_write_tests
If you want to contribute to this project, please clone and send pull requests via https://github.com/os-autoinst/os-autoinst.
More information on the contribution can be found on http://os-autoinst.github.io/openQA/contact/, too.
Issues are tracked on https://progress.opensuse.org/projects/openqav3/.
For an overview of the architecture, see doc/architecture.md.
-
Every commit is checked by our CI system as soon as you create a pull request but you should run the os-autoinst tests locally. Checkout the build instructions for further details.
-
For git commit messages use the rules stated on How to Write a Git Commit Message as a reference
-
Every pull request is reviewed in a peer review to give feedback on possible implications and how we can help each other to improve
If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal.
On openSUSE one can install the package
os-autoinst-devel
which
pulls all dependencies.
The required dependencies are also declared in dependencies.yaml
. (The names listed
within that file are specific to openSUSE but can be easily transferred to other
distributions.)
Simply call
make
in the top folder which automatically creates a build directory and builds the complete project.
Call
make help
to list all available targets.
The above commands use a convenience Makefile calling cmake
. For packaging,
when using an IDE or to conduct the steps manually it is suggested to use
CMake directly and do the following: Create a build directory outside of the
source directory. The following commands need to be invoked within that
directory.
Configure build:
cmake $path_to_os_autoinst_checkout
You can specify any of the standard CMake variables, e.g. -DCMAKE_BUILD_TYPE=Debug
and -DCMAKE_INSTALL_PREFIX=/custom/install/prefix
.
The following examples assume that GNU Make is used. It is possible to generate for
a different build tool by adding e.g. -G Ninja
to the CMake arguments.
Build executables and libraries:
make symlinks
This target also creates symlinks of the built executables and libraries within the
source directory so isotovideo
can find them.
Run all tests:
make check
By default CTest is invoked in verbose mode because prove already provides condensed
output. Add -DVERBOSE_CTEST=OFF
to the CMake arguments to avoid that.
Run all Perl tests (*.t
files found within the t
and xt
directories):
make test-perl-testsuite
Run individual tests by specifying them explicitly:
make test-perl-testsuite TESTS="t/15-logging.t t/28-signalblocker.t"
Notice that the user needs to include the test directory for each test (either t for normal or xt for developer-centric tests) when specifying individual tests.
Add additional arguments to the prove
invocation, e.g. enable verbose output:
make test-perl-testsuite PROVE_ARGS=-v
Gather coverage data while running tests:
make test-perl-testsuite WITH_COVER_OPTIONS=1
Generate a coverage report from the gathered coverage data:
make coverage
If no coverage data has been gathered so far the coverage
target will invoke the
testsuite automatically.
Reset gathered coverage data:
make coverage-reset
Install files for packaging:
make install DESTDIR=…
Further notes:
-
When using the
test-perl-testsuite
target,ctest
is not used (and thereforectest
specific tweaks have no effect). -
One can always run Perl tests manually via
prove
after the build has been conducted withmake symlinks
. Note that some tests need to be invoked within thet
directory. An invocation likeprove -vI.. -I../external/os-autoinst-common/lib 28-signalblocker.t
is supposed to work. -
It is also possible to run
ctest
within the build directory directly instead of using the mentioned targets. -
All mentioned variables to influence the test execution (
TESTS
,WITH_COVER_OPTIONS
, …) can be combined and can also be used with thecoverage
target.
When using the QEMU backend, also ensure your user running os-autoinst has access
to /dev/kvm
.
modprobe kvm-intel || modprobe kvm-amd chgrp kvm /dev/kvm ; chmod g+rw /dev/kvm # maybe redundant # optionally use a new user; just to keep things separate useradd -m USERNAME -G kvm passwd USERNAME # and/or add ~USERNAME/.ssh/authorized_keys