kata-containers/tests

Kata Containers tests, CI, and metrics

Kata Containers

Welcome to Kata Containers!

This repository is the home of the Kata Containers code for the 2.0 and newer releases.

If you want to learn about Kata Containers, visit the main Kata Containers website.

Introduction

Kata Containers is an open source project and community working to build a standard implementation of lightweight Virtual Machines (VMs) that feel and perform like containers, but provide the workload isolation and security advantages of VMs.

License

The code is licensed under the Apache 2.0 license. See the license file for further details.

Platform support

Kata Containers currently runs on 64-bit systems supporting the following technologies:

Architecture	Virtualization technology
x86_64, amd64	Intel VT-x, AMD SVM
aarch64 ("arm64")	ARM Hyp
ppc64le	IBM Power
s390x	IBM Z & LinuxONE SIE

Hardware requirements

The Kata Containers runtime provides a command to determine if your host system is capable of running and creating a Kata Container:

$ kata-runtime check

Notes:

• This command runs a number of checks including connecting to the network to determine if a newer release of Kata Containers is available on GitHub. If you do not wish this to check to run, add the --no-network-checks option.

• By default, only a brief success / failure message is printed. If more details are needed, the --verbose flag can be used to display the list of all the checks performed.

• If the command is run as the root user additional checks are run (including checking if another incompatible hypervisor is running). When running as root, network checks are automatically disabled.

Getting started

See the installation documentation.

Documentation

See the official documentation including:

• Installation guides
• Developer guide
• Design documents
  • Architecture overview
  • Architecture 3.0 overview

Configuration

Kata Containers uses a single configuration file which contains a number of sections for various parts of the Kata Containers system including the runtime, the agent and the hypervisor.

Hypervisors

See the hypervisors document and the Hypervisor specific configuration details.

Community

To learn more about the project, its community and governance, see the community repository. This is the first place to go if you wish to contribute to the project.

Getting help

See the community section for ways to contact us.

Raising issues

Please raise an issue in this repository.

Note: If you are reporting a security issue, please follow the vulnerability reporting process

Developers

See the developer guide.

Components

Main components

The table below lists the core parts of the project:

Component	Type	Description
runtime	core	Main component run by a container manager and providing a containerd shimv2 runtime implementation.
runtime-rs	core	The Rust version runtime.
agent	core	Management process running inside the virtual machine / POD that sets up the container environment.
dragonball	core	An optional built-in VMM brings out-of-the-box Kata Containers experience with optimizations on container workloads
documentation	documentation	Documentation common to all components (such as design and install documentation).
tests	tests	Excludes unit tests which live with the main code.

Additional components

The table below lists the remaining parts of the project:

Component	Type	Description
packaging	infrastructure	Scripts and metadata for producing packaged binaries (components, hypervisors, kernel and rootfs).
kernel	kernel	Linux kernel used by the hypervisor to boot the guest image. Patches are stored here.
osbuilder	infrastructure	Tool to create "mini O/S" rootfs and initrd images and kernel for the hypervisor.
kata-debug	infrastructure	Utility tool to gather Kata Containers debug information from Kubernetes clusters.
agent-ctl	utility	Tool that provides low-level access for testing the agent.
kata-ctl	utility	Tool that provides advanced commands and debug facilities.
trace-forwarder	utility	Agent tracing helper.
runk	utility	Standard OCI container runtime based on the agent.
ci	CI	Continuous Integration configuration files and scripts.
ocp-ci	CI	Continuous Integration configuration for the OpenShift pipelines.
katacontainers.io	Source for the katacontainers.io site.
Webhook	utility	Example of a simple admission controller webhook to annotate pods with the Kata runtime class

Packaging and releases

Kata Containers is now available natively for most distributions.

General tests

See the tests documentation.

Metrics tests

See the metrics documentation.

Glossary of Terms

See the glossary of terms related to Kata Containers.

tests's Issues

manage_ctr_mgr.sh assumes amd64

Once #71 lands, we need to update https://github.com/kata-containers/tests/blob/master/cmd/container-manager/manage_ctr_mgr.sh so that it doesn't assume amd64 when setting up the Ubuntu Docker repo.

I think we could just add in some logic that does:

goarch=$(go env GOARCH)

... and make use of that as it's the correct value (whereas arch(1) isn't).

Add docker terminal and top tests

Update checkcommits for gometalinter

gometalinter is showing a few issues for checkcommits.

Sync checkcommits from Clear Containers repo

Once clearcontainers/tests#861 lands, sync the checkcommits code in this repo.

Ignore linter warnings on auto-generated code

protobuf-generated code cannot be changed by us so ignore linters warnings for it.

Add docker integration test for restart, privileges and search

Add docker integration tests for docker restart, docker search and docker run with privileges flag.

add contribution guidelines

include minimal contribution guidelines

Add docker integration tests for attach, cp, create and commit

These will test docker attach, docker cp, docker create and docker commit functionality.

Docker integration tests for export, diff, env and exit code.

This will test docker integration functionality for export, diff, env and exit code.

Needs 2 approval to be able to merge a pull request

It would be better to rely on 2 approvals to consider a PR as mergeable.

Add developer variable for lenient linters

The static checks script currently only runs a subset of linters under the CI, but all linters by default otherwise.

Add a KATA_DEV_MODE variable that, if set, will run the same subset. This is useful and crucially does not require the developer to pretend to be running under a CI (which could be dangerous).

add docker integration test for volume, search and tag

Add docker integration tests for docker volume, docker search and docker tag.

Provide ability to skip a test based on the runtime being used

Currently, some of the tests make use of Skip() (ginkgo). At some future point they will also be using skip (bats) too. These calls skip the tests unconditionally. However, we are likely to need ways to skip a test but only for particular runtime(s) [1]:

Something like:

if kata.Runtime("cc") {
    Skip("Issue: ...")
}

We should also support any:

if kata.Runtime("any") {
    Skip("Issue: ...")
}

... and maybe a regex like:

if kata.Runtime("cc|runv") {
    Skip("Issue: ...")
}

We'll need kata.Runtime(name string) to check to see which runtime the kata-runtime symlink actually points to. If the link name ends in -cc [2], it's a Clear Containers-based runtime, else runv.

We can then compare the fully resolved value of the Runtime path with the runtime type the test is requesting be skipped and return true/false as appropriate.

---

[1] - runtime names should be specified using the same syntax as used for the build. Namely cc or runv (or all). See kata-containers/runtime#16.
[2] - Or, run the runtime and check for a known string such as:

$ kata-runtime --help | grep -q clearcontainers

Apply review changes to README

PR #32 got inadvertently merged (oops!) before the doc teams review could be applied.

checkcommits should allow fixes :

Currently checkcommits does not allow for whitespace after Fixes and before :

It should allow this syntax too.

Fixes : #< PR-number >

Add a README

This repo needs a README.md to explain various things including KATA_DEV_MODE.

add yum and swupd to the package_manager tests

We currently have a test that checks apt and dnf inside a container. We need to add yum and maybe swupd from CL

Add extra linters

Add the following golang linters to the static checks tool for more stringent checks:

• staticcheck
• structcheck
• unused

Add docker run with stdin

Add a docker integration test where stdin can be handled

Add teardown script

Add teardown script to collect logs in case a failure occurs.

Create central static checks test

Add a script that will be the single source of all static tests used to build components.

Initially, it will simply run checkcommits, but will be extended later to run linters, etc.

All other repos should invoke this script to avoid a proliferation of (different) static check scripts.

add functional test for run command

Add functional test for run command

Add initial docker integration tests

Add docker run tests for the kata-containers runtimes.

Skip failing tests that have opened issues

To have a stable CI, lets skip the failing tests that already have opened issues. Once those issues get resolved, we should remove the skip from the tests.

Don't run unit tests as root in dev mode

.ci/go-test.sh should not run tests as root if KATA_DEV_MODE is set.

Clean up CI scripts

shellcheck(1) is reporting a few issues for the CI scripts.

tests: Add docker integration build test

This test the docker build command.

Add linters to static checks

https://github.com/kata-containers/tests/blob/master/.ci/static-checks.sh needs to call the various golang linters to sanity check the code submissions.

Currently, the only kata repo using linters is the throttler:

• https://github.com/kata-containers/ksm-throttler/blob/master/.ci/go-lint.sh

I plan to remove that file entirely and add it into https://github.com/kata-containers/tests/blob/master/.ci/static-checks.sh, so that it will be invoked for all the kata repos.

Does that sound good to you @sameo, @gnawux, @grahamwhaley?

Add a fedora CI setup script

Add a .ci/setup_env_fedora.sh script.

See: #71 (comment)

Can we check that stdout/stderr produce correct output

Relating to kata-containers/agent#125 and kata-containers/agent#124 - can we add a test that checks that what we get over stdout and stderr are in fact correct and whole.
@jodh-intel has suggested https://github.com/jamesodhunt/utfout (oh, look, it is written by @jodh-intel ;-) ) might be a good tool to use for this.
Other simpler thoughts from me are to:

• just check we get the right size we expect to start with
• we could do an md5sum compare to what we expect

/cc @chavafg

CI: Refer to contributing doc if checkcommits fails

If checkcommits fails, point the user at the appropriate section of the central contributing guide.

Really disable checking vendor code with linters

gometalinter is not ignoring vendor code since --vendor seems to not work as advertised.

Fix checkcommits under TravisCI

TravisCI provides a TRAVIS_COMMIT variable which checkcommits uses to determine the newest commit in the PR branch. However, the value of the variable is not reliable
(see travis-ci/travis-ci#7830), so checkcommits shouldn't use it.

CI: fix proxy log selector in teardown script

.ci/teardown.sh currently tries to select on the proxy systemd unit if not asked to copy logs. However, the proxy is no longer a system service so this is invalid.

Don't run gofmt with go tip

Change the static checks script to disable running gofmt when testing with go tip.

Fix static checks script

The static checks script is not written in a way that allows calls to use it correctly.

Add docker integration tests for logs, pause and port

Add integration tests for:

docker logs
docker pause / unpause
docker port

Enable checkcommits

Enable checkcommits which will requires that all PRs have:

• a Signed-off-by: line.
• a Fixes #XXX comment where XXX is a github issue number.
• Requires all commits to have a "sub-system" followed by a colon, followed by a brief description.
• Requires a blank line after the subsystem.
• Requires all commits to have a fuller description after the sub-system that will contain the Fixes ### and Signed-off-by: comments.

Add jenkins_job_build.sh script

jenkins_job_build.sh script should be called in the Jenkins Jobs for all repos. This will minimize duplicated code in the jenkins job configuration

Enable '-race' flag depending on arch.

This issue is to add -race flag conditionally to go_test_flag.

checkcommits: Improve error message prefix

The errors generated by checkcommits now display the internal commit objects, but they should just display the commit hash (as they used to before #37 landed).

ci: Add installation scripts

Install runtime, shim, agent, proxy, qemu, and all dependencies to run kata containers

Don't call gometalinter unless the calling project contains go code

Currently, .ci/static-checks.sh unconditionally calls gometalinter. This is actually fine since gometalinter is clever enough to detect if there are any go packages to check and it will exit cleanly if not.

However, it's pointless installing gometalinter for those calling repositories that don't contain golang code, so add a check to speed up the CI in those scenarios.

Create a central go test script

Following the pattern of previous PRs, we can create a central go test script that can be used by all the repositories.

static-checks should ignore submodules

The static checks script currently cannot be called in the runtime repo as it tries to check the submodules (which it shouldn't).

Add a log parsing tool

Create a tool that can manipulate the structured logs the system components generate.

Initially the tool should be capable of accepting any number of log files which will then be:

• Parsed, sorted by timestamp and displayed (in a variety of formats) along with time deltas (showing the time between log calls).
• Checked for bogus/invalid log entries (or bogus/invalid log data).

Along with the above, I plan to add a check at the end of a CI run which will call this tool. If the tool fails, the PR will be failed. The rationale here being that all component logs should:

• encode a common set of fields.
• always be fully machine parseable.

Replaces: clearcontainers/tests#590

Require doc team sign-off for docs

We should require an "ack" from the documentation team before merging any doc changes on PRs.

CI: Ignore vendor/ directories when running go test

go list in golang 1.9 is clever and prunes vendor packages by default:

$ (GOROOT=/usr/local/go/go1.9/go PATH=$GOROOT/bin:$PATH go list ./...)
github.com/kata-containers/proxy

However, we're using golang 1.8 with Travis and the behaviour is different:

$ (GOROOT=/usr/local/go/go1.8.1/go PATH=$GOROOT/bin:$PATH go list ./...)
github.com/kata-containers/proxy
github.com/kata-containers/proxy/vendor/github.com/davecgh/go-spew/spew
github.com/kata-containers/proxy/vendor/github.com/hashicorp/yamux
github.com/kata-containers/proxy/vendor/github.com/pmezard/go-difflib/difflib
github.com/kata-containers/proxy/vendor/github.com/sirupsen/logrus
github.com/kata-containers/proxy/vendor/github.com/sirupsen/logrus/hooks/syslog
github.com/kata-containers/proxy/vendor/github.com/stretchr/testify/assert
github.com/kata-containers/proxy/vendor/golang.org/x/crypto/ssh
github.com/kata-containers/proxy/vendor/golang.org/x/crypto/ssh/terminal
github.com/kata-containers/proxy/vendor/golang.org/x/sys/unix
github.com/kata-containers/proxy/vendor/golang.org/x/sys/windows

This behaviour is causing .ci/go-test.sh to try to run tests on vendor packages, which we don't want.

We could update all the .travis.yml files to only use versions of go >= 1.9 but we specify go 1.8 as our minimum version. Hence, simply filter vendor packages out of go list before running go test.

Add docker integration test for info, inspect, and load

Add docker load, inspect, info test.

Enable tests for this repo

Currently, .travis.yml only runs the static tests on PRs raised for this repo.

As soon as possible, we need to add the following to .travis.yml (plus docker setup) so all PRs for this repo will trigger all tests to run:

script:
  - ".ci/go-test.sh"