kata-containers/tests

Kata Containers tests, CI, and metrics

tests's Issues

Enable tests for this repo

Currently, .travis.yml only runs the static tests on PRs raised for this repo.

As soon as possible, we need to add the following to .travis.yml (plus docker setup) so all PRs for this repo will trigger all tests to run:

script:
  - ".ci/go-test.sh"

Add teardown script

Add teardown script to collect logs in case a failure occurs.

Don't run unit tests as root in dev mode

.ci/go-test.sh should not run tests as root if KATA_DEV_MODE is set.

Create a central go test script

Following the pattern of previous PRs, we can create a central go test script that can be used by all the repositories.

Enable '-race' flag depending on arch.

This issue is to add -race flag conditionally to go_test_flag.

Require doc team sign-off for docs

We should require an "ack" from the documentation team before merging any doc changes on PRs.

Add developer variable for lenient linters

The static checks script currently only runs a subset of linters under the CI, but all linters by default otherwise.

Add a KATA_DEV_MODE variable that, if set, will run the same subset. This is useful and crucially does not require the developer to pretend to be running under a CI (which could be dangerous).

Provide ability to skip a test based on the runtime being used

Currently, some of the tests make use of Skip() (ginkgo). At some future point they will also be using skip (bats) too. These calls skip the tests unconditionally. However, we are likely to need ways to skip a test but only for particular runtime(s) [1]:

Something like:

if kata.Runtime("cc") {
    Skip("Issue: ...")
}

We should also support any:

if kata.Runtime("any") {
    Skip("Issue: ...")
}

... and maybe a regex like:

if kata.Runtime("cc|runv") {
    Skip("Issue: ...")
}

We'll need kata.Runtime(name string) to check to see which runtime the kata-runtime symlink actually points to. If the link name ends in -cc [2], it's a Clear Containers-based runtime, else runv.

We can then compare the fully resolved value of the Runtime path with the runtime type the test is requesting be skipped and return true/false as appropriate.

---

[1] - runtime names should be specified using the same syntax as used for the build. Namely cc or runv (or all). See kata-containers/runtime#16.
[2] - Or, run the runtime and check for a known string such as:

$ kata-runtime --help | grep -q clearcontainers

add docker integration test for volume, search and tag

Add docker integration tests for docker volume, docker search and docker tag.

Add extra linters

Add the following golang linters to the static checks tool for more stringent checks:

• staticcheck
• structcheck
• unused

static-checks should ignore submodules

The static checks script currently cannot be called in the runtime repo as it tries to check the submodules (which it shouldn't).

Add a README

This repo needs a README.md to explain various things including KATA_DEV_MODE.

Add a fedora CI setup script

Add a .ci/setup_env_fedora.sh script.

See: #71 (comment)

Clean up CI scripts

shellcheck(1) is reporting a few issues for the CI scripts.

Add docker integration tests for logs, pause and port

Add integration tests for:

docker logs
docker pause / unpause
docker port

Needs 2 approval to be able to merge a pull request

It would be better to rely on 2 approvals to consider a PR as mergeable.

Add initial docker integration tests

Add docker run tests for the kata-containers runtimes.

add functional test for run command

Add functional test for run command

Add docker integration test for info, inspect, and load

Add docker load, inspect, info test.

Fix static checks script

The static checks script is not written in a way that allows calls to use it correctly.

Add docker run with stdin

Add a docker integration test where stdin can be handled

Really disable checking vendor code with linters

gometalinter is not ignoring vendor code since --vendor seems to not work as advertised.

Add docker terminal and top tests

Add docker integration test for restart, privileges and search

Add docker integration tests for docker restart, docker search and docker run with privileges flag.

Skip failing tests that have opened issues

To have a stable CI, lets skip the failing tests that already have opened issues. Once those issues get resolved, we should remove the skip from the tests.

Add linters to static checks

https://github.com/kata-containers/tests/blob/master/.ci/static-checks.sh needs to call the various golang linters to sanity check the code submissions.

Currently, the only kata repo using linters is the throttler:

• https://github.com/kata-containers/ksm-throttler/blob/master/.ci/go-lint.sh

I plan to remove that file entirely and add it into https://github.com/kata-containers/tests/blob/master/.ci/static-checks.sh, so that it will be invoked for all the kata repos.

Does that sound good to you @sameo, @gnawux, @grahamwhaley?

checkcommits: Improve error message prefix

The errors generated by checkcommits now display the internal commit objects, but they should just display the commit hash (as they used to before #37 landed).

Apply review changes to README

PR #32 got inadvertently merged (oops!) before the doc teams review could be applied.

Create central static checks test

Add a script that will be the single source of all static tests used to build components.

Initially, it will simply run checkcommits, but will be extended later to run linters, etc.

All other repos should invoke this script to avoid a proliferation of (different) static check scripts.

Don't call gometalinter unless the calling project contains go code

Currently, .ci/static-checks.sh unconditionally calls gometalinter. This is actually fine since gometalinter is clever enough to detect if there are any go packages to check and it will exit cleanly if not.

However, it's pointless installing gometalinter for those calling repositories that don't contain golang code, so add a check to speed up the CI in those scenarios.

Add docker integration tests for attach, cp, create and commit

These will test docker attach, docker cp, docker create and docker commit functionality.

Don't run gofmt with go tip

Change the static checks script to disable running gofmt when testing with go tip.

checkcommits should allow fixes :

Currently checkcommits does not allow for whitespace after Fixes and before :

It should allow this syntax too.

Fixes : #< PR-number >

tests: Add docker integration build test

This test the docker build command.

Ignore linter warnings on auto-generated code

protobuf-generated code cannot be changed by us so ignore linters warnings for it.

add yum and swupd to the package_manager tests

We currently have a test that checks apt and dnf inside a container. We need to add yum and maybe swupd from CL

manage_ctr_mgr.sh assumes amd64

Once #71 lands, we need to update https://github.com/kata-containers/tests/blob/master/cmd/container-manager/manage_ctr_mgr.sh so that it doesn't assume amd64 when setting up the Ubuntu Docker repo.

I think we could just add in some logic that does:

goarch=$(go env GOARCH)

... and make use of that as it's the correct value (whereas arch(1) isn't).

Enable checkcommits

Enable checkcommits which will requires that all PRs have:

• a Signed-off-by: line.
• a Fixes #XXX comment where XXX is a github issue number.
• Requires all commits to have a "sub-system" followed by a colon, followed by a brief description.
• Requires a blank line after the subsystem.
• Requires all commits to have a fuller description after the sub-system that will contain the Fixes ### and Signed-off-by: comments.

Sync checkcommits from Clear Containers repo

Once clearcontainers/tests#861 lands, sync the checkcommits code in this repo.

Add jenkins_job_build.sh script

jenkins_job_build.sh script should be called in the Jenkins Jobs for all repos. This will minimize duplicated code in the jenkins job configuration

Can we check that stdout/stderr produce correct output

Relating to kata-containers/agent#125 and kata-containers/agent#124 - can we add a test that checks that what we get over stdout and stderr are in fact correct and whole.
@jodh-intel has suggested https://github.com/jamesodhunt/utfout (oh, look, it is written by @jodh-intel ;-) ) might be a good tool to use for this.
Other simpler thoughts from me are to:

• just check we get the right size we expect to start with
• we could do an md5sum compare to what we expect

/cc @chavafg

add contribution guidelines

include minimal contribution guidelines

CI: Ignore vendor/ directories when running go test

go list in golang 1.9 is clever and prunes vendor packages by default:

$ (GOROOT=/usr/local/go/go1.9/go PATH=$GOROOT/bin:$PATH go list ./...)
github.com/kata-containers/proxy

However, we're using golang 1.8 with Travis and the behaviour is different:

$ (GOROOT=/usr/local/go/go1.8.1/go PATH=$GOROOT/bin:$PATH go list ./...)
github.com/kata-containers/proxy
github.com/kata-containers/proxy/vendor/github.com/davecgh/go-spew/spew
github.com/kata-containers/proxy/vendor/github.com/hashicorp/yamux
github.com/kata-containers/proxy/vendor/github.com/pmezard/go-difflib/difflib
github.com/kata-containers/proxy/vendor/github.com/sirupsen/logrus
github.com/kata-containers/proxy/vendor/github.com/sirupsen/logrus/hooks/syslog
github.com/kata-containers/proxy/vendor/github.com/stretchr/testify/assert
github.com/kata-containers/proxy/vendor/golang.org/x/crypto/ssh
github.com/kata-containers/proxy/vendor/golang.org/x/crypto/ssh/terminal
github.com/kata-containers/proxy/vendor/golang.org/x/sys/unix
github.com/kata-containers/proxy/vendor/golang.org/x/sys/windows

This behaviour is causing .ci/go-test.sh to try to run tests on vendor packages, which we don't want.

We could update all the .travis.yml files to only use versions of go >= 1.9 but we specify go 1.8 as our minimum version. Hence, simply filter vendor packages out of go list before running go test.

Add a log parsing tool

Create a tool that can manipulate the structured logs the system components generate.

Initially the tool should be capable of accepting any number of log files which will then be:

• Parsed, sorted by timestamp and displayed (in a variety of formats) along with time deltas (showing the time between log calls).
• Checked for bogus/invalid log entries (or bogus/invalid log data).

Along with the above, I plan to add a check at the end of a CI run which will call this tool. If the tool fails, the PR will be failed. The rationale here being that all component logs should:

• encode a common set of fields.
• always be fully machine parseable.

Replaces: clearcontainers/tests#590

CI: fix proxy log selector in teardown script

.ci/teardown.sh currently tries to select on the proxy systemd unit if not asked to copy logs. However, the proxy is no longer a system service so this is invalid.

Fix checkcommits under TravisCI

TravisCI provides a TRAVIS_COMMIT variable which checkcommits uses to determine the newest commit in the PR branch. However, the value of the variable is not reliable
(see travis-ci/travis-ci#7830), so checkcommits shouldn't use it.

Docker integration tests for export, diff, env and exit code.

This will test docker integration functionality for export, diff, env and exit code.

ci: Add installation scripts

Install runtime, shim, agent, proxy, qemu, and all dependencies to run kata containers

CI: Refer to contributing doc if checkcommits fails

If checkcommits fails, point the user at the appropriate section of the central contributing guide.

Update checkcommits for gometalinter

gometalinter is showing a few issues for checkcommits.