huawei-openlab/oct

Testing cases and framework for OpenContainers project

oct's Issues

Status of runtime test of runc

root@szxbzci0039:~/gopath/src/github.com/huawei-openlab/oct# ./ocitest -r runc -o err-only 
INFO[0001] Failure Details:                             
INFO[0001] =============================================================================================== 
INFO[0001] 
BundleName:
  rlimit14
BundleDir:
  bundles/rlimit14
CaseArgs:
  --args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_SIGPENDING:1024:1024
Result:
  FAILED
ErrInfo:
  time="2015-12-01T10:08:18+08:00" level=fatal msg="Container start failed: Wrong rlimit value: RLIMIT_SIGPENDING" 
exit status 1

INFO[0001] 
BundleName:
  rlimit1
BundleDir:
  bundles/rlimit1
CaseArgs:
  --args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_AS:1024:1024
Result:
  FAILED
ErrInfo:
  exit status 139

INFO[0001] 
BundleName:
  rlimit15
BundleDir:
  bundles/rlimit15
CaseArgs:
  --args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_STACK:1024:1024
Result:
  FAILED
ErrInfo:
  exit status 139

INFO[0001] 
BundleName:
  platform1
BundleDir:
  bundles/platform1
CaseArgs:
  --args=./runtimetest --args=vpl --rootfs=rootfs --os=windows
Result:
  FAILED
ErrInfo:
  time="2015-12-01T02:08:18Z" level=fatal msg="Validation failed: \"OS expected: windows, actual: linux\"" 
exit status 1

INFO[0001] 
BundleName:
  rlimit10
BundleDir:
  bundles/rlimit10
CaseArgs:
  --args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_NPPOC:1024:1024
Result:
  FAILED
ErrInfo:
  time="2015-12-01T10:08:18+08:00" level=fatal msg="Container start failed: Wrong rlimit value: RLIMIT_NPPOC" 
exit status 1

INFO[0001] =============================================================================================== 
INFO[0001] Statistics:  25 bundles success, 5 bundles failed

It shows, rlimit are not support very well in runc, such as, RLIMIT_SIGPENDING, RLIMIT_AS, RLIMIT_STACK, RLIMIT_NPPOC

About the satisfaction test of Open Container Specifications

I have read the current document of Open Container Specifications. In my point of view , we can test whether a container tech satisfied the SPEC from the following items

Runtime

• The container lifecycle have 3 status(process): create, start, stop
• Can be invoked several times
• Provide the function to stop the contairner from outside (host？)

Configuration file

• The container’s top-level directory MUST contain a configuration file called config.json.
  it seems that current RKT us manifest.json as a configuration file
• Each container has exactly one root filesystem
• The path under the root configuration ,MUST exist at the relative path declared by the field.
• Supprot all linux filesystem listed in /proc/filesystems by the kernel and ntfs filesystem of windows

Talking about next to do on specstest

This issuse aims to talking about next todo on specstest tool

1. Enable specified output places in ./specs command line, such as "./specs -runc=xxx -specs=xxx -o=xxx.json". done
2. Add item of specs version in testcases to avoid build err with the newest specs, enable framework to choose the testcases to run on specified specs, thus to avoid err caused by huage changing of specs, such as changing of adding runtime.json of specs.
3. Using configValidator in specstest tool to Validate config when using newest specs revesion.
4. Enable register func in framework. done
5. Enable runc repo and set env path of root user on host machine in framework. done
6. Rich the testcases to overall the opencontainers/specs.
7. Enable RKT test in specstest/adaptor.
8. Enable the freamework of specstest supporting the guest programme running in containers.

@liangchenye @maquanyi @wangkirin

Sysbench testcase developing task

use sysbench as testcase, support docker,RKT, Kurma and Jetpack.

Shall we need a open mail list group for the project?

Hi @maquanyi @mengfanliang
Shall we need a open mail list group for the project? it should be open and provide by third part, what about using google mail list Temporarily?

Any plan for docker?

Hi guys, I found docker is not supported currently, do you have any plan for it?

Thanx

Disscusion about the specsValidator to support different version of specs.

Becasuse of the frequantly changing of specs structs in different version, we have no way to make specsValidator supporting the dynamic testing of specs on different version.

So, in my idea, we can make specsValidator to support testing of specs on different versions in build time.

such as,

1. Use golang func //+build to specify different cases belong to different specs version.
   In cases/specprocess , we will have testprocess_v0.0.1.go and testprocess_predraft.go,

// +build v0.0.1

// testprocess_v0.0.1.go
package specprocess

import (
    "fmt"
)

func Testprocess() {
    fmt.Println("v0.0.1")
}

// +build predraft

// testprocess_predraft.go
package specprocess

import (
    "fmt"
)

func Testprocess() {
    fmt.Println("predraft")
}

1. Use go build *_-tags xxx(v0.0.1 or predraft)_ to build.
2. Before building the programme, we have to checkout the specified specs verison in $GOPATH to support building.

In this way, the readme should be like,

1. go get https://github.com/huawei-openlab/oct.git
   if u want to test specified specs version is _vxxx_, just do as below,
2. cd $GOPATH/srg/github/com/opencontainers/specs/
3. git checkout _vxxx_*
4. cd $GOPATH/srg/github/com/huawei-openlab/oct/tools/specsValidator/
5. make vxxx(to do go build -tags vxxx in Makefile)
6. ./specsValidator -runc=xxxxxxx

@maquanyi @liangchenye @wangkirin what is other better suggestions?

Specstest tools moved to oct/tools.

@wangkirin Specstest tools moved to oct/tools.It is going to merged this afternoon, plz refer to Init Specstest in new OCT, Add build script for tools(only enable build specstest tool) #184

Not enough memory bug caused by oci2aci tool

If running oct for rkt testing frequently, bug for not enough memory for testing will occour, that is caused by the bug from oci2aci, relatived issuse as below,
aci images in /tmp dir is not necessary #6

Is there a way to provide the root user & passwd and the user root&passwd to the testcase?

@liangchenye
When I write the json file ,I met the problem, we usally have the both root and the user rights in the same programme, and always have to switch between both of them. So , Is there a way to provide the root user & passwd and the user root&passwd to the testcase?

A example is:
in Spectest for runc, we have to use root power to run runc, also ,we need user power to build the guest programe.

Hardware environment buiding task

@wangkirin plz update your progress and the plan for hardware environment buiding task here.thanks.

how to intergrate the cadvisor to our porject, in the case of having source code changing and adaptor.

@maquanyi @mengfanliang
As we disscussed last time, we should use cadvisor directly to do monitoring for containers,
And now, we have split the monitoring task into two parts,

1. enable support for ocp-testing project,
2. do develop on supporting the RKT container to the cadvisor.
   To the first task,
   we have a question about how to intergrate into the ocp-testing, the situation is,
   we should do some source code developing and adaptor in the cadvisor source code, thus, to enable the cadvisor for ocp-testing.

so , my question is , in this case, how can we intergrate cadvisor to our project?

my opinion is:
we can fork a repo from the cadvisor, and do the source changing in our forked repo, and then just use the forked repo in our project, and by the way, we do the pull&request to the main line of cadvisor.

so ,what about your opinion?

specstest-namespace.json

Hi @zenlinTechnofreak
Which case is the result file 'namespace_out.json' for?

Met the problem when pull code from openlab.

I met the issuse when pull the code from openlab, do anyone meet the same problem?

zenlin@zenlin-pc:~/workspace/ocp-testing$ git remote -v
openlab https://github.com/huawei-openlab/ocp-testing.git (fetch)
openlab https://github.com/huawei-openlab/ocp-testing.git (push)
origin  https://github.com/zenlinTechnofreak/ocp-testing.git (fetch)
origin  https://github.com/zenlinTechnofreak/ocp-testing.git (push)
zenlin@zenlin-pc:~/workspace/ocp-testing$ git pull openlab 
remote: Counting objects: 96, done.
remote: Compressing objects: 100% (71/71), done.
Unpacking objects:  36% (35/96) 

Talking about next to do list from Auth 29th on specsValidator

This issuse aims to talking about next to do jobs from today on specsValidator.
My suggestion:

• Support testing different security policies with runc at one time,
• Using habour in adaptor to support validating RKT container,
• Rich testcase to the entire opencaontainers/specs

Any other suggestions are welcomed.
@liangchenye @maquanyi @wangkirin

Test framework developing task

#1 the scheduler

1.1 parse the test case (check its validation first)
1.2 calling different services (test server, container pool)
in the first round (before 8.2), I'll use the demo test server and container pool
#2 test server daemon

2.1 return the resource id to the scheduler (before 8.2)
#3 container pool daemon

3.1 return the image availability and the id/name to the scheduler (before 8.2)

Benchmark developing

#1. Pressure test through sysbench, support docker,RKT, Kurma and Jetpack.
#2. Container sys monitor through cadvisor, support docker and RKT.

2.1 support docker container monitor through cadvisor
2.2 make cadvisor to support rkt container, and support RKT monitor in the testing project .

Disscustion for the format of the source code organization.

This issuse created to disscustion for the format of the source code organization.
such as,

Cases/Benchmark/sysbench/cpu_load/Dockerfile/

or

Cases/benchmark/sysbench/cpu_load/source/

or

cases/benchmark/sysbench/cpu_load/source/

should be several parts, name format, organization format...etc

how to write value field in "RUN" json when testcase is a golang source code?

@liangchenye If the engine should have a step to complie the source code(some testcase should not be scripts, but using golang)
So, my question is, how to write "value" field below in this case?who will do the call and compile job?

@liugenping

"Run"

The case developer should tell the host operation systems and the containers which commands should be used to run the test. The commands could be used directly or wrapped by a script. By default, after ruuning all the command, the test system will continue to the 'Collect' part.

"Run": [
            {
                "hostA": {
                    "run": null
                }
            },
            {
                "DockerA": {
                    "run": {
                        "type": "cmd",
                        "value": "ping -c 1 -s 1500 192.168.10.10"
                    }
                }
            }
        ]

OCP is now OCI

There was a renaming from "Project" to "Initiative".

Network testcase developing task

Network test:
finish network architechture coding, supportted docker only

Test cases discuss

The current test suite is quite good, the following suggestion may not be necessary, but let's discuss here.

1). Add an extra 'description' session to the cases.conf ?
for example, can we add 'check rootfs is false' to describe one of the process case?
So in the testing output part, it is easier for user to see what has been tested?
2). Add a '--error-only' parameter to the main function, so the user just need to know what is wrong.
3) Adda conclusion output?
for example. '190 out of 200 tests were passed.'

How to do bundle compatibility test?

This issuse opened for disscussion for test of bundle.
I have read the bundle.md, and try to find out how to test the bundle spec.
So far, my consider is, bundle test should be contain 2 parts:

• Use config.json and rootfs, container can be run normall;
• The whole container bundle(conclude config.json and rootfs) can be moved to another machine and run well;
• Some test about signature(no config for it in specs yet).

Milestones discussion

Hi @maquanyi @mengfanliang @liangchenye @ChengTiesheng @huawei-openlab/owners

I have wrote the milestone draft for the project ,we can disscuss it in this issuse.
List my draft below,

0.10.0
 Due by August 2, 2015   Last updated 20 minutes ago
OCP-Testing draft, suportted docker only.
Test Architecture:
finish native architecture coding, conclude request of test server & container pool
Runtime test:
finish benchmark testcase, supportted docker only
Network test:
finish network architechture coding, supportted docker only
Regitry protocol:
finish designe & interface for docker V1&V2 regitry protocol, supporting test architecture

0.20.0
 Due by August 16, 2015   Last updated 4 minutes ago
OCP-Testing draft, suportted docker only, runnable version
Test Architecture:
intergrate testcases
Runtime test:
finish docker specs testcase, supportted docker only
Network test:
finish network perfomance testcases coding, supportted docker only
Regitry protocol:
finish coding for docker V1&V2 regitry protocol, supporting test architecture

1.00
 Due by August 30, 2015   Last updated 2 minutes ago
V1 realease version, support docker & rkt container.

After read README, still don't know how to run the test

As it claimed:

Getting Started

• Read the README for build and test instructions

But I don't see any useful instructions about build and test. Can we improve this?

Thanks

Bug find while running runtimeValidator

./runtimeValidator -rtags=apparmor
Testing Revision:
Specs revision: newest commit
Runc revision: newest commit
Going to update revision, maybe need several mins, plz be patient!
Testing output:
./report/linuxspec.json
Starting to create boundle...
Going to create bind path for hostend-containerend communicating...
done
Creating root filesystem for boundle, maybe need several mins...
Create boundle done
TestReadOnlyTrue failed
TestReadOnlyFalse failed
TestPathError unknowErr
panic: runtime error: index out of range

goroutine 1 [running]:
runtime.panic(0x4ffdc0, 0x6990f7)
/usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.checkOutUser(0xc210068480, 0xbd, 0x51d360, 0x5, 0x5232a0, ...)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/process_v0.1.1.go:189 +0x750
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.testProcessUser(0xc21005c3c0, 0xc210079140, 0x1, 0x0, 0x0, ...)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/process_v0.1.1.go:67 +0x426
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.TestBase(0x0, 0x0)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/test_process_v0.1.1.go:25 +0x1b8
reflect.Value.call(0x4d9280, 0x563230, 0x130, 0x51dc20, 0x4, ...)
/usr/lib/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x4d9280, 0x563230, 0x130, 0x0, 0x0, ...)
/usr/lib/go/src/pkg/reflect/value.go:345 +0x9d
github.com/huawei-openlab/oct/tools/runtimeValidator/manager.(*TestSuite).Run(0xc210050080)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/manager/casemanager.go:117 +0xa1
main.main()
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/specs_v0.1.1.go:124 +0x75b

goroutine 4 [finalizer wait]:
runtime.park(0x40a960, 0x69c110, 0x69a7c8)
/usr/lib/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/usr/lib/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/usr/lib/go/src/pkg/runtime/proc.c:1394

Bug find in exec rkt run, test result of rkt is not correct yet.

Bug find in exec rkt run, test result of rkt is not correct yet, because rkt can not return the error code from the entry process,.
The bug will cause wrong success result of testing rkt.
I and @wangkirin just fixing the bug.

Delete the oct/cases folder for avoiding missunderstanding of new oct?

@liangchenye Shall we delete the oct/cases folder?
Today, some have attemped to use oct, but existing of oct/cases consult him.
Thanks.