huawei-openlab / oct Goto Github PK
View Code? Open in Web Editor NEWTesting cases and framework for OpenContainers project
License: Apache License 2.0
Testing cases and framework for OpenContainers project
License: Apache License 2.0
root@szxbzci0039:~/gopath/src/github.com/huawei-openlab/oct# ./ocitest -r runc -o err-only
INFO[0001] Failure Details:
INFO[0001] ===============================================================================================
INFO[0001]
BundleName:
rlimit14
BundleDir:
bundles/rlimit14
CaseArgs:
--args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_SIGPENDING:1024:1024
Result:
FAILED
ErrInfo:
time="2015-12-01T10:08:18+08:00" level=fatal msg="Container start failed: Wrong rlimit value: RLIMIT_SIGPENDING"
exit status 1
INFO[0001]
BundleName:
rlimit1
BundleDir:
bundles/rlimit1
CaseArgs:
--args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_AS:1024:1024
Result:
FAILED
ErrInfo:
exit status 139
INFO[0001]
BundleName:
rlimit15
BundleDir:
bundles/rlimit15
CaseArgs:
--args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_STACK:1024:1024
Result:
FAILED
ErrInfo:
exit status 139
INFO[0001]
BundleName:
platform1
BundleDir:
bundles/platform1
CaseArgs:
--args=./runtimetest --args=vpl --rootfs=rootfs --os=windows
Result:
FAILED
ErrInfo:
time="2015-12-01T02:08:18Z" level=fatal msg="Validation failed: \"OS expected: windows, actual: linux\""
exit status 1
INFO[0001]
BundleName:
rlimit10
BundleDir:
bundles/rlimit10
CaseArgs:
--args=./runtimetest --args=vrl --rootfs=rootfs --rlimit=RLIMIT_NPPOC:1024:1024
Result:
FAILED
ErrInfo:
time="2015-12-01T10:08:18+08:00" level=fatal msg="Container start failed: Wrong rlimit value: RLIMIT_NPPOC"
exit status 1
INFO[0001] ===============================================================================================
INFO[0001] Statistics: 25 bundles success, 5 bundles failed
It shows, rlimit are not support very well in runc, such as, RLIMIT_SIGPENDING, RLIMIT_AS, RLIMIT_STACK, RLIMIT_NPPOC
I have read the current document of Open Container Specifications. In my point of view , we can test whether a container tech satisfied the SPEC from the following items
config.json
.manifest.json
as a configuration filepath
under the root configuration ,MUST exist at the relative path declared by the field./proc/filesystems
by the kernel and ntfs filesystem of windowsuse sysbench as testcase, support docker,RKT, Kurma and Jetpack.
Hi @maquanyi @mengfanliang
Shall we need a open mail list group for the project? it should be open and provide by third part, what about using google mail list Temporarily?
Hi guys, I found docker is not supported currently, do you have any plan for it?
Thanx
Becasuse of the frequantly changing of specs structs in different version, we have no way to make specsValidator supporting the dynamic testing of specs on different version.
So, in my idea, we can make specsValidator to support testing of specs on different versions in build time.
such as,
// +build v0.0.1
// testprocess_v0.0.1.go
package specprocess
import (
"fmt"
)
func Testprocess() {
fmt.Println("v0.0.1")
}
// +build predraft
// testprocess_predraft.go
package specprocess
import (
"fmt"
)
func Testprocess() {
fmt.Println("predraft")
}
In this way, the readme should be like,
@maquanyi @liangchenye @wangkirin what is other better suggestions?
@wangkirin Specstest tools moved to oct/tools.It is going to merged this afternoon, plz refer to Init Specstest in new OCT, Add build script for tools(only enable build specstest tool) #184
If running oct for rkt testing frequently, bug for not enough memory for testing will occour, that is caused by the bug from oci2aci, relatived issuse as below,
aci images in /tmp dir is not necessary #6
@liangchenye
When I write the json file ,I met the problem, we usally have the both root and the user rights in the same programme, and always have to switch between both of them. So , Is there a way to provide the root user & passwd and the user root&passwd to the testcase?
A example is:
in Spectest for runc, we have to use root power to run runc, also ,we need user power to build the guest programe.
@wangkirin plz update your progress and the plan for hardware environment buiding task here.thanks.
@maquanyi @mengfanliang
As we disscussed last time, we should use cadvisor directly to do monitoring for containers,
And now, we have split the monitoring task into two parts,
so , my question is , in this case, how can we intergrate cadvisor to our project?
my opinion is:
we can fork a repo from the cadvisor, and do the source changing in our forked repo, and then just use the forked repo in our project, and by the way, we do the pull&request to the main line of cadvisor.
so ,what about your opinion?
Hi @zenlinTechnofreak
Which case is the result file 'namespace_out.json' for?
I met the issuse when pull the code from openlab, do anyone meet the same problem?
zenlin@zenlin-pc:~/workspace/ocp-testing$ git remote -v
openlab https://github.com/huawei-openlab/ocp-testing.git (fetch)
openlab https://github.com/huawei-openlab/ocp-testing.git (push)
origin https://github.com/zenlinTechnofreak/ocp-testing.git (fetch)
origin https://github.com/zenlinTechnofreak/ocp-testing.git (push)
zenlin@zenlin-pc:~/workspace/ocp-testing$ git pull openlab
remote: Counting objects: 96, done.
remote: Compressing objects: 100% (71/71), done.
Unpacking objects: 36% (35/96)
This issuse aims to talking about next to do jobs from today on specsValidator.
My suggestion:
Any other suggestions are welcomed.
@liangchenye @maquanyi @wangkirin
#1 the scheduler
1.1 parse the test case (check its validation first)
1.2 calling different services (test server, container pool)
in the first round (before 8.2), I'll use the demo test server and container pool
#2 test server daemon
2.1 return the resource id to the scheduler (before 8.2)
#3 container pool daemon
3.1 return the image availability and the id/name to the scheduler (before 8.2)
This issuse created to disscustion for the format of the source code organization.
such as,
Cases/Benchmark/sysbench/cpu_load/Dockerfile/
or
Cases/benchmark/sysbench/cpu_load/source/
or
cases/benchmark/sysbench/cpu_load/source/
should be several parts, name format, organization format...etc
@liangchenye If the engine should have a step to complie the source code(some testcase should not be scripts, but using golang)
So, my question is, how to write "value" field below in this case?who will do the call and compile job?
"Run"
The case developer should tell the host operation systems and the containers which commands should be used to run the test. The commands could be used directly or wrapped by a script. By default, after ruuning all the command, the test system will continue to the 'Collect' part.
"Run": [
{
"hostA": {
"run": null
}
},
{
"DockerA": {
"run": {
"type": "cmd",
"value": "ping -c 1 -s 1500 192.168.10.10"
}
}
}
]
There was a renaming from "Project" to "Initiative".
Network test:
finish network architechture coding, supportted docker only
The current test suite is quite good, the following suggestion may not be necessary, but let's discuss here.
1). Add an extra 'description' session to the cases.conf ?
for example, can we add 'check rootfs is false' to describe one of the process case?
So in the testing output part, it is easier for user to see what has been tested?
2). Add a '--error-only' parameter to the main function, so the user just need to know what is wrong.
3) Adda conclusion output?
for example. '190 out of 200 tests were passed.'
This issuse opened for disscussion for test of bundle.
I have read the bundle.md, and try to find out how to test the bundle spec.
So far, my consider is, bundle test should be contain 2 parts:
Hi @maquanyi @mengfanliang @liangchenye @ChengTiesheng @huawei-openlab/owners
I have wrote the milestone draft for the project ,we can disscuss it in this issuse.
List my draft below,
0.10.0
Due by August 2, 2015 Last updated 20 minutes ago
OCP-Testing draft, suportted docker only.
Test Architecture:
finish native architecture coding, conclude request of test server & container pool
Runtime test:
finish benchmark testcase, supportted docker only
Network test:
finish network architechture coding, supportted docker only
Regitry protocol:
finish designe & interface for docker V1&V2 regitry protocol, supporting test architecture
0.20.0
Due by August 16, 2015 Last updated 4 minutes ago
OCP-Testing draft, suportted docker only, runnable version
Test Architecture:
intergrate testcases
Runtime test:
finish docker specs testcase, supportted docker only
Network test:
finish network perfomance testcases coding, supportted docker only
Regitry protocol:
finish coding for docker V1&V2 regitry protocol, supporting test architecture
1.00
Due by August 30, 2015 Last updated 2 minutes ago
V1 realease version, support docker & rkt container.
As it claimed:
But I don't see any useful instructions about build and test. Can we improve this?
Thanks
./runtimeValidator -rtags=apparmor
Testing Revision:
Specs revision: newest commit
Runc revision: newest commit
Going to update revision, maybe need several mins, plz be patient!
Testing output:
./report/linuxspec.json
Starting to create boundle...
Going to create bind path for hostend-containerend communicating...
done
Creating root filesystem for boundle, maybe need several mins...
Create boundle done
TestReadOnlyTrue failed
TestReadOnlyFalse failed
TestPathError unknowErr
panic: runtime error: index out of range
goroutine 1 [running]:
runtime.panic(0x4ffdc0, 0x6990f7)
/usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.checkOutUser(0xc210068480, 0xbd, 0x51d360, 0x5, 0x5232a0, ...)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/process_v0.1.1.go:189 +0x750
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.testProcessUser(0xc21005c3c0, 0xc210079140, 0x1, 0x0, 0x0, ...)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/process_v0.1.1.go:67 +0x426
github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess.TestBase(0x0, 0x0)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/cases/specprocess/test_process_v0.1.1.go:25 +0x1b8
reflect.Value.call(0x4d9280, 0x563230, 0x130, 0x51dc20, 0x4, ...)
/usr/lib/go/src/pkg/reflect/value.go:474 +0xe0b
reflect.Value.Call(0x4d9280, 0x563230, 0x130, 0x0, 0x0, ...)
/usr/lib/go/src/pkg/reflect/value.go:345 +0x9d
github.com/huawei-openlab/oct/tools/runtimeValidator/manager.(*TestSuite).Run(0xc210050080)
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/manager/casemanager.go:117 +0xa1
main.main()
/Go/src/github.com/huawei-openlab/oct/tools/runtimeValidator/specs_v0.1.1.go:124 +0x75b
goroutine 4 [finalizer wait]:
runtime.park(0x40a960, 0x69c110, 0x69a7c8)
/usr/lib/go/src/pkg/runtime/proc.c:1342 +0x66
runfinq()
/usr/lib/go/src/pkg/runtime/mgc0.c:2279 +0x84
runtime.goexit()
/usr/lib/go/src/pkg/runtime/proc.c:1394
Bug find in exec rkt run, test result of rkt is not correct yet, because rkt can not return the error code from the entry process,.
The bug will cause wrong success result of testing rkt.
I and @wangkirin just fixing the bug.
@liangchenye Shall we delete the oct/cases folder?
Today, some have attemped to use oct, but existing of oct/cases consult him.
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.