kubernetes-retired/testing_frameworks

[EOL] test frameworks for testing kubernetes

Frameworks

Test frameworks for testing kubernetes

This was created as a result of kubernetes/community#1524

What lives here?

• The integration test framework

What is allowed to live here?

Any test framework for testing any part of kubernetes is welcome so long as we can avoid vendor loops.

Right now, the only things vendored into this repo are ginkgo and gomega. We would like to keep vendored libraries to a minimum in order to make it as easy as possible to import these frameworks into other kubernetes repos. Code in this repo should certainly never import k8s.io/kubernetes.

testing_frameworks's Issues

Support creating headless nodes

See kubemark tests for example. It would be useful to have fake nodes that can set the state of Pods.

Allow changing StartMessage in APIServer for secure servers

When building a custom APIServer, the only option built into the k8s.io/apiserver package is secure serving. Insecure serving is added as an entire custom listener in the main kubernetes apiserver, and the code for that is stored in k8s.io/kubernetes: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubeapiserver/server/insecure_handler.go

I've managed to get my custom apiserver to run without the aggregator in place by disabling authN and authZ - however the APIServer fixture in this framework checks for the string Serving insecurely on %s which is never printed by my apiserver. Instead, this is printed:

Serving securely on 127.0.0.1:50352

It'd be great if we can switch this somehow, either by passing a 'UseSecure' bool and having the APIServer handle it, or alternatively allowing the string itself to be overridden.

Setup prow job instead of travis

See the discussion here: kubernetes/test-infra#6363 (comment)

Support a pre-configured client-go client

Currently, it supports a pre-configured kubectl for a control plane.
We probably should also provide a pre-configured client-go client.

Integration: There should be an easy way check the process state

(This is mostly talking about Etcd & APIServer, but we should try to give users the same interface to do that on all processes, so also on KubeCtl)

When a process can be successfully started, but runs into an issue before we consider it to be up, all the feedback we get is a timeout error. We have no option yet to check on

• process state (is it still running?)
• its exit code (if it already terminated)
• its StdOut / StdErr

We should see to give the user access to the above mentioned items.

Decide how to keep binaries up to date

Right now all the binaries we support are compiled once on the fly and uploaded to some random google bucket.
We need a proper strategy on how and when we want to update the binaries and how to provide those selected binaries.

This came out of #33 .

Integration: Enhance address manager, make it less vulnerable to races

Can we do somewhat the same as in https://github.com/Yelp/ephemeral-port-reserve to make races on finding a free port a bit less likely?

Provide rest.Config to talk to apiserver

kube-apiserver binaries for alternate OS's

I'm trying to run some integration tests that depend on this framework on my development laptop, running OSX.

I've noticed that there are no kube-apiserver binaries published for darwin. When building myself with a plain ol' `go build, the apiserver starts with

F0126 10:19:06.353915   13847 openapi.go:37] Failed to register open api spec for root: cannot find model definition for k8s.io/api/apps/v1.StatefulSetList. If you added a new type, you may need to add +k8s:openapi-gen=true to the package or type and run code-gen again

I'm wondering if there are plans to start publishing kube-apiserver and potentially kube-controller-manager for alternate platforms? Else this form of testing is going to be a pain for developers consuming this framework!

Add ability to easily append a flag to the default flags

As it stands now, it is not possible to append a flag to the default flags. Either you send all of your own flags, or you use only the default flags. For users who need to modify flags, this is a poor interface.

Support control plane to start kube ControllerManager and Scheduler

For some tests it would be helpful to have the controller manager and scheduler start. Such as when testing CRD controllers you may want to test that the Deployment you created is behaving as expected

kube-apiserver >= 1.12 requires use of secured port

As per kubernetes-retired/kubefed#161, kube-apiserver built from master is now requiring the secured port to be configured. I had this working previously, and if nobody else is working on it I am likely to want to enable this behavior to allow users of the framework to test against pre-release kube and kube >= 1.12 when it is eventually released.

Setup merge automation for this repo

Use an OWNERS file, /approve, /lgtm to allow people to merge PR's into this repo:

• If there isn't an OWNERS file, create one and populate it with people who already have write access to this repo (as approvers)
• Configure prow to turn on the "approval" plugin for this repo
• Configure prow/tide to include this repo in the tide query

In addition, I'd like to enable the majority of prow plugins that we use in the kubernetes org, excluding:

• golint (no sense turning this on until we have golang code in here)
• slackevents (until we have automated merges setup, no sense alerting on manual merges)
• trigger (not going to kick off any jobs for this repo yet)
• milestonestatus (I don't think we have any interest in using the status/foo labels)
• sigmention (we don't have all the sig teams setup in this org)

Support TEST_ASSET_PATH as environment variable

It is annoying to set a different environment variable for each binary. If we add new binaries such as the controller-manager and scheduler we will have to update the documentation and users will have to update their test scripts.

• Support looking under TEST_ASSET_PATH for binaries if it is defined
• Have download script write binaries to TEST_ASSET_PATH if it is defined

download-binaries.sh defaults to download from a private bucket

integration/scripts/download-binaries.sh defaults to retrieve binaries from gs://k8s-c10s-test-binaries, however this bucket is not publicly accessible.

Is this intentional/can this bucket be opened up?

Provide simple tool to download all required test assets and put them some place

kube-test install-assets <location>

Create a SECURITY_CONTACTS file.

As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS
file.

The template for the file can be found in the kubernetes-template repository[2].
A description for the file is in the steering-committee docs[3], you might need
to search that page for "Security Contacts".

Please feel free to ping me on the PR when you make it, otherwise I will see when
you close this issue. :)

Thanks so much, let me know if you have any questions.

(This issue was generated from a tool, apologies for any weirdness.)

[1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE
[2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS
[3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

Tag a release?

It would be useful for people depending on test_frameworks to have a release tagged. This makes dependency management a bit easier for consumers, and works around some issues with exact SHA dependencies in dependency management tools (e.g. dep tends to be more picky then usual with exact SHAs). Even just a v0.1.0 release would be useful

Provide tar.gz for the assets

We could make the files smaller. This is especially noticeable when download them from a Dockerfile as part of the build.

Document environment variable to override the kubectl location

Enable RBAC and support using ServiceAccounts

As a CRD developer, in production I run my controller in a Namespace with a ServiceAccount and RBAC rules configured for it. I want the integration tests for my controller to mimic production as closely as possible and catch errors if the RBAC rules are incorrectly setup. To do this I need:

• RBAC to be enabled in the cluster
• To create a Namespace in the cluster and setup RBAC rules for my controller
• To get a Config (credentials) authorized as the ServiceAccount

Pinging for signs of life

Is this codebase actively in use? Is it being maintained?

It appears the testing-commons subproject is focused on refactoring kubernetes/kuberentes test/e2e/framework, so it's unclear to me whether it's worth keeping this repo active or not

Source binaries from the PATH

I think it would be preferable to source binaries from the PATH. This is a well-known and understood way of finding executables.

Tag next release

What is process for doing a release in testing_frameworks?

It will be helpful if you can do another release (may v0.1.1) with the fix in #62.
It will benefit us (kubebuilder team) and federation team.

Integration: Arguments for Etcd & APIServer should be configurable

Currently we assume that every version of the binaries take the same arguments:

• Etcd's arguments
• APIServer's arguments

With kubernetes/kubernetes#55988 the GenericAdmissionWebhook was renamed as ValidatingAdmissionWebhook in the apiserver's --admission-control flag. That means that the framework cannot start an APIServer which was built after that rename.

We need a way to version, overwrite or configure the arguments we pass when starting a binary.

[Question] Ready to use?

Is this ready for other project to use? I'd like to use this in https://github.com/kubernetes-sigs/kube-batch :)

The testing_ prefix to the repo name is no longer required

Feel free to rename the repo. (E.g., you could change the underscore to a dash)

Potentially reusable code in an internal package

The code implementing the integration fixtures is in an 'internal' package, which precludes use by anyone vendoring the repo. I think the goal of this repo should be creating reusable fixture but also enabling others to create their own fixture, which suggests that the code in question be importable.

Setup CNCF CLA bot for this repo

I'd like to go ahead and do it for all repos in this org if I can

/assign