deis/workflow-e2e

End-to-end tests for Deis Workflow

	Deis Workflow is no longer maintained. Please read the announcement for more detail.
09/07/2017	Deis Workflow v2.18 final release before entering maintenance mode
03/01/2018	End of Workflow maintenance: critical patches no longer merged
	Hephy is a fork of Workflow that is actively developed and accepts code contributions.

Deis Workflow End to End Tests v2

Deis (pronounced DAY-iss) Workflow is an open source Platform as a Service (PaaS) that adds a developer-friendly layer to any Kubernetes cluster, making it easy to deploy and manage applications on your own servers.

For more information about the Deis Workflow, please visit the main project page at https://github.com/deis/workflow.

We welcome your input! If you have feedback, please submit an issue. If you'd like to participate in development, please read the "Development" section below and submit a pull request.

About

The code in this repository is a set of Ginkgo and Gomega based integration tests that execute commands against a running Deis cluster using the Deis CLI.

Development

The Deis project welcomes contributions from all developers. The high level process for development matches many other open source projects. See below for an outline.

• Fork this repository
• Make your changes
• Submit a pull request (PR) to this repository with your changes, and unit tests whenever possible.
  • If your PR fixes any issues, make sure you write Fixes #1234 in your PR description (where #1234 is the number of the issue you're closing)
• The Deis core contributors will review your code. After each of them sign off on your code, they'll label your PR with LGTM1 and LGTM2 (respectively). Once that happens, the contributors will merge it

Prerequisities

Before you run the tests, you'll need a full Deis cluster up and running in Kubernetes. Follow the instructions here to get one running.

Run the Tests

There are three options for how to execute the tests. These include two options for executing the tests against Deis Workflow installed on a remote Kubernetes cluster, and one option for installing the same tests directly into a Kubernetes cluster and executing them there.

Remote Execution

Either of two options for remote execution of the test suite require the DEIS_CONTROLLER_URL environment variable to be exported. Its value should be the the controller endpoint you would normally use with the deis register or deis login commands:

$ export DEIS_CONTROLLER_URL=http://deis.your.cluster

Tests execute in parallel by default. If you wish to control the number of executors, export a value for the GINKGO_NODES environment variable:

$ export GINKGO_NODES=5

If this is not set, Ginkgo will automatically choose a number of test nodes (executors) based on the number of CPU cores on the machine executing the tests. It is important to note, however, that test execution is constrained more significantly by the resources of the cluster under test than by the resources of the machine executing the tests. The number of test nodes, therefore, should be explicitly set and scaled in proportion to the resources available in the cluster.

For reference, Workflow's own CI pipeline uses the following:

Test Nodes	Kubernetes Worker Nodes	Worker Node CPU	Worker Node Memory
5	3	4 vCPUs	15 GB

Setting the GINKGO_NODES environment variable to a value of 1 will allow serialized execution of all tests in the suite.

Native Execution

If you have Go 1.5 or greater already installed and working properly and also have the Glide dependency management tool for Go installed, you may clone this repository into your $GOPATH:

git clone [email protected]:deis/workflow-e2e.git $GOPATH/src/github.com/deis/workflow-e2e

One-time execution of the following will resolve the test suite's own dependencies:

$ make bootstrap

To execute the entire test suite:

$ make test-integration

To run a single test or set of tests, you'll need the Ginkgo tool installed on your machine:

$ go get github.com/onsi/ginkgo/ginkgo

You can then use the --focus option to run subsets of the test suite:

$ ginkgo --focus="deis apps" tests

Containerized Execution

If you do not have Go 1.5 or greater installed locally, but do have a Docker daemon running locally (or are using docker-machine), you can quite easily execute tests against a remote cluster from within a container.

In this case, you may clone this repository into a path of your own choosing (does not need to be on your $GOPATH):

git clone [email protected]:deis/workflow-e2e.git /path/of/your/choice

Then build the test image and execute the test suite:

$ make docker-build docker-test-integration

Within the Cluster

A third option is to run the test suite from within the very cluster that is under test.

To install the helm chart and start the tests, assuming helm and its corresponding server component tiller are installed:

helm repo add workflow-e2e https://charts.deis.com/workflow-e2e
helm install --verify workflow-e2e/workflow-e2e --namespace deis

To monitor tests as they execute:

$ kubectl --namespace=deis logs -f workflow-e2e tests

Special Note on Resetting Cluster State

All tests clean up after themselves, however, in the case of test failures or interruptions, automatic cleanup may not always proceed as intended. This may leave projects, users or other state behind, which may impact future executions of the test suite against the same cluster. (Often all tests will fail.) If you see this behavior, run these commands to clean up. (Replace deis-workflow-qoxhz with the name of the deis/workflow pod in your cluster.)

$ kubectl exec -it deis-workflow-qoxhz python manage.py shell
Python 2.7.10 (default, Aug 13 2015, 12:27:27)
[GCC 4.9.2] on linux2
>>> from django.contrib.auth import get_user_model
>>> m = get_user_model()
>>> m.objects.exclude(username='AnonymousUser').delete()
>>> m.objects.all()

Note that this is an ongoing issue for which we're planning a more comprehensive fix.

workflow-e2e's Issues

finish the builds tests

there are a few XIts in there: https://github.com/deis/workflow-e2e/blob/master/tests/builds_test.go

write the ps tests

they're all commented out: https://github.com/deis/workflow-e2e/blob/master/tests/ps_test.go

finish the apps tests

there are a few XIts in there: https://github.com/deis/workflow-e2e/blob/master/tests/apps_test.go

Fix ps tests and other pending tests introduced in #123

Due to the fact that we were seeing consistent failures related to curl behavior in ps tests, config tests, and a build test, we marked the tests in #123 as pending. For rc1 let's reenable those tests and make them consistently work reallll nice.

implement a "deis pull" test

So the three main mechanisms of buildpack, dockerfile, and deis pull are all tested. Currently this test is marked Pending.

When app is scaled to zero, should assert response is 503 (not 502)

e2e tests need to use random userids and passwords

$ deis register http://deis.staging-2.deis.com --username=admin --password=admin [email protected]
Registration failed: Error:
400 Bad Request
username: A user with that username already exists.

Failure [0.821 seconds]
[BeforeSuite] BeforeSuite
/Users/jonathanchauncey/go/src/github.com/deis/workflow-e2e/tests/tests_suite_test.go:129

  No future change is possible.  Bailing out early after 0.000s.
  Expected
      <int>: 1
  to match exit code:
      <int>: 0

  /Users/jonathanchauncey/go/src/github.com/deis/workflow-e2e/tests/tests_suite_test.go:182
------------------------------
$ deis login http://deis.staging-2.deis.com --username=test-81 --password=asdf1234
Error:
400 Bad Request
non_field_errors: Unable to log in with provided credentials.

Failure [0.712 seconds]
[AfterSuite] AfterSuite
/Users/jonathanchauncey/go/src/github.com/deis/workflow-e2e/tests/tests_suite_test.go:162

  No future change is possible.  Bailing out early after 0.706s.
  Expected
      <int>: 1
  to match exit code:
      <int>: 0

  /Users/jonathanchauncey/go/src/github.com/deis/workflow-e2e/tests/tests_suite_test.go:215

Scale to zero, expect 502, get 504

A number of us have seen this happening. I've tested with three different configurations. Those configurations and their results:

• kube-aws v0.4.0 w/ k8s v1.1.2: 504
• micro-kube v0.3.0 w/ k8s v1.1.7: 504
• Heard reports of GKE w/ k8s v1.1.8 yielding 502 (as the tests expect), so... micro-kube (unmerged branch) w/ k8s 1.1.8: 502

So it would seem that requests to k8s services with no endpoints in service yield different response code in k8s < v1.1.8 and k8s v1.1.8.

I believe the correct resolution to this is to make the router smarter, so the response from the router is more predictable, regardless of which minor release of k8s it is running on top of. Router did this aat one time, but the code in question was judged both buggy and unnecessary (based on what we knew at the time). It was cut instead of fixed. Hindsight being 20/20, that was a mistake.

I'll open a separate issue to track this in router, but I'm also filing it here so we keep good line of site between the e2e tests and whatever issues are blocking them from completing deterministically wherever they may run.

write the tags tests

they're all commented out: https://github.com/deis/workflow-e2e/blob/master/tests/tags_test.go

add kubectl checks to limits:set tests

As per

workflow-e2e/tests/limits_test.go

Line 10 in 003e0e6

// TODO (bacongobbler): inspect kubectl for limits being applied to manifest

, we need to add checks to kubernetes to validate that the correct limits have been applied to the deployed pods.

Proposal: split e2e tests into their respective repositories

For example, all the tests that use the deis CLI to run stuff against the controller, should be back in deis/workflow and the git push tests should be in deis/builder. This repository should pull all those tests together and run them in a harness.

Increase the "slow test" threshold

Ginkgo focuses on unit tests, so its default for when a test is considered abnormally slow is not generous enough for e2e's needs. If possible, this should be bumped up to...two minutes? To avoid spurious warnings like these:

• [SLOW TEST:65.671 seconds]

e2e job should be using latest helm CLI

The 0.4.0 version is out of date:

Downloading helm-0.4.0+1689ee4-linux-amd64.zip from Bintray...

(I know this isn't strictly in the workflow-e2e code, but this seems like the best place to enter general CI job issues.)

e2e errors when cli performs action before router ready

There are a handful of instances in e2e tests (see here, here and here) where the user is greeted with a 404 when curling the app after certain actions when they should reasonably expect a 200.

For an example, see the behavior documented in #69

Pass `make test-integration` params to test binary

When the deis-tests chart is run, it executes /bin/tests.test inside a container. This ignores the Ginkgo command-line flags used in make test-integration, so many "slow test" warnings are printed, for example.

Either tests.test should somehow accept the same command-line options, allow them to be baked in to the binary, or the Docker image for workflow-e2e should instead just call make test-integration internally.

apps_test: actually inspect 'app info' output

Currently the 'can get app info' test simply verifies that executing deis info does not produce an error.

It would be preferable to actually check the output values, e.g. those of url, owner, id among others.

implement a dockerfile app test

Currently only the buildpack app example-go is used in e2e.

Run test specs in parallel

Ginkgo has nice built-in support for this. It appears the main challenge for the workflow-e2e tests would be to allow multiple BeforeSuite / AfterSuite calls. Currently that would collide on registering and cancelling the default "admin" user expected by the tests.

"scale to 1" ps test failing (expecting: 200, got: 404)

Happening consistently on https://ci.deis.io/job/deis-v2-e2e/

$ deis login http://deis.10.95.244.165.xip.io --username=test-81 --password=asdf1234
17:24:52 Logged in as test-81
17:24:52 $ deis apps:create test-151820431
17:24:52 Creating Application... ...���done, created test-151820431
17:24:52 Git remote deis added
17:24:52 remote available at ssh://[email protected]:2222/test-151820431.git
17:24:52 $ GIT_SSH=/tmp/deis-workflow-home252139278/.ssh/git-ssh git push deis master
17:24:52 Warning: Permanently added '[deis.10.95.244.165.xip.io]:2222,[10.95.244.165]:2222' (ECDSA) to the list of known hosts.
17:24:52 remote: 2016/03/03 00:23:32 Running in debug mode        
17:24:52 remote: [DEBUG] [DEBUG] Running git hook        
17:24:52 remote: [DEBUG] read [0000000000000000000000000000000000000000,779ca3d7c6e4992783d4398499d34474fa55398a,refs/heads/master]        
17:24:52 remote: [DEBUG] Workflow request /v2/hooks/push (body elided)        
17:24:52 remote: Workflow request POST /v2/hooks/config        
17:24:52 remote: [DEBUG] {"receive_user":"test-81","receive_repo":"test-151820431"}        
17:24:52 remote: [DEBUG] got the following config back for app test-151820431: {Owner:test-81 App:test-151820431 Values:map[] Memory:map[] CPU:map[] Tags:map[] UUID:3a30de98-44dd-4b9a-95fa-87caa9d02802 Created:2016-03-03 00:23:32 +0000 UTC Updated:2016-03-03 00:23:32 +0000 UTC}        
17:24:52 remote: [DEBUG] running [git archive --format=tar.gz --output=test-151820431.tar.gz 779ca3d7] in directory /home/git/test-151820431.git        
17:24:52 remote: [DEBUG] running [tar -xzf test-151820431.tar.gz -C /home/git/test-151820431.git/build/tmp022286633/] in directory /home/git/test-151820431.git        
17:24:52 remote: ---> Uploading tar to http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/tar        
17:24:52 remote: [DEBUG] Starting build... but first, coffee!        
17:24:52 remote: [DEBUG] Starting pod slugbuild-test-151820431-779ca3d7-ba0438bc        
17:24:52 remote: Pod spec: {        
17:24:52 remote:   "metadata": {        
17:24:52 remote:     "name": "slugbuild-test-151820431-779ca3d7-ba0438bc",        
17:24:52 remote:     "namespace": "deis",        
17:24:52 remote:     "creationTimestamp": null,        
17:24:52 remote:     "labels": {        
17:24:52 remote:       "heritage": "slugbuild-test-151820431-779ca3d7-ba0438bc"        
17:24:52 remote:     }        
17:24:52 remote:   },        
17:24:52 remote:   "spec": {        
17:24:52 remote:     "volumes": [        
17:24:52 remote:       {        
17:24:52 remote:         "name": "minio-user",        
17:24:52 remote:         "secret": {        
17:24:52 remote:           "secretName": "minio-user"        
17:24:52 remote:         }        
17:24:52 remote:       }        
17:24:52 remote:     ],        
17:24:52 remote:     "containers": [        
17:24:52 remote:       {        
17:24:52 remote:         "name": "deis-slugbuilder",        
17:24:52 remote:         "image": "quay.io/deisci/slugbuilder:v2-beta",        
17:24:52 remote:         "env": [        
17:24:52 remote:           {        
17:24:52 remote:             "name": "DEBUG",        
17:24:52 remote:             "value": "1"        
17:24:52 remote:           },        
17:24:52 remote:           {        
17:24:52 remote:             "name": "TAR_URL",        
17:24:52 remote:             "value": "http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/tar"        
17:24:52 remote:           },        
17:24:52 remote:           {        
17:24:52 remote:             "name": "put_url",        
17:24:52 remote:             "value": "http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/push"        
17:24:52 remote:           }        
17:24:52 remote:         ],        
17:24:52 remote:         "resources": {},        
17:24:52 remote:         "volumeMounts": [        
17:24:52 remote:           {        
17:24:52 remote:             "name": "minio-user",        
17:24:52 remote:             "readOnly": true,        
17:24:52 remote:             "mountPath": "/var/run/secrets/object/store"        
17:24:52 remote:           }        
17:24:52 remote:         ],        
17:24:52 remote:         "imagePullPolicy": "Always"        
17:24:52 remote:       }        
17:24:52 remote:     ],        
17:24:52 remote:     "restartPolicy": "Never",        
17:24:52 remote:     "serviceAccountName": ""        
17:24:52 remote:   },        
17:24:52 remote:   "status": {}        
17:24:52 remote: }        
17:24:52 remote:         
17:24:52 remote: + app_dir=/app        
17:24:52 remote: + build_root=/tmp/build        
17:24:52 remote: + cache_root=/tmp/cache        
17:24:52 remote: + buildpack_root=/tmp/buildpacks        
17:24:52 remote: + mkdir -p /app        
17:24:52 remote: + mkdir -p /tmp/cache        
17:24:52 remote: + mkdir -p /tmp/buildpacks        
17:24:52 remote: + mkdir -p /tmp/build/.profile.d        
17:24:52 remote: + MC_PREFIX='mc --quiet -C /app/mc_config'        
17:24:52 remote: + [[ -z http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/tar ]]        
17:24:52 remote: + [[ -e /var/run/secrets/object/store/access-key-id ]]        
17:24:52 remote: + [[ -e /var/run/secrets/object/store/access-secret-key ]]        
17:24:52 remote: ++ cat /var/run/secrets/object/store/access-key-id        
17:24:52 remote: + keyID=8TZRY2JRWMPT6UMXR6I5        
17:24:52 remote: ++ cat /var/run/secrets/object/store/access-secret-key        
17:24:52 remote: + secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk        
17:24:52 remote: ++ awk -F/ '{print $3}'        
17:24:52 remote: ++ echo http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/tar        
17:24:52 remote: + domain=10.95.243.37:9000        
17:24:52 remote: + mc --quiet -C /app/mc_config config host add http://10.95.243.37:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk        
17:24:52 remote: + mc --quiet -C /app/mc_config cp http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/tar /tmp/slug.tgz        
17:24:52 remote: + tar -xzf /tmp/slug.tgz -C /app/        
17:24:52 remote: + unset TAR_URL        
17:24:52 remote: + [[ '' == \- ]]        
17:24:52 remote: + slug_file=/tmp/slug.tgz        
17:24:52 remote: + [[ -n '' ]]        
17:24:52 remote: + app_dir=/app        
17:24:52 remote: + build_root=/tmp/build        
17:24:52 remote: + cache_root=/tmp/cache        
17:24:52 remote: + buildpack_root=/tmp/buildpacks        
17:24:52 remote: + mkdir -p /app        
17:24:52 remote: + mkdir -p /tmp/cache        
17:24:52 remote: + mkdir -p /tmp/buildpacks        
17:24:52 remote: + mkdir -p /tmp/build/.profile.d        
17:24:52 remote: + '[' -d /tmp/app ']'        
17:24:52 remote: + cp -r /app/. /tmp/build        
17:24:52 remote: + export APP_DIR=/app        
17:24:52 remote: + APP_DIR=/app        
17:24:52 remote: + export HOME=/app        
17:24:52 remote: + HOME=/app        
17:24:52 remote: ++ openssl rand -base64 32        
17:24:52 remote: + export REQUEST_ID=++0YbC4LmRF0udb969Q5TDNVqsrw4egWXoHIU5s08iA=        
17:24:52 remote: + REQUEST_ID=++0YbC4LmRF0udb969Q5TDNVqsrw4egWXoHIU5s08iA=        
17:24:52 remote: + export STACK=cedar-14        
17:24:52 remote: + STACK=cedar-14        
17:24:52 remote: + [[ -n '' ]]        
17:24:52 remote: + buildpacks=($buildpack_root/*)        
17:24:52 remote: + selected_buildpack=        
17:24:52 remote: + [[ -n '' ]]        
17:24:52 remote: + for buildpack in '"${buildpacks[@]}"'        
17:24:52 remote: ++ /tmp/buildpacks/heroku-buildpack-clojure.git/bin/detect /tmp/build        
17:24:52 remote: + buildpack_name=no        
17:24:52 remote: + for buildpack in '"${buildpacks[@]}"'        
17:24:52 remote: ++ /tmp/buildpacks/heroku-buildpack-go.git/bin/detect /tmp/build        
17:24:52 remote: + buildpack_name=Go        
17:24:52 remote: + selected_buildpack=/tmp/buildpacks/heroku-buildpack-go.git        
17:24:52 remote: + break        
17:24:52 remote: + [[ -n /tmp/buildpacks/heroku-buildpack-go.git ]]        
17:24:52 remote: + echo_title 'Go app detected'        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Go app detected        
17:24:52 remote: -----> Go app detected        
17:24:52 remote: + ensure_indent        
17:24:52 remote: + read line        
17:24:52 remote: + /tmp/buildpacks/heroku-buildpack-go.git/bin/compile /tmp/build /tmp/cache        
17:24:52 remote: + [[ -----> Checking Godeps/Godeps.json file. == --* ]]        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Checking Godeps/Godeps.json file.        
17:24:52 remote: -----> Checking Godeps/Godeps.json file.        
17:24:52 remote: + read line        
17:24:52 remote: + [[ -----> Installing go1.6... done == --* ]]        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Installing go1.6... done        
17:24:52 remote: -----> Installing go1.6... done        
17:24:52 remote: + read line        
17:24:52 remote: + [[ -----> Running: go install -v -tags heroku . == --* ]]        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Running: go install -v -tags heroku .        
17:24:52 remote: -----> Running: go install -v -tags heroku .        
17:24:52 remote: + read line        
17:24:52 remote: github.com/deis/example-go        
17:24:52 remote: + /tmp/buildpacks/heroku-buildpack-go.git/bin/release /tmp/build /tmp/cache        
17:24:52 remote: + echo_title 'Discovering process types'        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Discovering process types        
17:24:52 remote: -----> Discovering process types        
17:24:52 remote: + [[ -f /tmp/build/Procfile ]]        
17:24:52 remote: ++ ruby -e 'require '\''yaml'\'';puts YAML.load_file('\''/tmp/build/Procfile'\'').keys().join('\'', '\'')'        
17:24:52 remote: + types=web        
17:24:52 remote: + echo_normal 'Procfile declares types -> web'        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '      ' Procfile declares types '->' web        
17:24:52 remote:        Procfile declares types -> web        
17:24:52 remote: + default_types=        
17:24:52 remote: + [[ -s /tmp/build/.release ]]        
17:24:52 remote: ++ ruby -e 'require '\''yaml'\'';puts (YAML.load_file('\''/tmp/build/.release'\'')['\''default_process_types'\''] || {}).keys().join('\'', '\'')'        
17:24:52 remote: + default_types=        
17:24:52 remote: + [[ -n '' ]]        
17:24:52 remote: + chown -R slug:slug /tmp/build/Godeps /tmp/build/LICENSE /tmp/build/Procfile /tmp/build/README.md /tmp/build/bin /tmp/build/mc_config /tmp/build/web.go        
17:24:52 remote: + [[ -f /tmp/build/.slugignore ]]        
17:24:52 remote: + cat        
17:24:52 remote: + tar -z --exclude=.git -C /tmp/build -cf /tmp/slug.tgz .        
17:24:52 remote: + [[ /tmp/slug.tgz != \- ]]        
17:24:52 remote: ++ cut -f1        
17:24:52 remote: ++ du -Sh /tmp/slug.tgz        
17:24:52 remote: + slug_size=2.2M        
17:24:52 remote: + echo_title 'Compiled slug size is 2.2M'        
17:24:52 remote: + output_redirect        
17:24:52 remote: + [[ /tmp/slug.tgz == \- ]]        
17:24:52 remote: + cat -        
17:24:52 remote: + echo '----->' Compiled slug size is 2.2M        
17:24:52 remote: -----> Compiled slug size is 2.2M        
17:24:52 remote: + [[ -n http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/push ]]        
17:24:52 remote: + [[ -e /var/run/secrets/object/store/access-key-id ]]        
17:24:52 remote: + [[ -e /var/run/secrets/object/store/access-secret-key ]]        
17:24:52 remote: ++ cat /var/run/secrets/object/store/access-key-id        
17:24:52 remote: + keyID=8TZRY2JRWMPT6UMXR6I5        
17:24:52 remote: ++ cat /var/run/secrets/object/store/access-secret-key        
17:24:52 remote: + secretKey=gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk        
17:24:52 remote: ++ awk -F/ '{print $3}'        
17:24:52 remote: ++ echo http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/push        
17:24:52 remote: + domain=10.95.243.37:9000        
17:24:52 remote: + mc --quiet -C /app/mc_config config host add http://10.95.243.37:9000 8TZRY2JRWMPT6UMXR6I5 gbstrOvotMMcg2sMfGUhA5a6Et/EI5ALtIHsobYk        
17:24:52 remote: [DEBUG] + mc --quiet -C /app/mc_config cp /tmp/slug.tgz http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/push/        
17:24:52 remote: ---> size of streamed logs 5075        
17:24:52 remote: ---> Build complete.        
17:24:52 remote: ---> Launching app.        
17:24:52 remote: [DEBUG] Launching...        
17:24:52 remote: Workflow request POST /v2/hooks/build        
17:24:52 remote: {"sha":"779ca3d7","receive_user":"test-81","receive_repo":"test-151820431","image":"http://10.95.243.37:9000/git/home/test-151820431:git-779ca3d7/push/slug.tgz","procfile":{"web":"example-go"},"dockerfile":""}        
17:24:52 remote: --->         
17:24:52 remote: Done, test-151820431:v2 deployed to Deis        
17:24:52 remote: --->         
17:24:52 remote: Use 'deis open' to view this application in your browser        
17:24:52 remote: --->         
17:24:52 remote: To learn more, use 'deis help' or visit http://deis.io        
17:24:52 remote: [DEBUG]         
17:24:52 remote: running [git gc] in directory /home/git/test-151820431.git        
17:24:52 To ssh://[email protected]:2222/test-151820431.git
17:24:52  * [new branch]      master -> master
17:24:52 $ deis ps:scale web=1 --app=test-151820431
17:24:52 Scaling processes... but first, coffee!
17:24:52 ...���done in 0s
17:24:52 === test-151820431 Processes
17:24:52 --- web:
17:24:52 test-151820431-v2-web-m7eba up (v2)
17:24:52 $ deis ps:list --app=test-151820431
17:24:52 === test-151820431 Processes
17:24:52 --- web:
17:24:52 test-151820431-v2-web-m7eba up (v2)
17:24:52 $ curl -sL -w "%{http_code}\\n" "http://test-151820431.10.95.244.165.xip.io" -o /dev/null
17:24:52 404
17:24:52 
17:24:52 ------------------------------
17:24:52 • Failure [23.872 seconds]
17:24:52 Processes
17:24:52 /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:155
17:24:52   with a deployed app
17:24:52   /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:154
17:24:52     can scale up and down
17:24:52     /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table.go:96
17:24:52       scales to 1 [It]
17:24:52       /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46
17:24:52 
17:24:52       No future change is possible.  Bailing out early after 0.154s.
17:24:52       Got stuck at:
17:24:52           404
17:24:52           
17:24:52       Waiting for:
17:24:52           200
17:24:52 
17:24:52       /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:71
17:24:52 ------------------------------

If someone wants to link this issue with an issue living in a component repo, feel free.

deis open file shim data race

In the deis open test in apps_test.go, we've seen behavior where the system shim/fake open utility is not ready when it is executed by the deis client:

Error: fork/exec /tmp/xdg-open: text file busy

As seen here: https://ci.deis.io/job/deis-v2-e2e/391/console

remove the _tests folder in deis/workflow

Ensure that all tests from that folder in deis/workflow are moved to here, and then that folder should be removed.

finish the perms tests

there's one last XIt to finis: https://github.com/deis/workflow-e2e/blob/master/tests/perms_test.go#L63

jenkins: link deis-v2-e2e-pr test results in workflow-e2e PRs

Currently, the Jenkins PR job in this repo, although it kicks off the e2e job it only outputs the following: default — Build finished. No test results found.

It would be preferable if this supplied a link to the test results as well as having the actual pass/fail be determined by the downstream e2e job

update README.md

We should revisit and update README.md. Here are some initial ideas:

1. example(s) on how to run tests via docker container (see make docker-build/docker-push/docker-test-integration)
2. example of how to run tests via a helm chart, i.e. launching a deis-tests (as it is currently known) pod against a workflow-dev cluster, a workflow-beta1-e2e pod against a workflow-beta1 cluster chart, etc.
3. amend Prerequisites section (for instance, helm install deis/deis should be helm install deis/workflow-dev)

Test panic in ps_tests

Looking at the CI logs, I saw this "invalid argument to Intn" panic that looks like something that could be fixed in the e2e tests themselves:

$ deis login http://deis.10.235.245.123.xip.io --username=test-81 --password=asdf1234
Logged in as test-81
$ deis ps:scale web=1 --app=test-693426652
Scaling processes... but first, coffee!
...���..o���done in 0s
=== test-693426652 Processes
--- web:
test-693426652-v2-web-er7c8 terminating (v2)
•! Panic [1.386 seconds]
Processes
/go/src/github.com/deis/workflow-e2e/tests/ps_test.go:155
  with a deployed app
  /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:154
    can restart processes
    /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table.go:96
      restarts one of 1 [It]
      /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46

      Test Panicked
      invalid argument to Intn
      /usr/local/go/src/runtime/panic.go:426

      Full Stack Trace
        /usr/local/go/src/runtime/panic.go:426 +0x521
      math/rand.(*Rand).Intn(0xc82000a550, 0x0, 0x1)
        /usr/local/go/src/math/rand/rand.go:98 +0x87
      math/rand.Intn(0x0, 0x3)
        /usr/local/go/src/math/rand/rand.go:213 +0x4b
      github.com/deis/workflow-e2e/tests.glob.func11.1.3(0xa06aa0, 0x3, 0x1, 0xc8)
        /go/src/github.com/deis/workflow-e2e/tests/ps_test.go:110 +0x2513
      reflect.Value.call(0x93f6a0, 0xc82018baa0, 0x13, 0xa02fe8, 0x4, 0xc8201e5680, 0x3, 0x4, 0x0, 0x0, ...)
        /usr/local/go/src/reflect/value.go:435 +0x1574
      reflect.Value.Call(0x93f6a0, 0xc82018baa0, 0x13, 0xc8201e5680, 0x3, 0x4, 0x0, 0x0, 0x0)
        /usr/local/go/src/reflect/value.go:303 +0xce
      github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table.TableEntry.generateIt.func1()
        /go/src/github.com/deis/workflow-e2e/tests/tests_suite_test.go:52 +0xa4
      testing.tRunner(0xc8200726c0, 0xc88720)
        /usr/local/go/src/testing/testing.go:473 +0xdd
      created by testing.RunTests
        /usr/local/go/src/testing/testing.go:582 +0xae3

write tests for certificates

test app needs to be configurable

Currently it's hardcoded to Deis' example-go app, but the actual git URL used should be customizable via environment variable or another mechanism, so that any 12-factor app repo (not just those in the deis org) can be run through its paces.

setup CI

This issue requires #5. For this issue to be accepted, CI should run make docker-build on all branches, on every commit.

Then, on pushes to master, CI should also run make docker-push to push our e2e testing image to quay and dockerhub. This functionality implies that we'll need a deploy script like the one in deis/builder

auth:cancel

Test all the auth:cancel scenarios

Containerize the integration test environment

Version 0.4.0 of our go-dev docker image includes ginkgo, so it should now be possible to containerize the integration test environment.

domains_test intermittent failure

The e2e job has seen a smattering of failures in the domains_test, which I have not been able to reproduce locally (on my k8s cluster) when focusing:

$ deis domains:remove my-custom-277364747.domain.com --app=test-295126610
Removing my-custom-277364747.domain.com from test-295126610... ...���done
$ deis domains:remove my-custom-277364747.domain.com --app=test-295126610
Removing my-custom-277364747.domain.com from test-295126610... ...���Error: 
404 Not Found
detail: Not found.

$ deis domains:list --app=test-295126610
=== test-295126610 Domains
test-295126610
Waiting up to 10 seconds for `curl -sL -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null` to return 200...
$ curl -sL -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
Waiting up to 10 seconds for `curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null` to return 404...
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
$ curl -sL -H "Host: my-custom-277364747.domain.com" -w "%{http_code}\\n" "http://test-295126610.10.131.243.248.xip.io" -o /dev/null
200
FAIL: '200
' does not match expected result of '404'
$ deis apps:destroy --app=test-295126610 --confirm=test-295126610
Destroying test-295126610...
done in 0s

• Failure [51.635 seconds]
Domains
/go/src/github.com/deis/workflow-e2e/tests/domains_test.go:159
  with a deployed app
  /go/src/github.com/deis/workflow-e2e/tests/domains_test.go:158
    can add, list, and remove domains [It]
    /go/src/github.com/deis/workflow-e2e/tests/domains_test.go:157

    Timed out after 10.000s.
    Expected
        <bool>: false
    to be true

    /go/src/github.com/deis/workflow-e2e/tests/domains_test.go:156
------------------------------

One theory is the 10 second timeout is just a tad too tight. If this issue keeps appearing, we should discuss increasing the timeout (as minimally as possible).

A corresponding indication curling removed domain still returns 200 after 10 secs has been added to Jenkins to catch.

Subset the tests in makefile targets or doc examples

Ginkgo has great support for "focused" specs, which are a convenient way to run a subset of the entire e2e suite. While the suite can and should be sped up, it would also be useful for devs to be aware of how to run only relevant e2e tests and speed up their development cycle.

Perhaps adding some example Makefile targets like make test-apps, and/or writing a couple examples for the project README.md?

test config:set with multi-line values in container

deis/deis#4844
deis/workflow#268

We have automated tests here, but we also need to ensure that the environment variable inside the container is properly escaped like it's done on Heroku.

Test container should be based on ubuntu-slim

To optimize layers in common with other deis components and presumably reduce the final image size. (Although I assume @arschles had good reasons for going with ubuntu-debootstrap, which is deprecated now BTW.)

e2e tests should have a "test all buildpacks" option

Ideally in the same test suite and in parallel. See also deis/deis#2879.

Proposal: Move workflow-e2e tests to using the SDK instead of the CLI

To help drive stability and speed within the workflow-e2e tests we should move these tests to be driven through the SDK. This means we can explicitly check on http responses instead of waiting for buffered returns with the CLI.

The SDK workflow-e2e tests would be the more comprehensive test suite and may include edge cases and failure modes.

The CLI workflow tests would validate the appropriate responses for all CLI actions but would not test the system as a comprehensive unit (for example we would try and stay away from exhaustive scaling of apps within the CLI test suite).

make this repository public

Before merging #5, we should make this repository public. After public, we should move on with #6.

cc/ @gabrtv

Include deis run tests in e2e tests

deis/workflow#406

Recent change in versioning on the workflow client broke tests

• Failure [1.593 seconds]
Version
/go/src/github.com/deis/workflow-e2e/tests/version_test.go:16
  prints its version [It]
  /go/src/github.com/deis/workflow-e2e/tests/version_test.go:15

  Expected
      <string>: 2.0.0-dev-916fe94

  to equal
      <string>: 2.0.0-dev

finish the releases tests

releases_test.go needs some love, with particular attention to:

        // 500's everytime
        XIt("can deploy the app", func() {
            sess, err := start("deis pull deis/example-go -a %s", appName)
            Expect(err).To(BeNil())
            Eventually(sess, (10 * time.Minute)).Should(Exit(0))
            Eventually(sess).Should(Say("Creating build... done"))
        })

In addition, the tests context currently claims Context("with a deployed app" but the app isn't actually being deployed in the BeforeEach step

organize common suite methods into own dir/package

Currently, certs_test.go and domains_test.go are two suites that have a collection of common action/expectation pairs in the form of utility methods at the top of the file (high and DRY).

It would make sense to unclutter the *_test.go file itself and put these into a dir or perhaps sister package (common is one suggestion) that any test in /tests can pull in as needed -- especially if more tests follow the same patterns.

See, for example, in certs_test.go:

// TODO: move to sister dir/package 'common'
//       for example, these could live in common/certs.go
// certs-specific common actions and expectations
func listCerts() *Session {
    sess, err := start("deis certs:list")
    Expect(err).NotTo(HaveOccurred())
    Eventually(sess).Should(Exit(0))

    return sess
}

write limits tests

https://github.com/deis/workflow-e2e/blob/master/tests/limits_test.go

Enable deis logs checks once available

Once we can invoke deis logs in e2e (read: once DaemonSets are supported in GKE), let's enable the following test verification in builds_test:

// TODO: check workers; can't currently use logs
// "deis logs -a %s", app
// sess, err = start("deis logs -a %s", testApp.Name)
// Expect(err).To(BeNil())
// Eventually(sess).Should(Say("hi"))
// Eventually(sess).Should(Exit(0))

and in apps_test:

XIt("can get app logs", func() {
    cmd, err := start("deis logs")
    Expect(err).NotTo(HaveOccurred())
    Eventually(cmd).Should(SatisfyAll(
        Say("%s\\[deis-controller\\]\\: %s created initial release", testApp.Name, testUser),
        Say("%s\\[deis-controller\\]\\: %s deployed", testApp.Name, testUser),
        Say("%s\\[deis-controller\\]\\: %s scaled containers", testApp.Name, testUser)))
})

Tests should run as a non-root user

All Deis components should run as regular, non-privileged users within their Docker containers, including this deis-e2e image.

Run docker build on all PR builds

Similar to what builder does in its .travis.yml file, workflow-e2e should run make ... docker-build to ensure that the Dockerfile is correct

Add k8s manifests

Continuation of #5 and deis/workflow#194. It should be possible for anyone to run our e2e tests in a pod. Optional for this issue: add a helm chart for e2e tests.

This issue implicitly requires #9.

Return appropriate exit code after test completion

I'm using the in-k8s method of running the tests (where a pod is spun up which runs the Docker image). See the following behavior:

Summarizing 4 Failures:

[Fail] Config with a deployed app [It] can list environment variables
/go/src/github.com/deis/workflow/_tests/tests/config_test.go:14

[Fail] Healthcheck with a deployed app [BeforeEach] can stay running during a scale event
/go/src/github.com/deis/workflow/_tests/tests/healthcheck_test.go:21

[Fail] Apps with a deployed app [BeforeEach] can't create an existing app
/go/src/github.com/deis/workflow/_tests/tests/apps_test.go:108

[Fail] Apps with a deployed app [BeforeEach] can get app info
/go/src/github.com/deis/workflow/_tests/tests/apps_test.go:108

Ran 35 of 56 Specs in 36.185 seconds
FAIL! -- 31 Passed | 4 Failed | 21 Pending | 0 Skipped --- FAIL: TestTests (36.19s)
FAIL

Then I run:

 ~/go/src/github.com/deis/charts  ⑂ e2e-tests    echo $?
0

This makes me sad.

write the domains tests

they're all commented out now: https://github.com/deis/workflow-e2e/blob/master/tests/domains_test.go

spawn a new deis cluster (in a new namespace) for each run

this one may be involved, but it would be nice to create a new deis cluster in its own namespace for each test run, and then tear it down at the end. this would eliminate a whole class of state management issues across test suite runs, and also allow us to run multiple suite concurrently on the same k8s cluster.

also note that I believe we have some outstanding issues that prevent us from running 2+ deis clusters reliably on the same k8s cluster, so those will have to be cleared before this can be possible.

Purposefully not assigned a milestone, as this is a nice to have and not critical.

finish the config tests

They are all XIted out currently.