docker version

[aws]

docker version

[samuelkarp]

Hi @rjschwei,

We intentionally and explicitly tie the version of ecs-init to our latest tested version of Docker. Currently, the latest version of Docker available in the Amazon Linux AMI's yum repositories is 1.6.2. While we believe that ecs-init (and our Agent) should work without changes on newer versions of Docker, we want to verify that it continues to work before we change the spec file.

Does that help answer your question?

Thanks!
Sam

[rjschwei]

On 08/11/2015 12:43 PM, Samuel Karp wrote:

Hi @rjschwei,

We intentionally and explicitly tie the version of ecs-init to our latest tested version of Docker. Currently, the latest version of Docker available in the Amazon Linux AMI's yum repositories is 1.6.2. While we believe that ecs-init (and our Agent) should work without changes on newer versions of Docker, we want to verify that it continues to work before we change the spec file.

Does that help answer your question?

Yes, but leads to a follow on question. Is there a process by which we
can test the functionality with a newer version of docker?

Do the tests that are part of the source code cover this or are there
additional test that should be run?

Thanks,
Robert

Robert Schweikert MAY THE SOURCE BE WITH YOU
Public Cloud Architect LINUX
[email protected]
IRC: robjo

[samuelkarp]

I've pushed a set of functional tests to the dev branch of the Agent (see aws/amazon-ecs-agent#161, thanks @euank for most of the work!), which should do a decent job of testing whether the Agent functions correctly. Note that you will need a fairly permissive role policy on the instance to run these tests (not sure of the complete set offhand, but at least ecs:StartTask, ecs:DescribeTasks, ecs:StopTask, ecs:RegisterTaskDefinition, ecs:DescribeTaskDefinition, ecs:DescribeContainerInstances, and the AmazonEC2ContainerServiceforEC2Role) as well as a cluster named ecs-functional-tests in order to actually run the tests. You'll also want to either build the Agent locally or re-tag one that you've pulled to amazon/amazon-ecs-agent:make since that's the image the tests expect to use.

Edit: Fixed some incorrect policy actions.

[rjschwei]

On 08/11/2015 03:13 PM, Samuel Karp wrote:

I've pushed a set of functional tests to the dev branch of the Agent (see aws/amazon-ecs-agent#161, thanks @euank for most of the work!), which should do a decent job of testing whether the Agent functions correctly. Note that you will need a fairly permissive role policy on the instance to run these tests (not sure of the complete set offhand, but at least ecs:StartTask, ecs:DescribeTask, ecs:StopTask, ecs:RegisterTaskDefinition, ecs:DescribeTaskDefinition, ecs:DescribeContainerInstance, and the AmazonEC2ContainerServiceforEC2Role) as well as a cluster named ecs-functional-tests in order to actually run the tests. You'll also want to either build the Agent locally or re-tag one that you've pulled to amazon/amazon-ecs-agent:make since that's the image the tests expect to use.

Great, thanks. Will not get to this right away thus I might come back
with more questions.

Thanks for the help,
Robert

Robert Schweikert MAY THE SOURCE BE WITH YOU
Public Cloud Architect LINUX
[email protected]
IRC: robjo

[samuelkarp]

Awesome. I'm going to close this for now, but feel free to either re-open or open a new issue when you have future questions.

[schaefi]

I tried to run the tests from the dev branch on our ecs instance with docker 1.7, however I have problems running the tests because of a missing go package requirement. I got the following:

./scripts/test 
+ make get-deps
go get github.com/tools/godep
go get golang.org/x/tools/cover
go get golang.org/x/tools/cmd/cover
+ make gotest
GOPATH="/home/ec2-user/amazon-ecs-init/ecs-init/Godeps/_workspace:/home/ec2-user/golang" go test -short -v -cover ./...
ecs-init/ecs-init.go:21:2: cannot find package "github.com/aws/amazon-ecs-init/ecs-init/config" in any of:
        /usr/lib64/go/src/pkg/github.com/aws/amazon-ecs-init/ecs-init/config (from $GOROOT)
        /home/ec2-user/amazon-ecs-init/ecs-init/Godeps/_workspace/src/github.com/aws/amazon-ecs-init/ecs-init/config (from $GOPATH)

anything I did wrong here ?

Thanks

[samuelkarp]

It looks like you didn't check out the package into your GOPATH. From the output you've listed, it looks like you've checked it out into /home/ec2-user/amazon-ecs-init, but your GOPATH is set to /home/ec2-user/golang. Can you try checking it out into /home/ec2-user/golang/src/github.com/aws/amazon-ecs-init instead?

[samuelkarp]

Also, note that my comment above about tests is referring to the ECS Agent repo rather than this repo, and that the tests have been merged into master of that repo.

[schaefi]

ok great that solves it, I checked it out to ~/golang/src/github.com/aws/amazon-ecs-agent
of course you are right for testing the agent I should better checkout the agent code :-)

when running the tests I got some error but this is due the environment as you mentioned. I'm looking into this now but might come back with further questions if that's ok for you ?

Thanks for your help

[schaefi]

I called

make test-in-docker

and got:

...
# Privileged needed for docker-in-docker so integ tests pass
docker run -v "/home/ec2-user/golang/src/github.com/aws/amazon-ecs-agent:/go/src/github.com/aws/amazon-ecs-agent" --privileged "amazon/amazon-ecs-agent-test:make"
go get github.com/tools/godep
go get golang.org/x/tools/cmd/cover
go get github.com/golang/mock/mockgen
go get golang.org/x/tools/cmd/goimports
cd misc/netkitten; make 
make[1]: Entering directory `/go/src/github.com/aws/amazon-ecs-agent/misc/netkitten'
Cannot connect to the Docker daemon. Is 'docker -d' running on this host?

docker runs on this host

6132 ?        Ssl    1:20 /usr/bin/docker -d -H fd://

and the testing procedure already communicated with docker, a new registry container was started

docker ps

3e0f76627601        registry           "docker-registry"   13 minutes ago      Up 13 minutes      127.0.0.1:51670->5000/tcp    test-ecs-registry   

8717eeb74a6e        amazon/amazon-ecs-agent:latest   "/agent"            21 hours ago        Up 21 hours         127.0.0.1:51678->51678/tcp   ecs-agent

An already running ecs-agent should not conflict with the testing, should it ?

[samuelkarp]

Thanks for reporting; it looks like test-in-docker is actually broken for me as well. I've been using make test lately for the unit tests and integration tests; that should still be working properly. We'll look into test-in-docker and see if we can fix it. Note that the functional tests, per my previous comment, should still be run with make run-functional-tests on an EC2 instance with appropriate permissions.

[schaefi]

The make test target on my system is complaining as follows:

...
open $WORK/github.com/docker/libcontainer/system/_obj/_cgo_gotypes.go: No such file or directory
go tool: no such tool "cover"; to install:
        go get code.google.com/p/go.tools/cmd/cover
godep: go exit status 3
Makefile:78: recipe for target 'test' failed

However the cover tool is provided and in the path

ls -l ~/golang/bin/cover 
   -rwxr-xr-x 1 ec2-user users 5399480 Sep  9 08:09 /home/ec2-user/golang/bin/cover

I didn't dive deeper into this because I'm actually most interested in the functional tests. Thus I started 'make run-functional-tests' which is still running. In my ecs-functional-test cluster I can see
that a new container instance was registered, thus it seems to do something...

Thanks

[schaefi]

Hmm run-functional-tests failed with

=== RUN TestTest
--- PASS: TestTest (0.00 seconds)
=== RUN TestRunManyTasks
panic: test timed out after 20m0s

followed by a stack trace

When I called the tests I saw that a new container was registered in the ecs-functional-test cluster

Sorry still no luck

[samuelkarp]

@schaefi What was the policy attached to your instance when you tried make run-functional-tests? As I noted above, you'll need a fairly permissive policy to allow the tests to actually run correctly (the functional tests interact directly with the ECS API). You might try watching docker ps in another terminal while the tests are running; you should see a number of containers created and stopped as part of the functional tests.

With respect to the cover tool, which version of Go are you using? It looks like the version of Go you have installed expects the cover tool to be located at code.google.com/p/go.tools/cmd/cover, which is no longer accurate (it should now be at golang.org/x/tools/cmd/cover).

[schaefi]

Here is the role policy I have setup according to your information:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecs:CreateCluster",
                "ecs:DeregisterContainerInstance",
                "ecs:DiscoverPollEndpoint",
                "ecs:Poll",
                "ecs:RegisterContainerInstance",
                "ecs:Submit*",
                "ecs:StartTask",
                "ecs:DescribeTask",
                "ecs:StopTask",
                "ecs:RegisterTaskDefinition",
                "ecs:DescribeTaskDefinition",
                "ecs:DescribeContainerInstance"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

The go version installed is

go-1.3.1-2.7.x86_64

[samuelkarp]

Thanks. It looks like my original comments were slightly wrong. You'll want to change ecs:DescribeTask to ecs:DescribeTasks, change ecs:DescribeContainerInstance to ecs:DescribeContainerInstances, and add ecs:StartTelemetrySession. Hopefully this should be enough to unblock you. If you still have issues, I can follow up with a more complete list (and get it correctly documented) but you should be able to unblock yourself by using ecs:* if absolutely necessary.

If I remember correctly, the cover tool changed between Go 1.3 and 1.4. If you don't want to upgrade Go versions, you should still be able to have it function correctly by removing the -cover argument from the test command in the Makefile.

[schaefi]

Hey good news, tests passed now :)

Thanks a ton for your help I would never have found out these role setup

[samuelkarp]

Great! Please let us know if you need any more help.