swipely/iam-docker

Use different IAM roles for each Docker container on an EC2 instance

iam-docker's Issues

New versions of logrus use a lowercase "sirupsen" import path

This was a change made in logrus a while back. It's a relatively simple change to update the import paths and use the new lowercase version which will allow unimpeded updates going forward. I can provide a PR.

Consider AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Amazon ECS addresses the container credentialing problem in a different way. See IAM Roles for Tasks for details. In this Amazon ECS model, containers are launched with the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable.

Every client application is supposed to look for this environment variable. If it's set, the client is to request credentials from http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI} instead of the usual /latest/meta-data/iam/security-credentials.

This change is recent, having landed in the various SDKs' default credentialing chains about a month ago. It will be some time before support for this environment variable is universal.

So: what, if anything, should iam-docker do differently?

Some ideas on possible things to do:

1. iam-docker can warn if it detects the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable on a new container. This seems appropriate, since this variable is set by magic and therefore iam-docker can't guarantee that the container will receive correct credentials.
2. iam-docker can intercept requests for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and serve them identically to /latest/meta-data/iam/security-credentials. This ensures that it handles all credentialing requests from a container, regardless of the container credentials relative URI.
3. iam-docker can proxy requests for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. This would allow use of iam-docker for some containers while using the official Amazon functionality for others.
4. iam-docker can provide per-container statistics, like { credentials_requested: 4, container_credentials_uri_set: true, container_credentials_requested: 0 }. This would assist administrators in determining which containers are aware of this variable and which are not.

Assuming roles won't work with custom network interfaces

Running containers with custom networks will result in the IPAddress field being empty

$ docker run --net local -l com.swipely.iam-docker.iam-profile=arn:aws:iam::************:role/some-role tutum/curl bash -c 'while true; do sleep 2; done' 
$ docker inspect bd46f10c6b87
[
        ....,
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "a46ba1358bd0d7096e7f054cbaf6162fb2353ac519f324417fab6d567b6bf65b",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/a46ba1358bd0",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "local": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "cf66e08b22506bf00ccc46dae42df46531628b7413a058c202983ebcff4d7957",
                    "EndpointID": "4c57b930b76089ce5ef655b1561cc283287c2d16459fb4a29adbc4050a914e21",
                    "Gateway": "192.168.0.1",
                    "IPAddress": "192.168.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:00:02"
                }
            }
        }
    }
]

which in turn will result in iam-docker not being able to find the container

2016-07-06T18:34:09Z [docker] Adding container event-handler=2 id=bd46f10c6b87784440b5cfb1bd34445292e5b6d7bcfea05be36fa28b30d5761e event=start
2016-07-06T18:34:09Z [docker] Fetching credentials event-handler=2 id=bd46f10c6b87784440b5cfb1bd34445292e5b6d7bcfea05be36fa28b30d5761e event=start role=arn:aws:iam::************:role/some-role
2016-07-06T18:20:56Z [iam] Credential successfully refreshed arn=arn:aws:iam::************:role/some-role
2016-07-06T18:38:00Z [http] Unable to find credentials method=GET remoteAddr=192.168.0.2:49782 error="Unable to find container for IP: 192.168.0.2" path=/latest/meta-data/iam/security-credentials/

Debug output shows the IP is clearly not set when creating the config in docker/container_store.go, line 171.
As stated in moby/moby#21658 the toplevel IPAddress will only be set for the default bridge device.

Our current use case is a docker swarm. This makes it practically impossible to use iam-docker with docker-compose files and overlay networks.

I propose an additional label to set the network interface and request the IPAddress from the specified interface instead of the toplevel IPAddress field, probably with a fallback to "bridge".

p.s. I might be able to supply a patch until next week, however I never worked with go and would be happy to accept another solution instead.

Multiple proxies for multiple roles.

I run some workloads outside ec2 but I need each container have its own set of permissions. On the doc is not so clear how to get that configuration. It is just use a different instance of iam-docker with its own key pair and port.
This config let you keep the minimum permissions principle also outside ec2.

Would be nice to have a example showing how to run it in different ports. There is no mention about that possibility in the docs

$ PORT="40100"`
$ iptables -t nat \
-I PREROUTING \
-p tcp \
-d 169.254.169.254 \
--dport 80 \
-j REDIRECT \
--to-ports "$PORT" \
-i "$INTERFACE"
$ docker run --name iam_proxy -d \
-e AWS_ACCESS_KEY_ID=key4role1 \
-e AWS_SECRET_ACCESS_KEY=secret4role1 \
-e AWS_DEFAULT_REGION=eu-west-1 \
--volume /var/run/docker.sock:/var/run/docker.sock \
--net=host \
--entrypoint "/iam-docker" \
swipely/iam-docker:latest --listen-addr ":${PORT}"

Of course each application requires its own iptables rule.

[Need help] Can't run the first docker command

I have followed the Usage README, set sts:assume-role and trust Relationship to be assumed by the root role.

I currently login the ec2 instance which has the iam role with above changes.

But when I run the first docker command (docker run --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --net=host swipely/iam-docker:latest), I got below error:

2018-07-02T05:14:59Z [docker] Starting event handler event-handler=1
2018-07-02T05:14:59Z [docker] Starting event handler event-handler=2
2018-07-02T05:14:59Z [docker] Starting event handler event-handler=3
2018-07-02T05:14:59Z [docker] Adding container event-handler=4 id=d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337 event=start
2018-07-02T05:14:59Z [docker] Unable to add container event-handler=4 id=d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337 event=start error="Unable to find label named 'com.swipely.iam-docker.iam-profile' or environment variable 'IAM_ROLE' for container: d50238ea7d161e5ed89844f638d7e10cf0750049678c237c1c11bc86640c2337"
panic: runtime error: index out of range

error="Unable to find label named 'com.swipely.iam-docker.iam-profile' or environment variable 'IAM_ROLE' for container:

then I feed environment variable iam-profile into container:

export PROFILE="arn:aws:iam::1234567:role/iam_role_dev "

docker run -e IAM_ROLE="$PROFILE"  --volume /var/run/docker.sock:/var/run/docker.sock --restart=always --net=host swipely/iam-docker:latest

I got another error

2018-07-02T05:26:01Z [docker] Adding container event-handler=4 id=2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534 event=start
2018-07-02T05:26:01Z [docker] Unable to add container id=2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534 event=start event-handler=4 error="Unable to find IP address for container: 2fd5a4a2aa09dd0cf9d733bc8977c718929dc5247d60a332c5468cf9141d3534"
panic: runtime error: index out of range

error="Unable to find IP address for container

how to fix the issue?

I already run the docker command with root permission.

Conslution

1. Only image swipely/iam-docker:v1.0.0 works in my enviroinment. Tags with latest, v1.2.0 and v1.1.0 doesn't work.
2. Make sure you set the assume role on both (instance iam role and the role which container will be assumed)
3. Make sure port 8080 is not used by the host, otherwise, use other ports (reference: #25 (comment))

Service Stack Networks

When spinning up a new service stack it creates a default network for your stack and the IPs allocated to that stack are for internal use only. If you need access outside of the stack it utilizes the docker_gwbridge (by default the 172.18.x.x network) which is a separate interface in the container but it is an interface that is not represented in the docker inspect command. This has me in a situation where I can get my traffic to the iam-proxy container but it fails with the error:

2017-03-30T15:53:00Z [http] Unable to find credentials path=/latest/meta-data/iam/security-credentials/my-role method=GET remoteAddr=172.18.0.4:49468 error="Unable to find container for IP: 172.18.0.4"

Does the service stack network model break iam-proxy beyond repair, or at least without a change to docker itself? Is there a means of tying the meta-data request back to the container without using IP? I don't immediately see a way but I was hoping you guys have given it some thought.

Working with a K8s cluster

When I tried to deploy it into a K8s cluster, I got logs liek the following

2016-10-21T12:25:16Z [app] Running the app
2016-10-21T12:25:16Z [app] Starting worker=event-handler
2016-10-21T12:25:16Z [docker] Starting event handler event-handler=4
2016-10-21T12:25:16Z [app] Starting worker=sync-containers
2016-10-21T12:25:16Z [app] Syncing containers worker=sync-containers
2016-10-21T12:25:16Z [docker] Syncing the running containers
2016-10-21T12:25:16Z [app] Starting worker=refresh-credentials
2016-10-21T12:25:16Z [app] Starting worker=http
2016-10-21T12:25:16Z [docker] Starting event handler event-handler=1
2016-10-21T12:25:16Z [docker] Starting event handler event-handler=2
2016-10-21T12:25:16Z [docker] Starting event handler event-handler=3
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=3
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=4
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=1
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=2
2016-10-21T12:25:17Z [app] Exited worker=event-handler error="Docker events connection closed"
2016-10-21T12:25:17Z [docker] Starting event handler event-handler=4
2016-10-21T12:25:17Z [docker] Starting event handler event-handler=1
2016-10-21T12:25:17Z [docker] Starting event handler event-handler=2
2016-10-21T12:25:17Z [docker] Starting event handler event-handler=3
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=3
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=4
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=1
2016-10-21T12:25:17Z [docker] Docker events channel closed event-handler=2
2016-10-21T12:25:17Z [app] Exited worker=event-handler error="Docker events connection closed"

Looks like not being able to get proper docker events.

Have anyone had similar experience please ?

Thanks

Ken

side effects to use "iam-docker"

After fix my issue (#25), I can assume the role with --label com.swipely.iam-docker.iam-profile="$PROFILE"

Then I found a problem.

Before using iam-docker, the containers inherit permissions from its host directly. So if host has permission to list s3 bucket, the containers running on it can as well.

But after enable iam-docker, I have to feed all permission to that container, otherwise, it will have no any permission.

Is this the design or we can do some improvements on this issue?

Second, can I feed several --label options, so I can group the permission in multiple roles, more than I have to put all permission into one role?

Support for EXTERNAL_ID for IAM Roles

AWS allows cross-account roles to use an external ID as an additional security mechanism for accessing role permissions. Can we add this as another option?

iam-docker fails to handle GET /latest/meta-data/iam/security-credentials

I have an app using a current AWS SDK for Go that fails to get credentials from iam-docker. Enabling WithCredentialsChainVerboseErrors(true) produces:

NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: EC2MetadataError: failed to make EC2Metadata request

iam-docker --disable-upstream logs say:

[http] Denying non-IAM endpoint request method=GET remoteAddr=172.17.0.2:34204 path=/latest/meta-data/iam/security-credentials

tcpdump -i docker0 confirms that Go is sending GET /latest/meta-data/iam/security-credentials HTTP/1.1 and iam-docker is returning HTTP/1.1 403 Forbidden.

iam-docker should treat this as a "list roles" request, even though it lacks a trailing slash.

Some issues between v1.0.0 and v1.1.0

I successfully use iam-docker roles for containers running in a Rancher v1.6.x platform finally, but only with an old version v1.0.0, no luck with v1.1.0 and v1.2.0.

Please reference the issue #25 for details.

I don't want to miss the new features and need understand what's the problem between the commits from v1.0.0 to v1.1.0, because I got this issue start from v1.1.0

The only notable change is about new IAM_ROLE environment variable (26680c8)

Seems some bugs in below codes, which was added into v1.1.0

https://github.com/swipely/iam-docker/blame/master/src/docker/container_store.go#L177-L185

-       iamRole, hasKey := container.Config.Labels[iamLabel]
-       if !hasKey {
-               return nil, fmt.Errorf("Unable to find label named '%s' for container: %s", iamLabel, id)
+       iamRole, hasLabel := container.Config.Labels[iamLabel]
+       if !hasLabel {
+               env := dockerClient.Env(container.Config.Env)
+               envRole := env.Get(iamEnvironmentVariable)
+               if envRole != "" {
+                       iamRole = envRole
+               } else {
+                       return nil, fmt.Errorf("Unable to find label named '%s' or environment variable '%s' for container: %s", iamLabel, iamEnvironmentVariable, id)
+               }

@willglynn

Could you take a look?

unable to use IAM docker from outside ec2

Hi ,
i was trying to use IAM docker from outside ec2...
(from inside ec2 everything was working fine)

when i configured it from outside ec2 i encountered the following error on the docker running aws-cli commands:
Unable to locate credentials. You can configure credentials by running "aws configure".

when i started iam docker i used the following run command:
sudo docker run -e AWS_ACCESS_KEY_ID='ACCESS KEY' -e AWS_SECRET_ACCESS_KEY='SECRET KEY' --volume /var/run/docker.sock:/varun/docker.sock --restart=always --net=host IAM_DOCKER_IMAGE -listen-addr=0.0.0.0:9999

i configured iptables as required on the documentation to forward the request to the iam-docker.

in order to debug the issue i started watching the traffic coming from the aws-cli docker.
its being correctly redirected to the iam-docker.

inside the iam docker i can see that credentials are loaded correctly:
2019-07-29T08:56:09Z [iam] Checking for stale credential arn=MY_ROLE
2019-07-29T08:56:09Z [iam] Credential is fresh arn=MY_ROLE
2019-07-29T08:56:09Z [iam] Done refreshing all IAM credentials

then i decided to execute tcp dump inside iam-docker to see what is happening there , and i can see that its unable to reach aws metadata url:
08:38:35.398839 IP 172.17.0.2.48848 > 169.254.169.254.80: Flags [S], seq 2662688784, win 29200, options [mss 1460,sackOK,TS val 17010679 ecr 0,nop,wscale 7], length 0
08:38:35.398959 IP 169.254.169.254 > 172.17.0.2: ICMP host 169.254.169.254 unreachable - admin prohibited, length 68

now my questions are:

1. why does it even try to go to fetch metadata if i provided AWS credentials to start with?

2. why doesnt iam-docker show an error message without using tcpdump?
   its hard to debug , i thought i have an issue getting from the aws-cli docker to iam-docker...
   as it turned out that iam-docker is not responding correctly to aws-cli docker...

Godep tool is "archived"

While working on #29 I found https://github.com/tools/godep is archived and it is recommended to use dep instead. As far as I can tell, the only potential issue in switching to dep is that it requires Go 1.9 instead of the current Go 1.6 requirement.

Support for Container in host mode

Current solution says to get network interface form the command, but for containers launched using --net=host it will return blank. So this will not work. Do we have support for host mode containers.

Having a hard time with my instance role.

Would it be possible to get a complete example of roles and policies? I am struggling with this quite a bit.

I am currently creating an instance role "MyInstanceRole" with the following attached policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::XXXXXXXXXXX:role/s3_access"
            ]
        }
    ]
}

The s3_access role has a normal S3 access policy and the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:sts::XXXXXXXXXXX:role/MyInstanceRole"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Running:

docker run -i -t --label com.swipely.iam-docker.iam-profile=arn:aws:iam::XXXXXXXXXXX:role/s3_access fstab/aws-cli /home/aws/aws/env/bin/aws s3 ls s3://my_bucket/

A client error (AccessDenied) occurred when calling the ListObjects operation: Access Denied

iam-docker container log

7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Handling event event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Adding container event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Attempting to add container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Inspecting container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Adding new container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a role=arn:aws:iam::XXXXXXXXXXX:role/s3_access ip=172.17.0.8
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Looking up IAM role id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [docker] Fetching credentials event-handler=1 id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a event=start role=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [iam] Checking for stale credential arn=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:15 AM2016-07-15T15:08:15Z [iam] Credential is fresh arn=arn:aws:iam::XXXXXXXXXXX:role/s3_access
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Handling event event-handler=4 event=die id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Removing container event-handler=4 event=die id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a
7/15/2016 11:08:17 AM2016-07-15T15:08:17Z [docker] Removing container id=1d640fbc7a76c15d0e7ff3ff35ef8317470758880a0d1717213f63841f48242a

If I attach the s3 access policy directly to the instance role, I can list the bucket content.

Would you be kind enough to explain what I am missing?