awslabs / amazon-ecs-local-container-endpoints Goto Github PK
View Code? Open in Web Editor NEWA container that provides local versions of the ECS Task Metadata Endpoint and ECS Task IAM Roles Endpoint.
License: Apache License 2.0
A container that provides local versions of the ECS Task Metadata Endpoint and ECS Task IAM Roles Endpoint.
License: Apache License 2.0
Hello!
Currently the dockerhub image is amd64 only. Would it be possible to push a multi-arch (amd64 + arm64) manifest so as to support Graviton/Graviton2-based instances?
Thanks!
Endpoint /creds
works just fine, but when I try to use /role/<role name>
like it says in this guide, the ecs-local-endpoints container keeps responding with the error:
ecs-local-endpoints_1 | time="2021-12-02T21:21:07Z" level=error msg="HTTP 500 - InvalidClientTokenId: The security token included in the request is invalid\n\tstatus code: 403, request id: 080b81eb-c67d-4b47-932f-276bb9c1d480"
See my docker-compose.override.yml:
version: "2"
networks:
# This special network is configured so that the local metadata
# service can bind to the specific IP address that ECS uses
# in production
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
# This container vends credentials to your containers
ecs-local-endpoints:
# The Amazon ECS Local Container Endpoints Docker Image
image: amazon/amazon-ecs-local-container-endpoints:1.4.0-amd64
volumes:
# Mount /var/run so we can access docker.sock and talk to Docker
- /var/run:/var/run
# Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
# On Windows, this directory can be found at "%UserProfile%\.aws"
- $HOME/.aws/:/home/.aws/
environment:
# define the home folder; credentials will be read from $HOME/.aws
HOME: "/home"
# You can change which AWS CLI Profile is used
AWS_PROFILE: "default"
networks:
credentials_network:
# This special IP address is recognized by the AWS SDKs and AWS CLI
ipv4_address: "169.254.170.2"
# Here we reference the application container that we are testing
# You can test multiple containers at a time, simply duplicate this section
# and customize it for each container, and give it a unique IP in 'credentials_network'.
app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-gov-west-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/test-role"
Extra info: I'm operating with an assumed role session principal. I.e., when I get my caller identity on the host, I get:
$ aws sts get-caller-identity
{
"UserId": "XXXXXXXXXX",
"Account": "XXXXXXXXXX",
"Arn": "arn:aws-us-gov:sts::XXXXXXXXXX:assumed-role/PowerUsers/XXXXXXXXXX"
}
I believe I'm having a similar issue to #89 but it's different enough that I'm opening a new issue. (Somewhat similar to #118 and maybe #35)
As snehal-kolte mentioned, I'm following the same document as testing containers: https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/
And also referencing the docs for how to vend credentials that are in this repo: https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/docs/features.md#vend-credentials-to-containers
Most of the time there are no issues but on occasion the following error appears in the logs:
{"level":"error","message":"Unhandled promise rejection: CredentialsProviderError: Could not load credentials from any providers","meta":{},"timestamp":"2022-03-22T21:26:36.174Z"}
You have triggered an unhandledRejection, you may have forgotten to catch a Promise rejection:
CredentialsProviderError: Could not load credentials from any providers
at providers (/<project-dir>/<project-dir>/node_modules/@aws-sdk/credential-provider-node/dist-cjs/index.js:25:19)
at /<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/chain.js:11:28
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async coalesceProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:13:24)
at async SignatureV4.credentialProvider (/<project-dir>/<project-dir>/node_modules/@aws-sdk/property-provider/dist-cjs/memoize.js:32:24)
at async SignatureV4.signRequest (/<project-dir>/<project-dir>/node_modules/@aws-sdk/signature-v4/dist-cjs/SignatureV4.js:84:29)
at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:13:18
at async StandardRetryStrategy.retry (/<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-retry/dist-cjs/StandardRetryStrategy.js:51:46)
at async /<project-dir>/<project-dir>/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:6:22
The workaround has been running the same command again to hit our endpoint and it will always work (we never see the error back-to-back).
My docker-compose.override.yml
:
version: "2"
networks:
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
- /var/run:/var/run
- $HOME/.aws/:/home/.aws/
environment:
AWS_PROFILE: "$AWS_PROFILE"
HOME: "/home"
networks:
credentials_network:
ipv4_address: "169.254.170.2"
app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/$APP_AWS_TASK_ROLE"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2"
ECS_CONTAINER_METADATA_URI_V4: "http://169.254.170.2/v3"
amazon-ecs-local-container-endpoints
.It seems this exporter can't collect MSK metrics ?
It would be great to publish this to the ECR Public Gallery: https://gallery.ecr.aws/
This would especially useful for anyone using Docker Hub anonymously or from a free account, in light of the new pull rate limits imposed.
Is there a schedule or plan to release new builds with recent changes to Docker Hub?
I've very modified the example to use a profile from my local config:
environment:
# define the home folder; credentials will be read from $HOME/.aws
HOME: "/home"
# You can change which AWS CLI Profile is used
AWS_PROFILE: "assumed-profile"
To authenticate as assumed profile at the command line I would need to provide an MFA code. My config looks something like:
[profile assumed_profile]
role_arn = arn:aws:iam::etc
mfa_serial = arn:aws:iam::etc
source_profile = default
region = eu-west-2
output = json
Running my docker compose file now I get the error:
Failed to create Credentials Service: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set.
How do i tell the local endpoint the MFA code so that it can assume the role?
The docs recommend mounting ~/.aws
into the container, but this may not work for a variety of reasons (e.g., a custom credential process that's not accessible to the container). An alternative I've discovered is to provide a local IMDSv2 server on the host and use the AWS_EC2_METADATA_SERVICE_ENDPOINT
environment variable on the container.
I've included this capability in aws-export-credentials
. It looks like this:
# in a terminal (choosing 8081 as the port for this example)
$ aws-export-credentials --imds 8081
then in your docker-compose.override.yml
from the tutorial, the relevant section would look something like this:
services:
# This container vends credentials to your containers
ecs-local-endpoints:
# The Amazon ECS Local Container Endpoints Docker Image
image: amazon/amazon-ecs-local-container-endpoints
volumes:
# Mount /var/run so we can access docker.sock and talk to Docker
- /var/run:/var/run
ports:
# Map the IMDS server's port from the host
- "8081:8081"
environment:
# use credentials from the IMDS server on the host
AWS_EC2_METADATA_SERVICE_ENDPOINT=http://host.docker.internal:8081/
networks:
credentials_network:
# This special IP address is recognized by the AWS SDKs and AWS CLI
ipv4_address: "169.254.170.2"
The "Generic Metadata Injection" section within the "features" document contains the wrong url to the generic example
Broken Link (last line)
See example for overriding task metadata response [here](../example/generic).
Should be set to:
See example for overriding task metadata response [here](../examples/generic).
๐ Hey all, thanks for making this image available.
Is there any planned support for V4 metadata endpoints? Do we have an ETA on this?
Thanks
Just wondering if there's any plans to support v4 of the ecs task metadata endpoint in the near future. Thanks
Are there plans to add the v1 endpoint?
I know that it's very different in the way that it's expose to the tasks since it's the agent directly, but I have a use case for it and was wondering if there are plans to add it.
Thx
I have some issues with setting it up correctly. Credentials are in place but Expiration is always empty and exceptions are thrown because of that.
I'm trying to use this image for the most basic scenario to debug application which uses dynamodb.
I've tried to add manually expiration to credentials file but didn't work.
Example response from /creds:
{ "AccessKeyId": "not empty", "Expiration": "", "RoleArn": "", "SecretAccessKey": "not empty", "Token": "not empty" }
Team,
We are referring the https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/ and set up docker-compose.yml and docker-compose.override.yml like below
docker-compose.yml
version: "2"
services:
app:
build: my_image
ports:
- 8080:80
environment:
PORT: "80"
docker-compose.override.yml
version: "2"
networks:
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
volumes:
- %UserProfile%\.aws
environment:
AWS_PROFILE: "default"
networks:
credentials_network:
# This special IP address is recognized by the AWS SDKs and AWS CLI
ipv4_address: "169.254.170.2"
app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_REGION: "us-east-1"
ABBREVIATED_REGION: "us01"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds"
Error:
Caused by: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@29c5ee1d: profile file cannot be null, WebIdentityTokenCredentialsProvider: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@59429fac: Failed to connect to service endpoint: ]
@hencrice Is there something missing in my configuration set up?
credentials file look like below
[default]
aws_access_key_id= xyz
aws_secret_access_key= xyz
aws_security_token="xyz"
aws_session_token="xyz"
Hi,
I'm trying to use the /role-arn/{role arn}
option for the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
. If I try using a role with a path in it e.g. arn:aws:iam::111122223333:role/division_abc/subdivision_xyz/custom_role
it does not appear to work. I believe it is due to the Gorilla Mux path pattern used at https://github.com/awslabs/amazon-ecs-local-container-endpoints/blob/mainline/local-container-endpoints/config/config.go#L73, it doesn't allow expansion to multiple /
.
I checked out the code and added a simple expansion RoleArnCredentialsPath = "/role-arn/{roleArn}/{roleName:.+}"
and this fixed the problem. I did need to update the STS role session name to exclude any /
as this is not allowed in the role session name.
Am I understanding this correctly or am I missing something? I can create a PR with a fix if needed.
This is a great project!
I'm just curious if there is a functional reason why the tutorial hasn't been updated to specify version 3 of the docker-compose files. I was able to remove the 'gateway' configuration of the credentials network (no longer honored in v3 of docker-compose) and the local container endpoint continued to vend my ECS task credentials perfectly. I was curious if other use cases aren't supported by the new docker-compose configuration schema or if the documentation is simply behind.
Thanks!
Same issue as here #23
This is path 1b (grabbing a temporary credential and stashing it in ~/.aws/credentials
) from issue #23 -- it makes Boto unhappy.
Retrieving the credentials, as @PettitWesley alludes to, doesn't include an expiration date because this isn't a thing in ~/.aws/credentials
:
% curl http://169.254.170.2/creds
{"AccessKeyId":"ASIA----------EGKVOM","Expiration":"","RoleArn":"","SecretAccessKey":"4su0....","Token":"FQoG...."}
Boto barfs on the empty Expiration
field:
# export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
# python3
Python 3.7.4 (default, Jul 30 2019, 19:56:38)
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> boto3.__version__
'1.9.246'
>>> boto3.client("ssm")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
return _get_default_session().client(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 827, in create_client
credentials = self.get_credentials()
File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 426, in get_credentials
'credential_provider').load_credentials()
File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1934, in load_credentials
creds = provider.load()
File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1797, in load
return self._retrieve_or_fail()
File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1812, in _retrieve_or_fail
expiry_time=_parse_if_needed(creds['expiry_time']),
File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 196, in _parse_if_needed
return parse(value)
File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1358, in parse
return DEFAULTPARSER.parse(timestr, **kwargs)
File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 652, in parse
raise ValueError("String does not contain a date:", timestr)
ValueError: ('String does not contain a date:', '')
My suggestion is to include expiration
as a valid field in the ~/.aws/credentials
file -- it really ought to be there for temporary credentials like this.
I'm fairly certain the session token does include an expiration date in there, but it's not for consumption outside of the AWS Auth service. (I'm fairly certain it's encrypted, and if we could decrypt it anyone could also go about generating secret keys and session tokens, which would be... bad. The Auth team would know for sure, though.)
Attempting to run the ecs-local-endpoints container on a system behind a corporate proxy that does SSL interception wtih self-signed certificates results in a sequence of errors of the form (excluding timestamp):
level=error msg="HTTP 500 - RequestError: send request failed\ncaused by: Post https://sts.amazonaws.com/: x509: certificate signed by unknown authority"
Other AWS tools (e.g., the AWS CLI and Python SDK) provide mechanisms to disable certificate verification, but no such mechanism appears to be included for the ecs-local-countainers-endpoints.
Use case:
I am working on an application that gets deployed to ECS (Fargate). Our team has had good success using Docker Compose and localstack to simulate an environment with DynamoDB, S3, and SQS while developing/testing locally. I just started working on a reusable library that will extract container metrics from the ECS Task Metadata Endpoint. I was excited after finding this project, and eager to plug it into our existing Docker Compose script to better simulate the ECS environment.
It appears the current version only works with legit AWS credentials. We have some automated build agents, test scripts, and team members that don't have access to AWS. Instead of having to provision and manage this additional access, it would be great if this container were configurable to pipe the IAM and STS requests to the localstack services running in our Docker Compose network.
I'm not very familiar with the inner-workings of this container. I'm not sure if any additional service-specific endpoints should be open to customization beyond IAM and STS.
Within the example Local Stack docker-compose file, custom iam and sts endpoint environment variables are passed as IAM_CUSTOM_ENDPOINT
and STS_CUSTOM_ENDPOINT
although there are not supported within the image which results in the image to route requests to AWS. Looking at the config.go file within the mainline
and v1.4.0
the correct environment variables should be IAM_ENDPOINT
and STS_ENDPOINT
to enable custom endpoints.
Hey,
For our particular setup, which requires the use of MFA, I get
ecs-local-endpoints_1 | time="2019-10-24T14:23:00Z" level=fatal msg="Failed to create Credentials Service: AssumeRoleTokenProviderNotSetError: assume role with MFA enabled, but AssumeRoleTokenProvider session option not set."
Is there an alternative way to do this?
Cheers!
I cannot perform iam:* commands (I tried create-role, delete-role, or list-groups) when the request is signed by a token issued by my ecs-local-container-endpoints via AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
I first encountered this using AWS SDK for Java 1.11.551 and then replicated it on aws-cli/1.16.148. I'm using endpoints 1.0.1 but 1.0.0 didn't work either.
Steps to reproduce using aws-cli/1.16.158 Python/3.6.8 Linux/5.0.13-arch1-1-ARCH botocore/1.12.148 are as follows:
# setup ~/.aws
aws configure
# verify role creation works with standard policy
aws iam create-role --role-name joe-test --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'
# break local .aws and enable ecs endpoints
mv ~/.aws ~/.aws-nope
export AWS_DEFAULT_REGION=us-east-1
export AWS_REGION=us-east-1
export AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
docker run -d -p 51679:51679 -v /var/run:/var/run -v $HOME/.aws-nope/:/home/.aws/ -e "ECS_LOCAL_METADATA_PORT=51679" --name ecs-local-endpoints amazon/amazon-ecs-local-container-endpoints:latest
# verifiy token service is working
aws s3 ls
# create a new role but it's busted
aws iam create-role --role-name joe-test-broken --assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":["ec2.amazonaws.com"]},"Action":["sts:AssumeRole"]}]}'
An error occurred (InvalidClientTokenId) when calling the CreateRole operation: The security token included in the request is invalid
# cannot delete other role
aws iam delete-role --role-name joe-test
An error occurred (InvalidClientTokenId) when calling the DeleteRole operation: The security token included in the request is invalid
# fix the cli
mv ~/.aws-nope ~/.aws
# delete the example role
aws iam delete-role --role-name joe-test
Running it in debug the response was:
2019-05-14 21:47:11,248 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': '5b2f97bc-76b3-11e9-95f1-97b56e28630b', 'Content-Type': 'text/xml', 'Content-Length': '305', 'Date': 'Wed, 15 May 2019 01:47:11 GMT'}
2019-05-14 21:47:11,248 - MainThread - botocore.parsers - DEBUG - Response body:
b'\n \n Sender\nInvalidClientTokenId
\n The security token included in the request is invalid\n \n 5b2f97bc-76b3-11e9-95f1-97b56e28630b\n\n'
I found this because my local CodeBuild was failing. I pulled a wire debug from a local build and compared it to the wire on CodeBuild and the requests look good.
Environment | amz-sdk-invocation-id | x-amzn-RequestId | Result |
---|---|---|---|
ECS Endpoints | 71988628-acf5-0d1b-b341-9382c38a7e93 |
076d4513-767a-11e9-89db-c3541b5d08ec |
InvalidClientTokenId |
CodeBuild | 00e8eab6-44c2-1126-224e-d5c9772e105f |
fab50e20-7688-11e9-9527-27af0247c969 |
Success (CreateRoleResponse) |
'sh' executable is required by aws-sdk-go to invoke credential_process.
See related PR for more context.
if you using multiple sso profiles and use the sso_session config in .aws/config it currently fails to correctly use a shared sso session.
version 1.44.298 of the AWS go sdk introduced support for sso_session https://github.com/aws/aws-sdk-go/releases/tag/v1.44.298
There is already a pr to bump the aws sdk, #275
Is a new release planned or is there something i can help with to prepare a new release?
Updating your example Docker Compose file
link in docs/setup-networking.md
from examples/docker-compose.yml
to ../examples/docker-compose.yml
should fix it.
I didn't want to make PR for something so trivial -- best to leave that for actual code related stuff. Anyway this tool is pretty cool and helping me as I learn ECS, so thank you!
When vending credentials using the AWS_CREDENTIALS_RELATIVE_URI=role/<role_name>
environment variable, it is only possible to assume a role in the same account as the credentials in ~/.aws/credentials
.
In line with a recommended account structure (Identity Account Structure) it is often the case that credentials on a developer machine are only for an identity account, but give permissions to assume roles in other accounts. This workflow does not work with this container, as the container can only assume roles in said identity account.
It would be good if this container could accept an optional environment variable for the account number in which the target role exists, and maybe even a corresponding role name that can be used for role discovery (via the iam.GetRole()
SDK call)
I'm trying to run ecs-local-container-endpoints using the recommended approach ( Bridge Network )
I've set up everything as the guide suggest but the ecs-local-container do not work and prints this message evey time I send a request to http://169.254.170.2/creds:
Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, sm-sdx.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: no route to host
Although if I set it up using the ip tables approach eveything works fine.
I'm running on Ubuntu 20.4 with Docker version 20.10.9 and Docker compose version 1.29.2
So different arch can pull the image of their own arch with pointing to "latest". https://hub.docker.com/layers/amazon/amazon-ecs-local-container-endpoints/latest/images/sha256-2f1be750a040f71540a1bca3c9fe3295030a022595a2fc8d3966623bd3f75bd2?context=explore
This does not support "aws configure sso". If I copy the credentials into the ~/.aws/credentials file it works. Can support for sso be added
My goal is to simulate in a local docker-compose
(or docker compose
) cluster the same credential acquisition that occurs when running my containers in Fargate (as deployed with the compose-cli).
I am able to start the amazon-ecs-local-container-endpoints
in my cluster using docker-compose
and I am following the example provided. However, because I want to simulate the Fargate experience, I use the /creds/{role name}
model. My application container therefore has the following env vars set:
AWS_REGION: "us-east-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/creds/pipeline-PlineTaskRole-XC885YZXQBDN"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"
In the amazon-ecs-local-container-endpoints
container, essentially copied the example:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
- /var/run:/var/run
- $HOME/.aws/:/home/.aws/
environment:
HOME: "/home"
AWS_PROFILE: "default"
networks:
credentials_network:
ipv4_address: "169.254.170.2"
When I start the cluster and attempt to retrieve credentials using DefaultCredentialsProvider
from the AWS SDK for Java (version 2.15.57) I get an exception:
Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): The requested metadata is not found at http://169.254.170.2/creds/pipeline-PlineTaskRole-XC885YZXQBDN, InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]
The last entry in that list is telling: it appears the SDK is making the right call, but the endpoints container is not answering.
Addendum: On the assumptions that my default
profile was somehow deficient, I attempted to add the (inline) trust policy per the documentation:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::264945611335:user/Administrator"
},
"Action": "sts:AssumeRole"
}
]
}
However, the AWS IAM console consistently fails to accept the policy and returns a vague This policy contains the following error: Has prohibited field Principal For more information about the IAM policy grammar, see AWS IAM Policies
In docker hub, the tag latest
etc is duplicated as latest-arm64
with each having one arch. This causes docker to select the incorrect build and run it by emulation on Docker for Mac on ARM. The latest
tag should provide both architectures.
I am running this trying to use sms, but I get the following when I run docker-compose up
File "/usr/local/lib/python3.7/site-packages/boto3/session.py", line 263, in client
app_1 | aws_session_token=aws_session_token, config=config)
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 826, in create_client
app_1 | credentials = self.get_credentials()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/session.py", line 425, in get_credentials
app_1 | 'credential_provider').load_credentials()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1688, in load_credentials
app_1 | creds = provider.load()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1551, in load
app_1 | return self._retrieve_or_fail()
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 1566, in _retrieve_or_fail
app_1 | expiry_time=_parse_if_needed(creds['expiry_time']),
app_1 | File "/usr/local/lib/python3.7/site-packages/botocore/credentials.py", line 137, in _parse_if_needed
app_1 | return parse(value)
app_1 | File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 1358, in parse
app_1 | return DEFAULTPARSER.parse(timestr, **kwargs)
app_1 | File "/usr/local/lib/python3.7/site-packages/dateutil/parser/_parser.py", line 652, in parse
app_1 | raise ValueError("String does not contain a date:", timestr)
app_1 | ValueError: ('String does not contain a date:', '')
app_1 | unable to load app 0 (mountpoint='') (callable not found or import error)
Hi there, can someone please help me understand what I'm doing wrong.
I'm following the docs to make local env setup working.
Here is my override file
version: "2"
networks:
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints
volumes:
- /var/run:/var/run
- "~/.aws:/home/.aws:ro"
# env_file:
# - ~/.clokta/fusion.env
environment:
HOME:
AWS_PROFILE: "fusion"
networks:
credentials_network:
ipv4_address: "169.254.170.2"
proxy:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
AWS_DEFAULT_REGION: "us-east-1"
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI: "/role/blah-role"
ECS_CONTAINER_METADATA_URI: "http://169.254.170.2/v3"
I've logged in through SSO and have STS issued credentials.
With this set up I'm making the following request:
GET 169.254.170.2/role/blah-role HTTP/1.1
Host:169.254.170.2
Connection: close
As a result, I'm getting the following response:
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 25 Feb 2020 19:06:08 GMT
Content-Length: 435
Connection: close
Internal Server Error: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, fusion.
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused
ecs local container logs has this line:
ecs-local-endpoints_1 | time="2020-02-25T19:06:07Z" level=error msg="HTTP 500 - NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, fusion.\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials/: dial tcp 169.254.169.254:80: connect: connection refused"
However, if I uncomment
# env_file:
# - ~/.clokta/fusion.env
and rerun docker-compose, everything seems fine.
~/.clokta/fusion.env contains AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Those values match the [fusion] section of the ~/.aws/credentials which I'm mapping to the container, except aws_access_key_id key and friends are in the lower case.
Is there something I'm missing in the setup?
I am unable to successfully get task metadata from the v3 metadata uri due to a failure to connect to the docker.sock file.
In order to avoid any issues stemming from my application itself, I opened a terminal in the ecs-local-endpoints
container, and tested some curl commands. The output matches what I see in the logs when I call from my application.
sh-4.2# curl http://169.254.170.2/v3
Internal Server Error: Failed to list running containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
sh-4.2# curl http://169.254.170.2/v3/task
Internal Server Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
The creds endpoint works just fine:
sh-4.2# curl http://169.254.170.2/creds
{"AccessKeyId":"XXXXXXXX","Expiration":"2022-12-06T16:58:13Z","RoleArn":"","SecretAccessKey":"XXXXXX","Token":"XXXXX"}
The docker.sock file is properly mapped into the container. And just in case the symlink was an issue, I also tried just mapping the direct path, but it doesn't seem to make a difference.
sh-4.2# ls -l /var/run/docker.sock
lrwxr-xr-x 1 root root 43 Nov 28 20:23 /var/run/docker.sock -> /Users/my.username/.docker/run/docker.sock
## and with direct path mapped ##
sh-4.2# ls -l /var/run/docker.sock
srwxr-xr-x 1 root root 0 Dec 6 16:08 docker.sock
Here is my docker-compose overrides file:
version: "2.4"
networks:
credentials_network:
driver: bridge
ipam:
config:
- subnet: "169.254.170.0/24"
gateway: 169.254.170.1
services:
ecs-local-endpoints:
image: amazon/amazon-ecs-local-container-endpoints:latest-arm64
volumes:
- /var/run:/var/run
- $HOME/.aws/:/home/.aws/
environment:
HOME: "/home"
AWS_PROFILE: "default"
networks:
credentials_network:
ipv4_address: "169.254.170.2"
app:
depends_on:
- ecs-local-endpoints
networks:
credentials_network:
ipv4_address: "169.254.170.3"
environment:
- AWS_DEFAULT_REGION=us-east-1
- AWS_REGION=us-east-1
- AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/creds
- ECS_CONTAINER_METADATA_URI_V4=http://169.254.170.2/v3
- ECS_CONTAINER_METADATA_URI=http://169.254.170.2
I am using the latest version of Docker Desktop -- 4.15.0 (93002) on an M1 Macbook running MacOS 13.0
Is there something obviously wrong here, or is there a known issue either with running on M1 Macs or newer versions of docker?
Hi,
I am working with compose based on scenario 1 (https://aws.amazon.com/blogs/compute/a-guide-to-locally-testing-containers-with-amazon-ecs-local-endpoints-and-docker-compose/), everything work well till I configured my environment to MFA authentication.
Before MFA i just used the profile name and mounting the .aws directory, I change the docker-compose.override to use the current session, the problem is that I get (from Boto) ResourceNotFoundException when I am trying to work with Dynamo DB. any idea what am I missing?
in order to use the authentication session I did the following changes (see env vars below):
services:
# This container vends credentials to your containers
ecs-local-endpoints:
# The Amazon ECS Local Container Endpoints Docker Image
image: amazon/amazon-ecs-local-container-endpoints
volumes:
# Mount /var/run so we can access docker.sock and talk to Docker
- /var/run:/var/run
# Mount the shared configuration directory, used by the AWS CLI and AWS SDKs
# On Windows, this directory can be found at "%UserProfile%\.aws"
- $HOME/.aws/:/home/.aws/
environment:
# define the home folder; credentials will be read from $HOME/.aws
HOME: "/home"
# You can change which AWS CLI Profile is used
AWS_PROFILE: "DevAdmin"
AWS_ACCESS_KEY_ID: "${AWS_ACCESS_KEY_ID}" ---- **Get the KEY_ID from the local machine after MFA authentication**
AWS_SECRET_ACCESS_KEY: "${AWS_SECRET_ACCESS_KEY}" ---- **Get the ACCESS_KEY from the local machine after MFA** authentication
AWS_SESSION_TOKEN: "${AWS_SESSION_TOKEN}" ---- **Get the SESSION_TOKEN from the local machine after MFA** authentication
AWS_REGION: "eu-west-1"
10X in advance.
Nir
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.