droplr / aws-env Goto Github PK
View Code? Open in Web Editor NEWSecure way to handle environment variables in Docker and envfile with AWS Parameter Store
License: MIT License
Secure way to handle environment variables in Docker and envfile with AWS Parameter Store
License: MIT License
Parameters with spaces/new lines are not exported correctly.
Detailed discussion is on SO: https://stackoverflow.com/questions/45098315/exporting-environment-variables-from-script-with-whitespaces
Thanks for setting up this package. I've followed the Docker instructions and successfully have the script running on my container startup. My app is receiving the variables, but with a $
appended to the variable. For example, the app is receiving VAR_NAME=$value
instead of VAR_NAME=value
. Let me know if this is an issue or if I'm just using the package incorrectly.
We're running into a bit of a problem with the -format dotenv
flag.
We've noticed that when reading values out of SSM and using -format dotenv
, all of the values are wrapped in double quotes. This ends up breaking most of the other services and tools we use that parse these quotes as literal double quotes.
For instance, if the .env
file was:
FOO="bar"
Then something that read that environment variable would read it as "bar"
instead of the expected bar
.
What's the expect way to deal with the double quotes?
I am getting the following error when my container executes:
2018-12-05 00:26:51panic: MissingRegion: could not find region configuration
2018-12-05 00:26:51goroutine 1 [running]:
2018-12-05 00:26:51log.Panic(0xc420107ee0, 0x1, 0x1)
2018-12-05 00:26:51 /usr/local/Cellar/go/1.10.3/libexec/src/log/log.go:326 +0xc0
2018-12-05 00:26:51main.ExportVariables(0xc42000c0a0, 0xc42001600d, 0x21, 0x0, 0x75f5f8, 0x7, 0x0, 0x0)
2018-12-05 00:26:51 /Users/aorfin/go/aws-env/aws-env.go:62 +0x17d
2018-12-05 00:26:51main.main()
2018-12-05 00:26:51 /Users/aorfin/go/aws-env/aws-env.go:37 +0x193
2018-12-05 00:26:512018/12/05 06:26:51 MissingRegion: could not find region configuration
This seems to be some sort of AWS configuration issue, but my IAM role has permissions to for ssm:Get*
on all resources.
Here is my Dockerfile:
FROM php:7.2.11-fpm
RUN apt-get update
RUN apt-get install wget
# Install opcache
RUN docker-php-ext-install opcache
# Install APCu
RUN pecl install apcu
RUN docker-php-ext-enable apcu
ADD webroot /webroot
ADD etc/zz-docker.conf /usr/local/etc/php-fpm.d/zz-docker.conf
RUN wget https://github.com/Droplr/aws-env/raw/9605a216913c18cf64338509204ca21379832b90/bin/aws-env-linux-amd64 -O /bin/aws-env && \
chmod +x /bin/aws-env
CMD ["/bin/bash", "-c", "eval $(aws-env) && php-fpm"]
It is my understanding, and my debug shows, that the AWS_REGION env variable is set in my codebuild project that builds the image. I did try changing the last line to the following with the same result:
CMD ["/bin/bash", "-c", "eval $(AWS_REGION=$AWS_REGION aws-env) && php-fpm"]
Hi,
What would you think about adding a customizable region that could be passed with a flag instead of using the AWS_REGION
environment variable ?
Here is my use case, I am currently using eu-west-3 (Paris) for most of my services. Unfortunately, SSM Parameter Store is not yet available in this region, thus I am currently putting my SSM Parameter Store data in eu-west-1 region.
But I don't want to put eu-west-1 as AWS_REGION
env variable as it could cause some side effect with other services.
Instead, I would like to be able to define my image entrypoint with something like this :
ENTRYPOINT ["/bin/bash", "-c", "eval $(aws-env --region eu-west-1)"]
What do you think about that ? Does it seem legit ?
I tried to supply AWS_ENV_PATH=/test and it produced no output. Once I realized I need at least two levels: AWS_ENV_PATH=/test/two, it worked great!
Reference: https://github.com/GoogleContainerTools/distroless
Guide me for integrating in distroless docker.
AWS_REGION is widely used across multiple aws tools, yet due to parameter-store nature we need to use different region quite often.
Instead of var cycling or sub-shelling (reducing env+stack traceability) its better to use dedicated variable (AWS_ENV_REGION)
proposal (logical solution):
AWS_ENV_REGION || AWS_REGION || current_region(user-data) || null
panic: AccessDeniedException: The ciphertext refers to a customer master key that does not exist, does not exist in this region, or you are not allowed to access. (Service: AWSKMS; Status Code: 400; Error Code: AccessDeniedException; Request ID: xxx)
Seeing the above message on launch, I've confirmed the region, and AWS_ENV_PATH are both set and consistent with what is in the parameter store.
Any ideas?
hi,
this tool is simple as awesome,
I've tested it against two dotenv libraries for node and for ruby,
but this is the result:
>> ENV['DB_PASSWORD']
=> "$'SecretPassword'"
App is starting...
key loaded: DB_USERNAME $'Username
key loaded: DB_PASSWORD $'SecretPassword
Can you provide two options for
Hi,
We're using aws-env in ECS and therefore get our credentials through an AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
environment variable that points to a relative URI where the credentials can be downloaded from. This doesn't seem to work with the current version of aws-env, as it's using the default config with anonymous credentials. However, if you supply the defaults.Get().Config
(defaults.go) when creating new session the SDK will look at the above env-var among other things.
I have made the below change in a fork and it works well. Though, I wanted to check with you before submitting a PR if there are consequences to this that I'm not aware of.
From
func CreateClient() *ssm.SSM {
sess := session.Must(session.NewSession())
return ssm.New(sess)
}
To
func CreateClient() *ssm.SSM {
cfg := defaults.Get().Config
sess := session.Must(session.NewSession(cfg))
return ssm.New(sess)
}
I switched some services in ECS to use Fargate instead of EC2 and it seems to cause aws-env to blow up.
2018/05/10 03:06:04 NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
panic: NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
goroutine 1 [running]:
panic(0x6adf20, 0xc42000bfc0)
/usr/local/Cellar/go/1.7.4/libexec/src/runtime/panic.go:500 +0x1a1
log.Panic(0xc4200c5eb0, 0x1, 0x1)
/usr/local/Cellar/go/1.7.4/libexec/src/log/log.go:320 +0xc9
main.ExportVariables(0xc42001200d, 0x19, 0x0, 0x0)
/Users/aorfin/go/aws-env/aws-env.go:42 +0x254
main.main()
/Users/aorfin/go/aws-env/aws-env.go:19 +0x113
Binary is being pulled from latest commit.
This looks like it could be a bug in the aws go sdk or something wonky in Fargate; I am pretty green to AWS and have zero go experience so I wasn't sure where to start on debugging this.
Any suggestions?
Can we set up a proper release of the binaries with hash keys?
"In [ecs-agent] v1.17.0, a rate a limiter was added to the credentials endpoint, which allows clients to invoke the endpoint at a steady state rate of 10 per second and a burst of 15 per second" (aws/amazon-ecs-agent#1231)
aws-env authenticates every time the ssm api call executes, which hits the rate limit when a lot of variables are declared in the parameter store.
You can see the credential requests on ECS nodes using:
docker logs --tail 50 -f $(docker ps |grep ecs-agent |cut -d ' ' -f1 ) |grep credentials
I'll submit a PR to resolve this
We just ran into an issue where we had some HTML in an environment variable which was quoted, so for example:
aws ssm put-parameter --name /my/path/HTML_BODY --value "'Lorem Ipsum<br><br>'" --type String
Now eval $(aws-env) breaks:
AWS_ENV_PATH=/my/path aws-env-linux-amd64 export HTML_BODY=$''Lorem Ipsum<br><br>''
eval $(AWS_ENV_PATH=/my/path aws-env-linux-amd64)
-bash: syntax error near unexpected token '<'
If aws-env would escape these we would not have a problem:
$ export HTML_BODY=$'\'Lorem Ipsum<br><br>\''
$ echo $HTML_BODY
'Lorem Ipsum<br><br>'
Hi Droplr folks,
Thank you for publishing your project to the internet! I appreciate the functionality you've created here - I'd really like to use it in development of my project at work. We're deploying our system to AWS and the injection of encrypted Parameters as env vars would be awesome for us. One tiny bump is holding us back at the moment.
Would you be ok adding a license file to your repository? GitHub has a super easy set of license templates to use.
I'd also kindly suggest/hope for Apache 2.0, MIT, or BSD. Again, many thanks for the OSS contribution!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.