Method to read/write encrypted config values

[tmaiaroto]

Method to read/write encrypted config values

[tmaiaroto]

Two options here to consider.

1. reading from S3 (on each cold invocation) would allow for "live" config updates.
2. the deploy command could fetch the encrypted config file from S3 and bundle it with the binary in the zip file sent to Lambda. this would avoid an extra get operation from s3. which would speed up cold Lambda invocations. though it would also "freeze" configuration. the deploy command would update the config. which is also kinda nice if you want to be in control of "when" things update. given unpredictable container re-use, etc. it can also be per "stage/environment."

[tmaiaroto]

3. Another option could be to manage secure config settings from the CLI (store where ever, S3, etc.) and then a new push config command of the sorts could merely update existing Lambdas or API Gateways with new environment variables/stage variables. That way the concerns about requests to retrieve a file full of encrypted credentials and then another request to KMS to decrypt go away. There's no up front cost in cold starts. Config changes are deterministic. They only occur on deploy or config push commands...Or perhaps only a config push command.

This also has the added benefit of standardizing on os.Getenv() so it works when testing locally (a lot more consideration to running locally is required too). That's if Lambda environment variables are used. If API GW stage variables are used, it's a different call...but there can of course be an interface for getting values which falls back to getting environment variables so it works for the two possible locations of variables.

I think this works a lot better. It's the best of both worlds. It provides a way to manage sensitive credentials. Secure them. Securely share them even, in case you're working with other people, and then easily and deterministically update configurations.

The only thing with stage variables and Lambda env vars to note here is that they are per "stage" in API GW and are constant for Lambdas. So if config per stage is desired, there will be some extra considerations.

[tmaiaroto]

Maybe just integrate with secrets manager. Though I'm really curious about how it would compare with a custom usage of KMS. Sure there's rotation and all sorts of extras on top, but the cost is $0.40/mo per secret. That might kinda add up. It's a lot more than it would cost to store a simple file in S3 that can be decrypted and set on environment or stage variables.

[tmaiaroto]

This will use AWS Secrets manager. There will be some new CLI methods that allow for the management of secrets (just an alternative to using AWS web console).

Then upon deploy if aegis.yaml defines any secrets in Lambda environment variables or API Gateway stage variables, they will be retrieved from secrets manager and added upon deploy. Therefore, config updates will be governed by deploy actions.

There will be helper functions to make it nicer to use/read these values. Looking at os.Getenv() (Lambda environment) and then API GW stage variables.

The helper will also see if the value is base64 encoded and then decode it. This is important: Secret values may need to be encoded as base64 IF they are used on API GW stage variables because they have a limited set of characters. It will be very common for passwords with special characters to have problems with API GW stage variables. A base64 encoded string will work just fine...But that's not so obvious and making someone encode their values to use them there is silly. That's why it's useful to use the CLI to store secrets and then use the helper to retrieve them, because Aegis will handle that automatically.

Still. All of this remains completely optional and no functionality is hidden nor destroyed.

Additionally, there will be a SecretsManager service added so that one can retrieve secrets from their Lambdas if so desired (live updates, etc.).

Last, there will also be a new event handler/router or AWS Secrets Manager since it can invoke Lambdas.

TBD: the convention for using secrets.
Likely something like <secretName.keyName> will denote a secret being used and the deploy command will catch that and set a Lambda env variable or API GW stage variable depending on where that conventional string was placed in aegis.yaml.

Note: It will be up the user to rename variables in aegis.yaml in the event of conflicts. For example, if using two databases there will be two port keys on different secrets. The problem should be quite evident just by looking at the aegis.yaml config that there's a conflict.

[tmaiaroto]

Just want to double check some things. Maybe add some more test cases (always helpful and really behind on it now). The example Cognito app has been updated to use the new secrets manager. Loving the secrets manager CLI.