nzoschke/gofaas

A boilerplate Go and AWS Lambda app. Demonstrates an expert configuration of 10+ AWS services to support running Go functions-as-a-service (FaaS).

gofaas's Issues

Wishlist

What are the things that still feel painful to you and that you hope AWS fixes in the next 6-12 months?

API Caching

Consider exploring the caching feature of API gateway:

https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-caching.html

Change Sets

It might be nice to display the CloudFormation deploy change set:

Something like this but it needs to wait for the change set execute like cloudformation deploy does.

deploy: BUCKET = pkgs-$(shell aws sts get-caller-identity --output text --query 'Account')-$(AWS_REGION)
deploy:
	@aws s3api head-bucket --bucket $(BUCKET) || aws s3 mb s3://$(BUCKET) --region $(AWS_REGION)
	@aws cloudformation package --output-template-file out.yml --s3-bucket $(BUCKET) --template-file template.yml
	@aws cloudformation deploy --capabilities CAPABILITY_NAMED_IAM --no-execute-changeset --template-file out.yml --stack-name $(APP)
	@aws cloudformation describe-change-set \
		--change-set-name $$(aws cloudformation list-change-sets --stack-name gofaas --output text --query 'Summaries[0].ChangeSetName') \
		--output table \
		--query 'Changes[*].{Action:ResourceChange.Action,Resource:ResourceChange.LogicalResourceId,Type:ResourceChange.ResourceType,Replacement:ResourceChange.Replacement,Target:ResourceChange.Details[0].Target.Name}' \
		--stack-name $(APP)
	@aws cloudformation execute-change-set \
		--change-set-name $$(aws cloudformation list-change-sets --stack-name gofaas --output text --query 'Summaries[0].ChangeSetName') \
		--stack-name $(APP)
	@aws cloudformation describe-stacks --output table --query 'Stacks[*].Outputs' --stack-name $(APP)

Project organization

After adding the auth javascript handler things are starting to feel a bit disorganized. Some ideas:

• Kill the checked in Go vendor directory
  • Move yamllint up into bios
• Introduce handlers/go and handlers/js hierarchy
• Move .go files into hierarchy of smaller packages, e.g. pkg/user/user.go, pkg/work/work.go

Why SAM?

Maybe interesting to compare and contrast SAM, Serverless, Apex, ... and the benefits you get from building within the native AWS ecosystem.

Databases: Note about Serverless Aurora?

I believe all of these reasons are why they're building Amazon Aurora Serverless. It's in preview right now, but it might be worth a callout as an option in the future?

PostgreSQL, one of the goto databases for web apps, may not handle 100 simultaneous connections without adding connection pooling, or may require migrating data to a higher capacity server. Both are heavy operational tasks.

DynamoDB is better suited to this challenge.

DynamoDB not as easy to use as a developer. It lacks transactions so if we need to update multiple records atomically, our code has to handle locking, updating, then unlocking. It has a simplistic indexing model so we have to design our table keys and limited indexes carefully to avoid scanning the entire table. It's scaling model isn't perfect, so there are scenarios where DynamoDB will be inefficient and expensive at medium to large scale.

DynamoDB autoscaling

It'd be interesting to demonstrate how to use DDB autoscaling to provision a 1,1 table but that could burst to 10x throughput without any management from the ops side of the table.

Go Debugger Docs

Step Functions

Demonstrate and document step functions to automate things

Lambda@Edge and OAuth doc tweak?

The docs about private static websites show a bucket configuration that implies usage of the website endpoint (WebsiteConfiguration):

Resources:
  WebBucket:
    Properties:
      BucketName: !Ref WebDomainName
      WebsiteConfiguration:
        ErrorDocument: 404.html
        IndexDocument: index.html
    Type: AWS::S3::Bucket

But the CloudFront distribution created there is not pointing at the website endpoint:

Origins:
          - DomainName: !Sub ${WebBucket}.s3.amazonaws.com
            Id: !Ref WebBucket
            S3OriginConfig:
              OriginAccessIdentity: !Sub origin-access-identity/cloudfront/${WebOriginAccessIdentity}

Which, if I understand it correctly, is because accessing S3 objects through the website endpoint is incompatible with the origin access identity feature that is used here.

First, this is an awesome project, thank you! I found it to be very helpful.

Second, a question: is my reading accurate that WebsiteConfiguration is not really doing anything here? If so I think it would be helpful to remove it from the template to clarify that the WebsiteConfiguration and S3 website endpoint are incompatible with locking down access to the bucket in this way.

I'm happy to send a PR if it makes sense.

Docs license

Explicitly license the docs with something like:

This work is copyright Noah Zoschke and licensed under a Creative Commons Attribution 4.0 Unported License.

More App Functionality

There's still more functionality to port over:

• HTTP Auth and Tokens
• KMS env encryption
• Worker API
• Users API + Table
• KMS data encryption

I'm also considering including:

• S3 / CloudFront static site

This could be a nice demo of multiple stacks and export, where the S3 bucket is an export the app can import.

Security docs

#45 was merged without docs:

• Docs - Static web security with CloudFront, Lambda@Edge and Google OAuth
• Docs - Function security with JWT
• Docs - Parameter store
• Docs - Update static sites with improved S3 origin strategy

Debugging

It would be nice to support and document debugging Go apps. However it looks like it might not work upstream yet:

aws/aws-sam-cli#281

admin and one-off tasks

Lambda poses a challenge running interactive sessions for things like database migrations, reporting, etc. It could be useful to at least document some techniques if not build a tool.

CloudTrail

It is always nice to be able to audit KMS access. Should this stack create a CloudTrail?

CloudFormation CustomResources Docs

A custom resource appeared in #67 to toggle API Gateway tracing. It would be good to document how this works with Go and Lambda.

Encryption at rest confusion

Encrypting data before saving it to the database is a security best practice called "encryption at rest".

Generally encryption at rest would be more along the lines of what AWS can do for you: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/EncryptionAtRest.html

It has limited security value (basically guarding against someone walking out of the data center with a hard drive). However it is extremely simple to use when AWS supports it and great for checking off compliance boxes ✅.

Your example would be closer to column-level encryption or something like that.

SQS

SQS is an event source worth demoing and documenting

More docs

• Idiomatic AWS
• Admin and one-off tasks

API endpoint to invoke worker

API Gateway Authorizer

We can lift the JWTClaims helper out of our handler functions and into an API Gateway Authorizer, further reducing application code and increasing security.

Support for this in SAM is questionable. It doesn't look like a Serverless::Function supports adding an authorizer yet, but maybe you can assign then via Swagger.

• aws/serverless-application-model#248
• aws/serverless-application-model#25
• aws/serverless-application-model#49

Auth

I plan to port an OAuth / JWT example over.

One question is how to get the OAuth redirect url. Perhaps you can generate it from the Request:

2018/03/04 18:26:03 EVENT: {Resource:/auth Path:/auth HTTPMethod:GET Headers:map[Accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 Accept-Language:en-US,en;q=0.9 Host:jkpyz7xxxx.execute-api.us-west-2.amazonaws.com X-Amzn-Trace-Id:Root=1-5a9c3a3b-90b30f21a64603017bc3891e X-Forwarded-Proto:https X-Forwarded-For:73.92.1.8, 205.251.214.101 Accept-Encoding:gzip, deflate, br CloudFront-Is-Desktop-Viewer:true CloudFront-Viewer-Country:US Via:2.0 3cc911e7eb2df956e3f7c8f27c19xxxx.cloudfront.net (CloudFront) X-Amz-Cf-Id:c-rDYArkjqKDPoUwiTCMgHEj29egDuMOiVLX-v-vLjCm_i51DYoASQ== CloudFront-Forwarded-Proto:https CloudFront-Is-SmartTV-Viewer:false CloudFront-Is-Tablet-Viewer:false User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36 CloudFront-Is-Mobile-Viewer:false upgrade-insecure-requests:1 X-Forwarded-Port:443] QueryStringParameters:map[] PathParameters:map[] StageVariables:map[] RequestContext:{AccountID:XXXXXXXXXXXX ResourceID:mh6gbm Stage:Prod RequestID:7ee1c72a-1fd9-11e8-8695-f9317416d457 Identity:{CognitoIdentityPoolID: AccountID: CognitoIdentityID: Caller: APIKey: SourceIP:73.92.1.8 CognitoAuthenticationType: CognitoAuthenticationProvider: UserArn: UserAgent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36 User:} ResourcePath:/auth Authorizer:map[] HTTPMethod:GET APIID:jkpyz7xxxx} Body: IsBase64Encoded:false}

Lambda challenges / advanced topics

It might be interesting to dive into some more advanced topics and challenges with building serverless apps:

• Cold starts
  • How they're impacted by VPC vs. public lambdas
  • Warming strategies (and whether they're worthwhile)
  • Strategies to minimize
• Shared code across functions
• How to manage dev / stage / prod types of multiple environments
• Lambda concurrency configuration
• Monitoring costs
• Maybe worth linking to the Well Architected Serverless Lens

Google OAuth 2.0 "state"

A reviewer reminded me of The Most Common OAuth2 Vulnerability

How to detect, is certain OAuth implementation vulnerable?

If site doesn't send 'state' param and redirect_uri param is static and doesn't contain any random hashes - it's vulnerable.

Reviewing the implementation, there is no "state" parameter on the OAuth redirect or callback.

Digging into the Passport code, it looks like state isn’t enforced. It’s ok if it isn’t provided and also ignored if there isn’t a session store configured. I don’t have a session store configured, since that's more of an Express.js thing.

https://github.com/jaredhanson/passport-oauth2/blob/master/lib/strategy.js#L204

Here’s the some Google guides that talks about state.

https://developers.google.com/identity/protocols/OAuth2WebServer
https://developers.google.com/identity/protocols/OpenIDConnect#state-param
https://developers.google.com/identity/protocols/OpenIDConnect#createxsrftoken

So I think we need to come up with a Lambda@Edge friendly way to set “state” and verify it….

Periodic Event Rule

I don't think the period worker is wired up quite right. It seems to get triggered by all cloudwatch events, not just the related rule in the template

Faster development server

The current make dev is passable for simple apps, but for more complex apps I the Go http server that mounts the handlers with a HTTP handler middleware of sorts.

It would be nice to add cmd/server/main.go to this project to demonstrate that pattern.

Use x-ray in Periodic Worker

There's a bug in the aws sdk where the s3manager isn't using context properly.

I reported it upstream so hopefully it gets addressed there.

If not, we could use S3() more directly to clean the bucket, or live without tracing.

watchexec + make + start-api

It would be nice if aws-sam-local start-api would pick up the latest code changes. We should be able to do this with a tool like watchexec calling make handlers on changes. Go 1.10 build caching might make this a fast enough experience.

New sam CLI install method

See https://github.com/awslabs/aws-sam-cli/releases/tag/v0.6.2

$ brew tap aws/tap
$ brew install aws-sam-cli

$ pip uninstall --user aws-sam-cli
$ pip3 uninstall --user aws-sam-cli

CodeBuild & CodePipeline

Might be nice to use CodeBuild & CodePipeline for the deployment process and have it be setup as part of the CF template.

handler -> main

It might be a bit more idiomatic to compile the binaries to main and make main.zip. Shorter too.

seeding DynamoDB with data

What's the best way to seed DB with data from a json file?

• Another lambda/worker function?
• How to best make sure the function is run one-off & only if DB is not already seeded?

KMS Environment Encryption

Seems like we can encrypt environment variables effectively for free by adding a KmsKeyArn property to functions. See: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md

This would be an easy contribution for a first timer...

for custom domains for regions other than us-east-1

#12 introduced a side effect where the cert has to be in us-east-1.

If the stack was created in us-west-2, etc. you get this error: Invalid certificate ARN: arn:aws:acm:us-west-2:572007530218:certificate/26e813b5-3a21-43e0-bbfc-ad02afa17cb0. Certificate must be in 'us-east-1'.

It looks like you can manually create the ACM cert in us-east-1 and assign it to an API Gateway in another region, but that defeats the automation.

We could still support us-west-2, etc, by adding a cert ARN parameter and the various conditional bits

Update logs doc to use SAM CLI

The SAM CLI makes it easy to get all the logs for an app now. Should update docs to reflect