rails-lambda/lamby

🐑🛤 Simple Rails & AWS Lambda Integration

lamby's Issues

Synchronous Migrations possible?

Looking to discuss this topic as I haven't found a solution that I'm 100% sold on.

Migrations are discussed in the starting docs and here as directly callable.

For me, ideally I'd like to keep the migrations as part of the deployment process, synchronous specifically. Async migrations require the code to be pretty robust with regards to referencing new columns and multiple deployments to get to the finalized state.

Fargate is a path I currently use for another production product:

• Allows migration to run longer than 15 minutes (crazy but I've done them before).
• aws ecs run-task allows single execution
• Can be synchronous via polling through aws ecs describe-tasks

Lambda could do similar if migrations are limited to 15 minutes.

• Deploy code to a special migration Lambda function
• curl to a special/locked-down endpoint/url and wait for response to return.

To make it synchronous, the a separate template/stack would have to hold the migration lambda function.

Thoughts?

ImageMagick & Lambda Layers

Find something user contributed already to help ease this and test in SAM and that our native gem extension build strategy works well for this use case.

• https://github.com/mthenw/awesome-layers
• https://twitter.com/clare_liguori/status/1087861037712400385

Async Lambda Extension (ActiveJob, Suckerpunch, Async NewRelic, Etc)

If you use :async it will not work on Lambda. Maybe we can use Lambda Extensions to buffer and run things?

https://twitter.com/julian_wood/status/1396882542737903624

How Does Lamby Work?

https://lamby.custominktech.com/docs/how_does_lamby_work

[SAM] Docker Perf on Mac & Root Proxy

Unfortunately, the SAM CLI project has a few needed enhancements and if you decide to do any local development via the sam local start-api command, it will need to be patched on your local machine. I used the commands below to find the two files needing patching.

$ which sam # Use directory info below in find command.
$ find /usr/local -name container.py | grep samcli
$ find /usr/local -name local_apigw_service.py | grep samcli

Here are the issues we have created on the SAM CLI project. Details on the patches below.

• Add Docker Delegated Consistency to Volume FIXED ✅
• Missing Authentication Token for root path '/' ** FIXED ✅** (aws/aws-sam-cli#1046)

Docker Volume Mount Performance on Mac

If you are on a Mac, Docker has a known performance issue when sharing volumes due to the overhead of keeping files in sync. This performance issue means your Rails application could take ~60 seconds to load in development. Thankfully, Docker has tooling in place to help.

Once you found the correct container.py file, make this change below to tell the Docker SDK to mount the volume with both the ro (read-only) option and delegated consistency mode.

@@ -95,7 +95,7 @@
                     # https://docs.docker.com/storage/bind-mounts
                     # Mount the host directory as "read only" inside container
                     "bind": self._working_dir,
-                    "mode": "ro"
+                    "mode": "ro,delegated"
                 }
             },
             # We are not running an interactive shell here.

Fixing Root SAM Local Proxy Paths

We need our API Gateway to use both a root path and a greedy proxy path to forward to Rails in development. SAM has a feature compatibility bug where it forces static file hosting on the root path. Once you found the correct local_apigw_service.py file, make the change below to disable static public directory assets on the root path. Don't worry, Rails & Rack will serve your static files automatically instead.

@@ -71,7 +71,7 @@
         """

         self._app = Flask(__name__,
-                          static_url_path="",  # Mount static files at root '/'
+                          static_url_path=None,  # Mount static files at root '/'
                           static_folder=self.static_dir  # Serve static files from this directory
                           )

Deploy Script not Working in Linux Environment

I've been having success deploying just fine with the ./bin/deploy script in my Mac with docker and SAM Cli installed. However, I wanted to create a development environment in AWS Cloud 9 and all the scripts run fine for the exception of the deploy script. I started with the ./bin/bootstrap, then I did ./bin/setup and finally ./bin/deploy. The error that shows up is asking if docker is running when the step of running SAM build. I first try running this in an Ubuntu instance, but after a few failed tries I tried using Amazon Linux 2 instance on my cloud 9. Both OS failed the same way with the same error. I think the issue is that the deploy script runs SAM build that creates a docker image, but cannot find docker. I do not understand why is it working fine in the MacBook and I also had a coworker test these applications deploy in his Macbook and it works also (both computers are running different version of MacOS), but it only fails on a Linux instance (I did not try on a Windows).

I tried deleting the container by running docker system prune -a and running the ./bin/bootstrap script again and the same errors show.

Have you guys tested the Lamby cookie cutter applications in a Linux environment? Any ideas on what might be happening?

Content Discussion: Moving off sam deploy

Again, thanks for all your continued work on this gem + the support elements.

Precursor - I think this should be pure content as opposed to part of the boilerplate. It's an unnecessary complexity and more sugar on top.

One great piece that helps towards that end is using AutoPublishAlias: live for blue/green deployments.

Currently, running bin/_deploy uses sam deploy which waits for the auto-built CodeDeploy to finish its job. In a Canary or a Linear traffic swap situation, the shortest increment of 1 minute but really should be shifting traffic in 3 or 5 minute increments resulting in a 6+ minute deploy cycle.

This means we're paying for Github Actions or CodeBuild to wait while CodeDeploy itself is free. Trivial for small applications, but adds up in business environments.

Personally I'm not there yet. I don't fully understand CodeDeploy, AutoPublishAlias is magical, so happy to learn/discuss together if this makes sense.

Read only file system error

I followed the quick guide to deploy a sample rails api app to AWS Lambda. When I hit the API, I am getting following error:

Anything I need to set to customize the tmp path? or am I missing anything?

Rails version: 6.0.3
Ruby Version: 2.7.1

 "errorMessage": "Read-only file system @ dir_s_mkdir - /var/task/tmp",
    "errorType": "Init<Errno::EROFS>",
    "stackTrace": [
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:250:in `mkdir'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:250:in `fu_mkdir'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:228:in `block (2 levels) in mkdir_p'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:226:in `reverse_each'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:226:in `block in mkdir_p'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:211:in `each'",
        "/var/lang/lib/ruby/2.7.0/fileutils.rb:211:in `mkdir_p'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/application.rb:599:in `generate_development_secret'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/application.rb:421:in `secret_key_base'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/application.rb:177:in `key_generator'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/globalid-0.4.2/lib/global_id/railtie.rb:28:in `block (2 levels) in <class:Railtie>'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:68:in `block in execute_hook'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:51:in `each'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.0.3/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/application/finisher.rb:129:in `block in <module:Finisher>'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/initializable.rb:32:in `instance_exec'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/initializable.rb:32:in `run'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/railties-6.0.3/lib/rails/initializable.rb:61:in `block in run_initializers'",
        "/var/lang/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'",

GitLab CI/CD

GitHub actions are great, but support in the docs/demo for Gitlab CI/CD would be much welcome as an alternative/free/self hosted solution.

https://docs.gitlab.com/ee/user/project/clusters/serverless/aws.html#aws-serverless-application-model

Better Gemfile Usage for Dev/Test Groups

Basically the Gemfile is flat now and doing a build #17 will install gems not needed. One thing I'd like to think about during this work is which AWS SDK gems are also installed for us too on the Lambda images. If needed, this knowledge could even be applied to the space savings part of the build script where we remove wasted space.

Cold Start Guides w/Bootsnap

I've done several weeks of work to both measure and update/adopt Bootsnap (http://github.com/Shopify/bootsnap) patterns with Lambda. I'd like to add a Cold Start section that speaks to this and other data points for optimizing Rails performance on Lambda.

Encrypted Session Secret via AWS System Manager Parameter Store.

What Is AWS Systems Manager?

Parameter Store provides secure, hierarchical storage for configuration data and secrets management. You can store data such as passwords, database strings, and license codes as parameter values. You can store values as plain text or encrypted data. You can then reference values by using the unique name you specified when you created the parameter.

Open Source Research

• https://github.com/knowledge-ruby/knowledge
• https://github.com/phstc/param_store
• https://github.com/randywallace/configruous
• https://github.com/dkolb/ssm_utils

AWS Resources

• Aws::SSM::Client
• The Right Way to Store Secrets using Parameter Store
• Using AWS Systems Manager Parameter Store Secure String parameters in AWS CloudFormation templates

Basic Ruby with Lambda & AWS SAM?

Curious about using AWS SAM with Ruby without Rails? Please see this 757rb (Norfolk Ruby User's Group) repository for an overview on how to kick-start your next Ruby project.

• Using Ruby with AWS Lambda & SAM

Support for AWS CodePipeline

How would one go about getting Lamby working with AWS Code Pipeline, more specifically CodeBuild. I'm encountering the following error:

Creating network "src_default" with the default driver
--
75 | Building socket
76 | Step 1/8 : FROM amazon/aws-sam-cli-build-image-ruby2.7
77 | latest: Pulling from amazon/aws-sam-cli-build-image-ruby2.7
78 | Digest: sha256:ad2f34851f004b92448cf4334fe2d312960f52e17cbc32231292b0e915ae979f
79 | Status: Downloaded newer image for amazon/aws-sam-cli-build-image-ruby2.7:latest
80 | ---> eb23e25775d9
81 | Step 2/8 : ARG HOST_UID
82 | ---> Running in 15ff0f44dc96
83 | Removing intermediate container 15ff0f44dc96
84 | ---> c98987b652a5
85 | Step 3/8 : ARG HOST_GID
86 | ---> Running in a3cf2b67181e
87 | Removing intermediate container a3cf2b67181e
88 | ---> 07a25f6f9cfd
89 | Step 4/8 : RUN mkdir /lamby     && /usr/sbin/groupadd --gid $HOST_GID --system --force lamby     && /usr/sbin/useradd --uid $HOST_UID --gid $HOST_GID --non-unique --home-dir /lamby --shell /bin/bash --system lamby     && chown $HOST_UID:$HOST_GID /lamby
90 | ---> Running in d9e9fe139f10
91 | groupadd: invalid group ID '--system'
92 | Service 'socket' failed to build: The command '/bin/sh -c mkdir /lamby     && /usr/sbin/groupadd --gid $HOST_GID --system --force lamby     && /usr/sbin/useradd --uid $HOST_UID --gid $HOST_GID --non-unique --home-dir /lamby --shell /bin/bash --system lamby     && chown $HOST_UID:$HOST_GID /lamby' returned a non-zero code: 3
93 |  
94 | [Container] 2021/05/26 22:52:14 Command did not exit successfully ./bin/setup exit status 1
95 | [Container] 2021/05/26 22:52:14 Phase complete: BUILD State: FAILED

Explore New Dev/Build Container Story / Codespaces

Now that it seems Codespaces (https://github.com/features/codespaces) is getting close and there seems to be so much documentation (https://github.blog/changelog/2021-05-25-tutorials-guides-and-documentation-updates-for-codespaces/) around dev containers for VS Code.... maybe we should re-think how we use Docker here.

Dynamic Handlers

Now that we have EventBridge Events with #82 and we can even add a dynamic event checker for Lambdakiq. But why stop there. Technically we can easily detect the event payload and use whatever Rack adapter is needed. Meaning we can drop/ignore the options hash and let the handler do all the work.

def handler(event:, context:)
  Lamby.handler $app, event, context
end

Railtie Hooks & Templates

Will we need a Railtie hooks so we can initialize better? Maybe templates for SAM YAML or bin scripts?

File Uploads Not Working

Found out that file uploads are not working. Digging into Rack now to find out what I need to pass to ensure this works well. This may be related to #25 too at least for the response. But for the request we need to get the right multi-part headers down.

Hot Spots

• If tempfiles are used make sure they are deleted after each request!

Key Rotation Transform

Looking into using (or documenting) AWS::SecretsManager-2020-07-23 secret rotation.

Resources

AWS::SecretsManager::RotationSchedule HostedRotationLambda
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-secretsmanager-rotationschedule-hostedrotationlambda.html

For the @AWSCloudFormer nerds out there, there's a new hosted template transform that snuck in last week. It generates a Lambda function to rotate secrets for RDS / Redshift / DocDB. Protip: You can use multiple template transforms as a list 💡
https://twitter.com/iann0036/status/1287596970916233217

More than just HTTP API

The starter project assumes a public facing application using HTTP API. But Lamby works fine for REST API & ALBs. Outline why these are important and demo code changes to use. Propose the following guides additions and updates.

• New "Using REST API" section. Warn that this is not an ideal simple public proxy since HTTP API is easier and cheaper. Showcase how to change starter template code to use. Include guides on this page for Custom Domain Names.
• New "Private VPC Applications" section. Calls out two methods. First is the easier ALB Only method with 1MB response limit callout. Note here about how rack-deflater can help. Second method is the fun REST API (link above) with a crazy VPC Endpoint and ALB proxy. AWS just published (finally) guides (https://docs.aws.amazon.com/apigateway/latest/developerguide/private-api-tutorial.html) on this too so we can link to it. Include guides on this page for Custom Domain Names.
• Update "Custom Domain Names" with callouts to each new section above and link to each page. Keep this guide page assuming public HTTP API with callouts.

Bin Scripts - Setup, Build, Server, & Deploy

https://lamby.custominktech.com/docs/bin_scripts_build_and_deploy

Integration with non rails

Can Lamby be integrated with other non rails apps such as Sinatra?

thanks!

Match APIGateway Request IDs to Rails UUID Request IDs. Good CloudWatch Usage.

Basically make these match up.

Custom Domain Name, Edge/Regional, & CloudFront

https://lamby.custominktech.com/docs/custom_domain_name_and_cloudfront

Ruby 2.7: Logger Warnings

Hi There,

Thanks again for making a great library!

I recently wrote the following code in my app running ruby 2.7.1:

require 'logger'
logger = Logger.new(STDOUT, level: Logger::INFO)

I noticed the following warning that was emitted by the interpreter:

/Users/me/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/lamby-3.0.2/lib/lamby/logger.rb:12: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/me/.rbenv/versions/2.7.1/lib/ruby/2.7.0/logger.rb:379: warning: The called method `initialize' is defined here

It looks like Lamby is overwriting the constructor for the Logger class in the standard library - is this necessary?

An easy fix may be to use the double-splat (**) operator in the definition of the override. If you want, I can submit a PR to make this change?

Thanks!

Is `isBase64Encoded` To Response Needed

Right now the Lamby handler returns just the basic hash needed.

    def response
      { statusCode: status,
        headers: headers,
        body: body }
    end

However, there is another and when #23 is done, maybe we need to add some thing around isBase64Encoded boolean and/or automatic translation of the response body too. Will see.

More Runner Documentation

Right now we only mention the runner in the context of DB migrations here (https://lamby.custominktech.com/docs/database_options#rails-database-migrations) but there is a lot more to the story. Share how it can be used with the Rails' runner and maybe showcase a GitHub Action example.

      - name: Inovke
        run: |
          echo $'${{ github.event.inputs.payload }}' > payload.json
          FUNCTION_NAME=$(aws cloudformation describe-stack-resources \
            --stack-name "$STACK_NAME" \
            --query "StackResources[?LogicalResourceId=='RailsLambda'].PhysicalResourceId" \
            --output text)
          aws lambda invoke \
            --function-name "$FUNCTION_NAME" \
            --cli-binary-format raw-in-base64-out \
            --payload file://./payload.json \
            /dev/stdout | jq -r .body

JSON encoding error when serving images in HTTP rack

When trying to serve a non-UTF8 file (e.g. a static image file) I get the following error:

mt$ ruby app.rb 
Traceback (most recent call last):
	9: from app.rb:23:in `<main>'
	8: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/core_ext/object/json.rb:43:in `to_json'
	7: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/json/encoding.rb:22:in `encode'
	6: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/json/encoding.rb:35:in `encode'
	5: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/json/encoding.rb:110:in `stringify'
	4: from /var/lang/lib/ruby/2.7.0/json/common.rb:224:in `generate'
	3: from /var/lang/lib/ruby/2.7.0/json/common.rb:224:in `generate'
	2: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/json/encoding.rb:57:in `to_json'
	1: from /var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/core_ext/object/json.rb:40:in `to_json'
/var/task/vendor/bundle/ruby/2.7.0/gems/activesupport-6.1.1/lib/active_support/core_ext/object/json.rb:40:in `to_json': "\\x89" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

Steps to reproduce:

1. Add a PNG file to app/assets/images/
2. Call your lambda url assets/my.png
3. You should get a response like {"message":"Internal Server Error"}
4. Check your CloudWatch log to find an error message similar to above.

The way I understand Lamby it seems like the app.rb#handler function returns a Hash response ala:
{status: 200, headers: {...}, body: 'String content'}.
Lambda then takes this respons and encode it to JSON (at least it's my assumption that it's Lambda doing this conversion?)

RuntimeError: Container does not exist. Cannot get logs for this container

I get this error when running ./bin/deploy.

I'm using cloud9 with ubuntu 18.04.

The full trace:

Starting Build inside a container
Building resource 'RailsFunction'

Fetching lambci/lambda:build-ruby2.5 Docker container image........................................................................................................................
Mounting /home/ubuntu/environment/readpaths as /tmp/samcli/source:ro,delegated inside runtime container
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/bin/sam", line 11, in <module>
    load_entry_point('aws-sam-cli==0.39.0', 'console_scripts', 'sam')()
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 96, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/lib/telemetry/metrics.py", line 62, in wrapped
    return_value = func(*args, **kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 127, in cli
    mode,
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/commands/build/command.py", line 192, in do_cli
    artifacts = builder.build()
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 104, in build
    lambda_function.runtime)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 195, in _build_function
    runtime)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/lib/build/app_builder.py", line 267, in _build_function_on_container
    container.wait_for_logs(stdout=stdout_stream, stderr=stderr_stream)
  File "/home/linuxbrew/.linuxbrew/Cellar/aws-sam-cli/0.39.0/libexec/lib/python3.7/site-packages/samcli/local/docker/container.py", line 197, in wait_for_logs
    raise RuntimeError("Container does not exist. Cannot get logs for this container")
RuntimeError: Container does not exist. Cannot get logs for this container

quickstart doenst start

Im getting the next log

Successfully built 9ace8a356c94b5b4581b63de7ae42807592f214de87e6f89a281c4479d292e2f
WARNING: The SSH_AUTH_SOCK variable is not set. Defaulting to a blank string.
Creating my_awesome_lambda_my-awesome-lambda_run ... error

ERROR: for my_awesome_lambda_my-awesome-lambda_run Cannot create container for service my-awesome-lambda: invalid volume specification: '.:.:rw': invalid mount config for type "volume": invalid mount path: '.' mount path must be absolute

ERROR: for my-awesome-lambda Cannot create container for service my-awesome-lambda: invalid volume specification: '.:.:rw': invalid mount config for type "volume": invalid mount path: '.' mount path must be absolute
ERROR: Encountered errors while bringing up the project.

I used the quickstart guide, but it didnt event start.-

Environments & Configuration

https://lamby.custominktech.com/docs/environment_and_configuration

undefined method sub for nil:NilClass

Getting the below error for a POST request which generates below event:

## EVENT
{"queryStringParameters"=>{}, "multiValueHeaders"=>{"host"=>["api.***.com"], 
"content-type"=>["application/x-www-form-urlencoded"], 
"x-amzn-trace-id"=>["Root=1-5ebd4017-2bac93a418a717c03d540fd8"], 
"x-forwarded-port"=>["443"], "accept"=>["application/json"], "scope"=>["user"],
 "x-authenticated-userid"=>[" *******"], "x-forwarded-proto"=>["https", " https"], 
"content-length"=>["233"], "x-forwarded-for"=>["3.123.100.237"], "authorization"=>["Bearer *******"], "x-consumer-id"=>["7386304e-f7bd-4815-a086-030cfb0237fd"], "x-consumer-custom-id"=>["authoring_platform"], 
"x-consumer-username"=>["authoring_platform"], "x-authenticated-scope"=>["user"]}, "httpMethod"=>"POST", "pathParameters"=>{}, 
"path"=>"/industries/export_to_google_drive", "multiValueQueryStringParameters"=>{}, 
"headers"=>{"host"=>"api.****.com", "content-type"=>"application/x-www-form-urlencoded", "x-amzn-trace-id"=>"Root=1-5ebd4017-2bac93a418a717c03d540fd8", "x-forwarded-port"=>"443", 
"x-authenticated-scope"=>"user", "x-consumer-username"=>"authoring_platform", 
"x-authenticated-userid"=>"****", "x-forwarded-proto"=>"https", "content-length"=>"233", 
"x-forwarded-for"=>"3.123.100.237", "x-consumer-custom-id"=>"authoring_platform", 
"x-consumer-id"=>"7386304e-f7bd-4815-a086-030cfb0237fd", "authorization"=>"Bearer ****", "scope"=>"user", "accept"=>"application/json"}, "body"=>"X2NzcmY9eGw2cWJpcHUtM1VDanRteVBNb1h5QTg2UUN5alNId0J4V0tVJmluZHVzdHJ5JTVCZW1haWwlNUQ9cHJhc2hhbnQlNDBqb21iYXkuY29tJmluZHVzdHJ5JTVCZm9sZGVyJTVEJTVCdXJsJTVEPWh0dHBzJTNBJTJGJTJGZHJpdmUuZ29vZ2xlLmNvbSUyRmRyaXZlJTJGdSUyRjAlMkZmb2xkZXJzJTJGMEJ6Vjl6TjI1ak5XVlNXbHpkbVkzWjNsTVdFVSZjb250ZW50PWluZHVzdHJpZXMmYXV0aF90b2tlbj0=", 
"isBase64Encoded"=>true}

Error Stacktrace

{
    "errorMessage": "undefined method `sub' for nil:NilClass",
    "errorType": "Function<NoMethodError>",
    "stackTrace": [
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/rack_http.rb:58:in `path_info'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/rack_http.rb:17:in `env_base'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/rack.rb:19:in `env'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/handler.rb:79:in `call_app'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/handler.rb:42:in `call'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby/handler.rb:7:in `call'",
        "/var/task/vendor/bundle/ruby/2.7.0/gems/lamby-2.0.0/lib/lamby.rb:21:in `handler'",
        "/var/task/app.rb:22:in `handler'"
    ]
}

Lambda Handler

def handler(event:, context:)
  logger = Logger.new($stdout)
  logger.info('## ENVIRONMENT VARIABLES')
  logger.info(ENV.to_a)
  logger.info('## EVENT')
  logger.info(event)
  
  Rails.logger = logger
  Rails.application.config.logger = logger
  
  Lamby.handler $app, event, context, rack: :http
end

[TODO] Access Resources in an Amazon VPC

Configuring a Lambda Function to Access Resources in an Amazon VPC
https://docs.aws.amazon.com/lambda/latest/dg/vpc.html

Support Multi-Value Headers for ALB

Using the cookie-cutter app.rb w/ the ALB rack middleware, I'm seeing malformed responses. Looks like Lamby doesn't handle responses with multi-value headers. Works for me if I add the following, but I think it probably makes sense to fix the underlying issue.

def handler(event:, context:)
  result = Lamby.handler $app, event, context, rack: :alb

  # Lamby doesn't handle responses with multi-value headers.
  result[:multiValueHeaders]['Set-Cookie'] = result[:multiValueHeaders]['Set-Cookie'][0].split("\n")
  result.delete(:headers)

  result
end

Execution failed due to configuration error: Malformed Lambda proxy response

After upgrading Lamby I have the following error in API Gateway:

Execution failed due to configuration error: Malformed Lambda proxy response

This is because the output for the Lamby lambda function contains a key 'cookies' that is not supposed to be there according to the AWS doc, format must be:

{
    "isBase64Encoded": true|false,
    "statusCode": httpStatusCode,
    "headers": { "headerName": "headerValue", ... },
    "multiValueHeaders": { "headerName": ["headerValue", "headerValue2", ...], ... },
    "body": "..."
}

This is most likely due to this commit: 5fdb0c4

My output before the upgrade:

{
	"statusCode": 200,
	"headers": {
		"X-Frame-Options": "SAMEORIGIN",
		"X-XSS-Protection": "1; mode=block",
		"X-Content-Type-Options": "nosniff",
		"X-Download-Options": "noopen",
		"X-Permitted-Cross-Domain-Policies": "none",
		"Referrer-Policy": "strict-origin-when-cross-origin",
		"Content-Type": "text/html; charset=utf-8",
		"Set-Cookie": "...",
		"ETag": "...",
		"Cache-Control": "max-age=0, private, must-revalidate",
		"X-Request-Id": "...",
		"X-Runtime": "0.003934"
	},
	"body": "<!DOCTYPE html>..."
}

And after the upgrade:

{
	"statusCode": 200,
	"headers": {
		"X-Frame-Options": "SAMEORIGIN",
		"X-XSS-Protection": "1; mode=block",
		"X-Content-Type-Options": "nosniff",
		"X-Download-Options": "noopen",
		"X-Permitted-Cross-Domain-Policies": "none",
		"Referrer-Policy": "strict-origin-when-cross-origin",
		"Content-Type": "text/html; charset=utf-8",
		"ETag": "...",
		"Cache-Control": "max-age=0, private, must-revalidate",
		"X-Request-Id": "...",
		"X-Runtime": "0.003851"
	},
	"body": "<!DOCTYPE html>...",
	"cookies": [
		"_interslice_session...",
		"_interslice_session=..."
	]
}

ActiveRecord query_cache not cleared between requests

Hi @metaskills,

Again, thanks for this fantastic set of work! I've been having a lot of fun these past few days on a path finding mission.

One issue I ran across is ActiveRecord query_cache is maintained between requests.

A quick example with my discovery project:

[REDACTED] - See comment below, I have a fix & won't keep the demo sever alive much longer.

Thoughts? Recommendations?

Not sure if I'm simply missing a setting because I added ActiveRecord manually, or is something deeper like connection pool releasing?

[EDIT]
Adding stack info
Rails: 6.1.3.2
Ruby: 2.7.3
RDS: Postgres 11.11

[TODO] Database Connections

https://lamby.custominktech.com/docs/database_connections

Getting Started - Deploy Button

I wonder what it might take to get someone setup using GitHub using a Heroku style button?

EventBridge Guide

I forgot we built in EventBridge support and need to add some guides around it. Here is the PR outlining the features. #82 We will need docs on permissions and rules. For ex:

EventBridgeRule:
  Type: AWS::Events::Rule
  Properties:
    Description: String
    EventBusName: arn:aws:events:us-east-1:123456789012:event-bus/merchandising
    EventPattern: { "source": [ "myorg.products@StyleUpdate" ], "detail-type": [ "PricingChange" ] }
    Targets:
      - Arn: !GetAtt RailsLambda.Arn
        Id: MyAppProductUpdates
        RetryPolicy:
          MaximumRetryAttempts: 10
          MaximumEventAgeInSeconds: 400

EventBridgeLambdaPermission:
  Type: AWS::Lambda::Permission
  Properties:
    FunctionName:
      Ref: "RailsLambda"
    Action: "lambda:InvokeFunction"
    Principal: "events.amazonaws.com"
    SourceArn: !GetAtt EventBridgeRule.Arn

It might be cool to explore a few handler alternative steps. Like maybe promote a single PORO. Maybe the Lamby interface could include some source interface patterns.

Installing AWS CLI and AWS SAM

https://lamby.custominktech.com/docs/installing_aws_sam

Performance & Real World Usage

Because your Rails app is initialized outside the handler it should be loaded into memory and respond to requests in a very timely manner. Make sure to set the MemorySize as needed in your template.yml and you should get comparable performance to EC2. Initial tests show that basic view rendering only takes a few milliseconds. Please do share your performance results if you have any! However, I will be reporting some real world usage benchmarks soon.

Lamby Site Application

This is product site application https://github.com/customink/lamby_site running and host https://lamby.custominktech.com running on Lamby.

[TODO] Application Load Balancer Support

AWS Lambda functions as targets for Application Load Balancers - AWS Docs

As of yet this is not part of the SAM spec (aws/aws-sam-cli#1008) however it is possible if you decide to manually set things up. At some later time I will update this issue to explore and describe that manual process if others are interested.

Traditional New Relic Usage

Does the traditional way of using New Relic work or do we have to use their new distributed tracing tools?

Asset Precompile & S3 Asset Host

The idea here is to explore using a S3 bucket as an asset host and precompile the assets during build/deploy. Lambda never the asset host or concerned with compiling.

• CloudFormation to create bucket in bootstrap.
  • Ensure public access
  • Response cache headers with 1 year TTLs for all objects.
• Configure asset host in Rails to point to this bucket.
• Document if any CORS work is needed.
• Build and/or deploy process integration in bin scripts. #17

ActionCable in Lambda

How would I go by using/implementing websocket with this? Would action cable work or should a different approach be used? WIP Application: https://github.com/customink/lamby-ws

• Do custom domain names work for WebSocket API Gateways?
• Can the WS APIGW all be on the /cable path?
• Will the Cable JavaScript use the current host?
• Will APIGW be ws:// or wss:// and other security issues?

API Gateway Permissions & CloudWatch Logs

#6

Asset Hosts & Precompiling

https://lamby.custominktech.com/docs/asset_host_and_precompiling

Updateing ReadMe with Getting Started section

Hi,

I really think the ReadMe.md deserves a Getting Started section. The cookiecutter documentation currently linked is a good place to get started of course, but it also makes a number of assumptions.

I would be happy to take a stab at writing a few lines for this including:

• What AWS services are used and why
• Setting up AWS cli
• What AIM permissions are required to deploy your first function
• Running generators
• What files you might wanna go through yourself after having run the generator (e.g. template.yaml and bin/deploy to change function names etc.)
• Info about working in readonly filesystem (disabling e.g. bootsnap gem in Rails 6+, sessions etc)
• What if you wanna use asset pipeline (uncommenting nodejs in Dockerfile-build and bin/_build precompiling)
• Serving base64 (images etc) https://github.com/customink/lamby-cookiecutter/blob/master/inserts/config/environments/production.rb#L7-L10
• etc.

Speeding Up Deploys via Caching

Hi customink/@metaskills,

Thanks for this great set of gems. It really helped me wrap my head around lambda, API Gateway, SQS etc. I'm still in the learning phase so you'll have to excuse my ignorance.

At the moment, each deploy goes through a full, brand new build including installing of Gems and scratch asset precompile.

# bin/_build
rm -rf ./.lamby ./.aws-sam
cp -r . "$SAM_TEMP"
mkdir -p ./.lamby
cp -r "$SAM_TEMP" "./.lamby/$RESOURCE"
pushd "./.lamby/$RESOURCE"

To allow caching, it can become like:

LAMBY_BUILD_CACHE_DIR="./.lamby/$RESOURCE"
rsync -avu --delete \
  --exclude=log \
  --exclude=node_modules \
  --exclude=vendor/bundle \
  --exclude=vendor/bundle-dev \
  --exclude=".bundle" \
  --exclude=".lamby" \
  --exclude="tmp" \
  --exclude=".git" \
  --exclude="public/packs" \
  --exclude="public/assets" \
  --exclude="config/master.key" \
  . $LAMBY_BUILD_CACHE_DIR

cd $LAMBY_BUILD_CACHE_DIR

echo '== Bundle For Deployment =='
bundle lock --add-platform x86_64-linux
bundle config --local deployment true
bundle config --local without 'development test'
bundle config --local path './vendor/bundle'
bundle install --quiet --jobs 4

echo "== Asset Hosts & Precompiling =="
NODE_ENV='production' ./bin/rails assets:precompile

Removing all the rm -rf and moving them to build specific dockerignores allows maintaining the cache and using incremental gem-install & assets-precompile during the build stage.

First end-to-end deploy time is the same, but repeated deploys drop the time from ~15min to ~3min (my machine).

Obviously for larger projects with lots of gems & node packages this improves it further. Can add a $CLEAR_CACHE to handle busting too.

Thoughts before I dive any deeper?

Application Load Balancer (ALB) Support

For a few months now Lambda supports execution via an ALB vs ApiGateway which is overkill for a simple HTTP/HTTPS proxy. I recently saw this tweet which stated the same.

https://twitter.com/alexbdebrie/status/1131921493875712000

However, this feature request in SAM has been open for awhile.

• aws/aws-sam-cli#1008
• aws/serverless-application-model#721

What interests me most is learning what the event & context look like when using an ALB and how/if Lamby is needed.

Resources

• Using AWS Lambda with an Application Load Balancer
• ALB-Lambda-Target-HelloWorld=
• User Contributed SAM Template.yml Example
• Same But With Auth0
• Tutorial: Create an Application Load Balancer Using the AWS CLI
• ALB-Lambda-Target-HelloWorld