fierycod / holy-lambda Goto Github PK

The extraordinary simple, performant, and extensible custom AWS Lambda runtime for Clojure.

Home Page: https://fierycod.github.io/holy-lambda

License: MIT License

Clojure 81.44% Makefile 13.12% Shell 0.75% Dockerfile 0.33% Emacs Lisp 0.06% Java 1.17% C# 0.52% Go 0.35% Haskell 0.57% JavaScript 0.16% Python 0.15% Ruby 0.12% Rust 0.33% BitBake 0.94%

clojure aws-lambda aws graalvm native-image babashka java

holy-lambda's Introduction

The extraordinary simple, performant, and extensible custom AWS Lambda runtime for Clojure.

Holy Lambda supports multiple backends

Babashka,
Native Clojure (GraalVM compiled),
Clojure (much faster than official AWS Java runtime),

Goals

Low cold starts - Clojure goes fast on AWS Lambda.
Multiple backends support - Unified runtime for Clojure.
Minimal API - Just stuff that gets the job done.

Non-Goals

Tight integration with deployment tools - I don't want to do this.
ClojureScript support - there is no additional value HL may provide for Clojurescript. Use either nbb or shadow-cljs with advanced compilation.

Companies & Inviduals using Holy Lambda?

nextdoc.io - 6 native lambdas: api-gateway custom authorizer, file access control, openapi data source etc.
scalably.ai - 14 native lambdas: xml transformations, sftp interactions, message routing, encryption etc.
retailic - native lambda that uses compatibility layer with ring, regular Clojure api on ARM64
day8

Extensions

holy-lambda-ring-adapter - Supports running full featured Ring apps on AWS Lambda

Resources

Current Version

Getting Help

License

Released under the MIT license.

holy-lambda's People

Contributors

Stargazers

Watchers

Forkers

ekoontz timotheosh martinklepsch eoliphan brandonstubbs nextdoc acgfbr mkropat lowecg rkx-forks rastandy vandr0iy sneakibrina viesti erickisos the-alchemist nikolap brianium drewverlee

holy-lambda's Issues

Commands which depend on :stack:sync should be restricted to use before using stack:sync

stack:compile
stack:invoke
stack:deploy
stack:pack

Errors when trying to run the code locally via `sam invoke`

Hi @FieryCod,

Thanks for the great library.

I am getting the following errors when trying to run the code locally via SAM.

My env:

java -version
openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

The command executed:

make dry-api # works great I can see the server is start locally

# then I try to invoke sam locally using this command
echo '{"message": "Hey, are you there?" }' | sam local invoke "HelloLambdaFunction" --event -

I am getting the following errors:

Reading invoke payload from stdin (you can also pass it from file with --event)
Invoking hello-lambda.core.HelloLambda (java11)
Decompressing /Users/bchoomnuan/b12n/FieryCod--holy-lambda/examples/hello-lambda/target/output.jar
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-java11:rapid-1.15.0.

Mounting /private/var/folders/c7/gkpgtgtj2fd40xv6v4ll6tpdhxy_sz/T/tmp3wf1wzgt as /var/task:ro,delegated inside runtime container
START RequestId: 12bc765d-b2f9-413d-823a-d018f3aecd63 Version: $LATEST
Error loading class hello-lambda.core.HelloLambda: java.lang.ExceptionInInitializerError
java.lang.ExceptionInInitializerError
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
    at clojure.lang.RT.classForName(RT.java:2212)
    at clojure.lang.RT.classForName(RT.java:2221)
    at clojure.lang.RT.loadClassForName(RT.java:2240)
    at clojure.lang.RT.load(RT.java:449)
    at clojure.lang.RT.load(RT.java:424)
    at clojure.core$load$fn__6857.invoke(core.clj:6115)
    at clojure.core$load.invokeStatic(core.clj:6114)
    at clojure.core$load.doInvoke(core.clj:6098)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5897)
    at clojure.core$load_one.invoke(core.clj:5892)
    at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
    at clojure.core$load_lib.invokeStatic(core.clj:5936)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$load_libs.invokeStatic(core.clj:5974)
    at clojure.core$load_libs.doInvoke(core.clj:5958)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$require.invokeStatic(core.clj:5996)
    at clojure.main$loading__6738__auto____9000.invoke(main.clj:11)
    at clojure.main__init.load(Unknown Source)
    at clojure.main__init.<clinit>(Unknown Source)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
    at clojure.lang.RT.classForName(RT.java:2212)
    at clojure.lang.RT.classForName(RT.java:2221)
    at clojure.lang.RT.loadClassForName(RT.java:2240)
    at clojure.lang.RT.load(RT.java:449)
    at clojure.lang.RT.load(RT.java:424)
    at clojure.core$load$fn__6857.invoke(core.clj:6115)
    at clojure.core$load.invokeStatic(core.clj:6114)
    at clojure.core$load.doInvoke(core.clj:6098)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5897)
    at clojure.core$load_one.invoke(core.clj:5892)
    at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
    at clojure.core$load_lib.invokeStatic(core.clj:5936)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$load_libs.invokeStatic(core.clj:5974)
    at clojure.core$load_libs.doInvoke(core.clj:5958)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$require.invokeStatic(core.clj:5996)
    at clojure.core.server$loading__6738__auto____8867.invoke(server.clj:9)
    at clojure.core.server__init.load(Unknown Source)
    at clojure.core.server__init.<clinit>(Unknown Source)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
    at clojure.lang.RT.classForName(RT.java:2212)
    at clojure.lang.RT.classForName(RT.java:2221)
    at clojure.lang.RT.loadClassForName(RT.java:2240)
    at clojure.lang.RT.load(RT.java:449)
    at clojure.lang.RT.load(RT.java:424)
    at clojure.core$load$fn__6857.invoke(core.clj:6115)
    at clojure.core$load.invokeStatic(core.clj:6114)
    at clojure.core$load.doInvoke(core.clj:6098)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5897)
    at clojure.core$load_one.invoke(core.clj:5892)
    at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
    at clojure.core$load_lib.invokeStatic(core.clj:5936)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$load_libs.invokeStatic(core.clj:5974)
    at clojure.core$load_libs.doInvoke(core.clj:5958)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$require.invokeStatic(core.clj:5996)
    at clojure.core$require.doInvoke(core.clj:5996)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.lang.Var.invoke(Var.java:384)
    at clojure.lang.RT.doInit(RT.java:491)
    at clojure.lang.RT.init(RT.java:467)
    at clojure.lang.Util.loadWithClass(Util.java:248)
    at hello-lambda.core.HelloLambda.<clinit>(Unknown Source)
    at java.base/java.lang.Class.forName0(Native Method)
    at java.base/java.lang.Class.forName(Unknown Source)
Caused by: Syntax error macroexpanding clojure.core/ns at (clojure/walk.clj:21:1).
Syntax error compiling at (clojure/core/specs/alpha.clj:1:1).
    at clojure.lang.Compiler.checkSpecs(Compiler.java:6976)
    at clojure.lang.Compiler.macroexpand1(Compiler.java:6992)
    at clojure.lang.Compiler.macroexpand(Compiler.java:7079)
    at clojure.lang.Compiler.eval(Compiler.java:7165)
    at clojure.lang.Compiler.load(Compiler.java:7640)
    at clojure.lang.RT.loadResourceScript(RT.java:381)
    at clojure.lang.RT.loadResourceScript(RT.java:372)
    at clojure.lang.RT.load(RT.java:459)
    at clojure.lang.RT.load(RT.java:424)
    at clojure.core$load$fn__6857.invoke(core.clj:6115)
    at clojure.core$load.invokeStatic(core.clj:6114)
    at clojure.core$load.doInvoke(core.clj:6098)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5897)
    at clojure.core$load_one.invoke(core.clj:5892)
    at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
    at clojure.core$load_lib.invokeStatic(core.clj:5936)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$load_libs.invokeStatic(core.clj:5974)
    at clojure.core$load_libs.doInvoke(core.clj:5958)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$require.invokeStatic(core.clj:5996)
    at clojure.core$require.doInvoke(core.clj:5996)
    at clojure.lang.RestFn.invoke(RestFn.java:436)
    at clojure.spec.alpha$loading__6434__auto____1663.invoke(alpha.clj:9)
    at clojure.spec.alpha__init.load(Unknown Source)
    at clojure.spec.alpha__init.<clinit>(Unknown Source)
    ... 84 more
Caused by: Syntax error compiling at (clojure/core/specs/alpha.clj:1:1).
    at clojure.core$throw_if.invokeStatic(core.clj:5856)
    at clojure.core$check_cyclic_dependency.invokeStatic(core.clj:5988)
    at clojure.core$load.invokeStatic(core.clj:6109)
    at clojure.core$load.doInvoke(core.clj:6098)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core$load_one.invokeStatic(core.clj:5897)
    at clojure.core$load_one.invoke(core.clj:5892)
    at clojure.core$load_lib$fn__6797.invoke(core.clj:5937)
    at clojure.core$load_lib.invokeStatic(core.clj:5936)
    at clojure.core$load_lib.doInvoke(core.clj:5917)
    at clojure.lang.RestFn.applyTo(RestFn.java:142)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$load_libs.invokeStatic(core.clj:5974)
    at clojure.core$load_libs.doInvoke(core.clj:5958)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at clojure.core$apply.invokeStatic(core.clj:669)
    at clojure.core$require.invokeStatic(core.clj:5996)
    at clojure.core$require.doInvoke(core.clj:5996)
    at clojure.lang.RestFn.invoke(RestFn.java:408)
    at clojure.core.specs.alpha$eval1309$loading__6738__auto____1310.invoke(alpha.clj:1)
    at clojure.core.specs.alpha$eval1309.invokeStatic(alpha.clj:1)
    at clojure.core.specs.alpha$eval1309.invoke(alpha.clj:1)
    at clojure.lang.Compiler.eval(Compiler.java:7181)
    at clojure.lang.Compiler.eval(Compiler.java:7170)
    at clojure.lang.Compiler.load(Compiler.java:7640)
    at clojure.lang.RT.loadResourceScript(RT.java:381)
    at clojure.lang.RT.loadResourceScript(RT.java:372)
    at clojure.lang.RT.load(RT.java:459)
    at clojure.lang.RT.load(RT.java:424)
    at clojure.lang.Compiler.ensureMacroCheck(Compiler.java:6962)
    at clojure.lang.Compiler.checkSpecs(Compiler.java:6974)
    ... 113 more
Caused by: java.lang.Exception: Cyclic load dependency: [ /clojure/spec/alpha ]->/clojure/walk->[ /clojure/spec/alpha ]->/clojure/main->/clojure/core/server
    ... 139 more

time="2021-01-03T21:27:08.903" level=panic msg="ReplyStream not available"
2021/01/03 21:27:08 http: panic serving 127.0.0.1:53576: &{0xc000158000 map[] 2021-01-03 21:27:08.9036281 +0000 UTC m=+7.279948101 panic <nil> ReplyStream not available <nil> <nil> }
goroutine 14 [running]:
net/http.(*conn).serve.func1(0xc0000755e0)
    /usr/local/go/src/net/http/server.go:1800 +0x139

Any help would be greatly appreciated.
Thanks

(5) fix `lein new holy-lambda test-holy-lambda`

This is an important entry point for beginners to understand how to use holy-lambda combine Clojure with AWS Lambda. It has some bugs that prevent new users from creating a basic beginning lambda. This issue will cover fixing this basic functionality.

[FEATURE] Add debug mode

It would be useful to provide a debug mode which could be set up via environment variable. In debug mode HL should print each step of the runtime with "time" how much something took.

Babashka runtime layer sometimes pushes artifact without `bb`.

cp: resources/native-deps: No such file or directory

Hi! Thanks this neat project :)

I created a new project with lein new holy-lambda hello-world and ran make, but ended up with the following after a native-image build:

...
[output:25]        write:   4,450.84 ms,  3.73 GB
# Printing build artifacts to: output.build_artifacts.txt
[output:25]      [total]: 121,535.17 ms,  3.73 GB
cp: resources/native-deps: No such file or directory
make: *** [latest.zip] Error 1

I guess the Leiningen template should also create the resources/native-deps directory, even though it can be empty?

Add documentation for native configurations generation

scaffold template fails if using qualified naming

cli-new fails if using a fully qualified name

clj -X:new clj-new/create :template holy-lambda :name com.company/holy-lambda-example 
Generating fresh 'lein new' holy-lambda project.
Could not create directory /Users/lowecg/tmp/com.company/holy-lambda-example. Maybe it already exists?  See also :force or --force

Given the name used above, I'd expect the following project folder and src structure (amongst other files)

.
└── holy-lambda-example
    └── src
        └── com
            └── company
                └── holy_lambda_example.cljs

Example using figwheel-main template - also note the use of the getting started message, which is a nice touch.

$ clj -X:new clj-new/create :template figwheel-main :name com.company/holy-lambda-example 
Generating fresh figwheel-main project.
  To get started:
  -->  Change into the 'holy-lambda-example' directory
  -->  Start build with 'clojure -A:fig:build'

$ tree
.
└── holy-lambda-example
    ├── README.md
    ├── deps.edn
    ├── dev.cljs.edn
    ├── figwheel-main.edn
    ├── resources
    │   └── public
    │       ├── css
    │       │   └── style.css
    │       ├── index.html
    │       └── test.html
    ├── src
    │   └── com
    │       └── company
    │           └── holy_lambda_example.cljs
    ├── target
    │   └── public
    ├── test
    │   └── com
    │       ├── company
    │       │   └── holy_lambda_example_test.cljs
    │       └── test_runner.cljs
    └── test.cljs.edn

12 directories, 11 files

Cannot create an instance of com.amazonaws.auth.AWS4Signer

oracle/graal#1367

Support setting params for stack:invoke, stack:api, stack:deploy

Provide automation for new releases via CircleCI

:infra should support :profile keyword which points to AWS profile which should be used

On nil the "default" profile should be used

(2) Support async handlers

@ekoontz This will be breaking change. Please let me know what do you think about it.

All handlers are async by default.
Regular handler is 3 arity function as in async ring:

(h/deflambda hello-lambda <
  [{:keys [event ctx] :as payload} respond raise]
  ;; A) On error do
  (raise (ex-info "Some error" {}))
  ;; B) On success do
  (respond {:body "Hello world"}))

Lambada style could be supported almost identically

(h/deflambda hello-lambda <
  [in out ctx respond raise]
  ;; A) On error do
  (raise (ex-info "Some error" {}))
  ;; B) On success do
  (respond) <-- This is only to notify that we should process the response from *out* object

Middlewares will be supported in future via injection to deflambda macro as in rum/defc:

(defn mw1
  [handler]
  (fn ([payload respond raise]
      (handler (assoc-in payload [:event :message] "Hello world") respond raise))
    ([in out ctx respond raise]
     ;; This would be awful to change anything in *in*, but it's not forbidden.
     ;; But just not recommended
     (handler in out ctx respond raise))))
  
(h/deflambda hello-lambda <
  {:middlewares [mw1]
   :custom-logger nil} ;; we can support some more opts
  [{:keys [event ctx] :as payload} respond raise]
  ;; A) On error do
  (raise (ex-info "Some error" {}))
  ;; B) On success do
  (respond {:body "Hello world"}))

stack:sync can fail if the AWS region hasn't been configured

[FEATURE] CDK support

Would be nice to integrate with CDK projects, use holy-lambda for clojure goodness in lambdas, CDK to glue together larger app. There's beta SAM/CDK support here

(1) Remove `choly`

@ekoontz I will put the number on every task which will indicate the order in which they should be realized. Please let me know whether you are interested in doing it.

Remove choly completely.

The idea behind choly was to allow:

automatic translation from template.yml (Java) -> native-template.yml (HCR)
I think that this feature is buggy and over engineered. Most people will probably target HCR only and the translation is so simple that doing it by hand should not be problematic.

Another thing is choly template supports only the AWS SAM which might suggest that HCR will work only with that tool which is not true.
managing graalvm via the docker image
This brings the new level of complexity which is not necessary for that small project.
managing envs
The core should not take care of managing the envs.

Let's keep holy-lambda as minimal runtime as it's only possible.

The core of the project should:

allow execution in both runtimes (Java + HCR)
not include 3rd party libraries in it's core
support reflect-config.json generation via native-agent and native-agent-payloads
support async mode (preferably via callback to not include 3rd party library like core.async)
be as fast as possible (add benchmark)
allow swap of default logger
not use any of dynamic bindings
provide tutorial how to develop and deploy simple lambda

(3) Remove logging

@ekoontz

Remove all calls with holy-lambda/impl.logging.clj
Remove all mentions of it from docs, examples
Favour custom logging libraries like tools.logging. Switch to it in examples
Remove LambdaLogger from context
Use println instead of log/fatal in internal code. Standarize decorations on println calls.

Use babashka tasks instead of Makefile

Insert a custom logger into aws context (native)

Only after #3
Hint in source code

Log when there is no event or when the status is different than 200 (native)

Hint in source code.
Only after #3

Strip unnecessary reflections and resources for native-image. (Experiment)

bucket exists check is failing

I am able to create the bucket successfully with bb bucket:create

[holy-lambda] Command <bucket:create>
[holy-lambda] Creating a bucket normon-monitor-96661897a56449f1954de8a4627c
[holy-lambda] Bucket normon-monitor-96661897a56449f1954de8a4627c has been succesfully created!

If I go to the AWS console, I see the bucket created

,However whenever I try to pack or deploy, the command fails because it said that the bucket does not exists and it is not able to create it

$ bb stack:pack
[holy-lambda] Command <stack:pack>
[holy-lambda] Bucket normon-monitor-96661897a56449f1954de8a4627c does not exists. Creating one!
[holy-lambda] Command <bucket:create>
[holy-lambda] Creating a bucket normon-monitor-96661897a56449f1954de8a4627c
[holy-lambda] Unable to create a bucket normon-monitor-96661897a56449f1954de8a4627c.
[holy-lambda] AWS command output:
------------------------------------------
make_bucket failed: s3://normon-monitor-96661897a56449f1954de8a4627c5555 An error occurred (BucketAlreadyOwnedByYou) when calling the CreateBucket operation: Your previous request to create the named bucket succeeded and you already own it.
------------------------------------------

This is the output of bb stack:doctor

[holy-lambda] Command <stack:doctor>
[holy-lambda] ---------------------------------------
[holy-lambda]  Checking health of holy-lambda stack
[holy-lambda]  Home directory is:        /home/miguel
[holy-lambda]  Project directory is:     /home/miguel/git/onetech/normon-monitor
[holy-lambda]  AWS SAM version:          SAM CLI, version 1.23.0
[holy-lambda]  AWS CLI version:          aws-cli/2.0.42 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.18
[holy-lambda]  AWS directory is:         /home/miguel/.aws
[holy-lambda]  AWS directory exists?:    true
[holy-lambda]  Docker version:           Docker version 20.10.6, build 370c289
[holy-lambda]  Babashka tasks sha:       75831dc72f0ddee55a151bc8e7442b67214607fc
[holy-lambda]  Babashka tasks version:   0.1.49
[holy-lambda]  Babashka version:         babashka v0.4.4
[holy-lambda]  Runtime:                  :babashka
[holy-lambda]  Runtime entrypoint:       com.nubizzi.normon-monitor.core
[holy-lambda]  Stack name:               normon-monitor-96661897a56449f1954de8a4627c40d1-stack
[holy-lambda]  S3 Bucket name:           normon-monitor-96661897a56449f1954de8a4627c
[holy-lambda]  S3 Bucket prefix:         holy-lambda
[holy-lambda]  S3 Bucket exists?:        false
[holy-lambda] ---------------------------------------

[holy-lambda] :runtime looks good
[holy-lambda] :runtime:entrypoint looks good
[holy-lambda] :stack:capabilities looks good
[holy-lambda] :runtime:bootstrap-file is supported only for :native runtime
[holy-lambda] :runtime:native-deps is supported only for :native runtime
[holy-lambda] :runtime:native-image-args are supported only for :native runtime
[holy-lambda] :stack:name looks good
[holy-lambda] property :mvn/local-repo in file deps.edn is correct
[holy-lambda] property :mvn/local-repo in file bb.edn is correct
[holy-lambda] Syncing stack is not required
[holy-lambda] :infra:region definition looks good
[holy-lambda] :infra:bucket-prefix looks good
[holy-lambda] :stack:template looks good
[holy-lambda] :infra:bucket-name looks good, but normon-monitor-96661897a56449f1954de8a4627c does not exists (use bb :bucket:create)
[holy-lambda] Required commands ["aws" "sam" "bb" "docker" "clojure" "zip" "id" "clj-kondo" "bash"] installed!

[holy-lambda] Validating template.yml
2021-06-05 20:42:07 Loading policies from IAM...
2021-06-05 20:42:09 Finished loading policies from IAM.
/home/miguel/git/onetech/normon-monitor/template.yml is a valid SAM Template

I think the significant line is

[holy-lambda] S3 Bucket exists?: false

Implement a logger which streams logs to CloudWatch (Native only)

Document new babashka runtime in whatitis.md

:infra:region should override the profile default AWS region

Optimize code, provide benchmarks

@ekoontz

Optimizations:

Remove dynamic bindings. Prefer direct calls
impl.util/call should take specified numbers of arguments afn, payload, respond, raise
Use jsonista instead of data.json
Pre generate routing on the compilation stage.

Benchmark

For each runtime: java, native

Measure invocations on cold start
Measure invocations of warmed lambda

Compare with Node.js, Go, Python runtimes on both: figure + table.

Provide nice looking documentation

fix links
fix grammar
add explanation for rest of the files in project structure section
explain how to generate the native-configuration in separate .md

template is missing Makefile

I may be using this feature prematurely, but the lein template does not appear to include a Makefile:

lein new holy-lambda getting_started
cd getting_started/
make dry-api
make: *** No rule to make target `dry-api'.  Stop.

$ ls
README.md	bb.edn		deps.edn	envs.json	resources	src		template.yml

PS This looks like being a really useful project, thank you for making it available.
PPS Any chance of publishing the work so far for the "getting started" guide?

Try out the new version of Graal and make it work with the holy-lambda

Log AWS error via custom logger (native)

Only after #3
Hint is in source code

Find out how many people would like to use 3 arity lambda invocation (Lambada style)

Vote please:
http://www.strawpoll.me/17982830

Deadline:
2019-05-14 "End of the day"

Linked documentation is not available.

Describe the bug
The documentation linked from the README is unavailable.

Page reports "API & Git import failed in build #47800" and error is reported as "java.lang.OutOfMemoryError: Java heap space"

To Reproduce
Click on doc link in README or navigate to https://cljdoc.org/d/io.github.FieryCod/holy-lambda/0.3.0/doc/tutorial.

Expected behavior
Docs.

Screenshots

Extra information:
Works if I navigate to version 0.2.1.

Fix CircleCi builds

Tests are randomly failing from time to time:
https://circleci.com/gh/FieryCod/holy-lambda/384

[BUG] Issue building/deploying

Describe the bug
I'm following the steps laid out in the tutorial, but I'm getting the following error while running (in both docker and on Lambda)
Method setDoOutput on class sun.net.www.protocol.http.HttpURLConnection not allowed!.

Full error locally

[holy-lambda] Command <stack:invoke>
Invoking school.yardstick.jobs.core.ExampleLambda (provided)
arn:aws:lambda:us-east-1:705159276995:layer:holy-lambda-babashka-runtime:1 is already cached. Skipping download
Skip pulling image and use local one: samcli/lambda:provided-ce5d90f5a0a163c2b5000f7db.

Mounting /Users/ryan/github/school-yardstick-jobs/src as /var/task:ro,delegated inside runtime container
START RequestId: 21090918-8ff9-420e-99d3-e82115f3183e Version: $LATEST
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Method setDoOutput on class sun.net.www.protocol.http.HttpURLConnection not allowed!

time="2021-07-01T04:00:24.881" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-07-01T04:00:24.881" level=error msg="INIT DONE failed: Runtime.ExitError"

Full error on Lambda

Function Logs
START RequestId: 63eb643c-0908-4693-8289-87debb9c9616 Version: $LATEST
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Method setDoOutput on class sun.net.www.protocol.http.HttpURLConnection not allowed!

END RequestId: 63eb643c-0908-4693-8289-87debb9c9616
REPORT RequestId: 63eb643c-0908-4693-8289-87debb9c9616	Duration: 399.37 ms	Billed Duration: 400 ms	Memory Size: 256 MB	Max Memory Used: 39 MB	
RequestId: 63eb643c-0908-4693-8289-87debb9c9616 Error: Runtime exited with error: exit status 1
Runtime.ExitError

To Reproduce
AFAICT, I'm following the tutorial exactly as-is.

Expected behavior
I'm not expecting this error.

Extra information:

OS: MacOS
holy-lambda version:

{:deps {org.clojure/clojure             {:mvn/version "1.10.3"}
       io.github.FieryCod/holy-lambda  {:mvn/version "0.2.1"}}

bb stack:doctor output:

[holy-lambda] Command <stack:doctor>

[holy-lambda] ---------------------------------------
[holy-lambda]  Checking health of holy-lambda stack
[holy-lambda]  Home directory is:        /Users/ryan
[holy-lambda]  Project directory is:     /Users/ryan/github/school-yardstick-jobs
[holy-lambda]  AWS SAM version:          SAM CLI, version 1.24.1
[holy-lambda]  AWS CLI version:          aws-cli/2.2.1 Python/3.8.8 Darwin/19.6.0 exe/x86_64 prompt/off
[holy-lambda]  AWS directory is:         /Users/ryan/.aws
[holy-lambda]  AWS directory exists?:    true
[holy-lambda]  Docker version:           Docker version 20.10.7, build f0df350
[holy-lambda]  Babashka tasks sha:       1c814389a321e295fd8c214ce3a541980e818fc3
[holy-lambda]  Babashka tasks version:   0.1.49
[holy-lambda]  Babashka version:         babashka v0.4.6
[holy-lambda]  Runtime:                  :babashka
[holy-lambda]  Runtime entrypoint:       school.yardstick.jobs.core
[holy-lambda]  Stack name:               jobs-fb5383-stack
[holy-lambda]  S3 Bucket name:           jobs-fb5383
[holy-lambda]  S3 Bucket prefix:         holy-lambda
[holy-lambda]  S3 Bucket exists?:        true
[holy-lambda] ---------------------------------------

[holy-lambda] :runtime looks good
[holy-lambda] :runtime:entrypoint looks good
[holy-lambda] :stack:capabilities looks good
[holy-lambda] :runtime:bootstrap-file is supported only for :native runtime
[holy-lambda] :runtime:native-deps is supported only for :native runtime
[holy-lambda] :runtime:native-image-args are supported only for :native runtime
[holy-lambda] :stack:name looks good
[holy-lambda] property :mvn/local-repo in file deps.edn is correct
[holy-lambda] property :mvn/local-repo in file bb.edn is correct
[holy-lambda] Syncing stack is not required
[holy-lambda] :infra:region definition looks good
[holy-lambda] :infra:bucket-prefix looks good
[holy-lambda] :stack:template looks good
[holy-lambda] :infra:bucket-name looks good
[holy-lambda] Required commands ["aws" "sam" "bb" "docker" "clojure" "zip" "id" "clj-kondo" "bash"] installed!

[holy-lambda] Validating template.yml
2021-07-01 00:02:59 Loading policies from IAM...
2021-07-01 00:03:01 Finished loading policies from IAM.
/Users/ryan/github/school-yardstick-jobs/template.yml is a valid SAM Template

[BUG] bb stack:invoke fails with babashka library error, event if runtime is java

Describe the bug
A correct code, one that can be deployed in AWS with the native or java runtime, fails when called with bb stack:invoke using java runtime. The lambda function uses the cognitect/aws-lib library

NOTE: I can not guarantee is correct with babashka runtime because the layers + application is bigger than 250Mb limit imposed by AWS, so the code can't be deployed in AWS.

To Reproduce

bb stack:invoke :runtime java :event-file resources/event-test-schedule.json :envs-file envs.json

This produces this output:

bb stack:invoke :runtime :java :event-file resources/event-test-schedule.json :envs-file envs.json 
[holy-lambda] Command <stack:invoke>
[holy-lambda] Build is stale. Consider recompilation via stack:compile
Invoking com.nubizzi.normon-monitor.core.ServerMonitor (java8)
Decompressing /home/miguel/git/onetech/normon-monitor/.holy-lambda/build/output.jar
arn:aws:lambda:eu-west-3:395496933615:layer:holy-lambda-babashka-runtime:2 is already cached. Skipping download
Skip pulling image and use local one: samcli/lambda:java8-2ff027c923cd792b48aa34530.

Mounting /tmp/tmpew282zgw as /var/task:ro,delegated inside runtime container
START RequestId: e9fcea64-8937-404a-87d7-a9b2f8e512ba Version: $LATEST
Downloading pod org.babashka/aws (0.0.5)
Successfully installed pod org.babashka/aws (0.0.5)
----- Error --------------------------------------------------------------------
Type:     clojure.lang.ExceptionInfo
Message:  Could not resolve symbol: clojure.lang.LockingTransaction/isRunning
Location: /var/task/clojure/tools/logging.clj:59:15
Phase:    analysis

----- Context ------------------------------------------------------------------
55:   [logger level throwable message]
56:   (if (case *force*
57:         :agent  true
58:         :direct false
59:         (and (clojure.lang.LockingTransaction/isRunning)
                  ^--- Could not resolve symbol: clojure.lang.LockingTransaction/isRunning
60:              (*tx-agent-levels* level)))
61:     (send-off *logging-agent*
62:       (fn [_#] (impl/write! logger level throwable message)))
63:     (impl/write! logger level throwable message))
64:   nil)

time="2021-06-22T14:36:32.503" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2021-06-22T14:36:32.503" level=error msg="INIT DONE failed: Runtime.ExitError"

Expected behavior
The same behavior that I have with native runtime

bb stack:invoke :runtime native :event-file resources/event-test-schedule.json :envs-file envs.json
Invoking com.nubizzi.normon-monitor.core.ServerMonitor (provided)
Decompressing /home/miguel/git/onetech/normon-monitor/.holy-lambda/build/latest.zip
arn:aws:lambda:eu-west-3:395496933615:layer:holy-lambda-babashka-runtime:2 is already cached. Skipping download
Skip pulling image and use local one: samcli/lambda:provided-2ff027c923cd792b48aa34530.

Mounting /tmp/tmpu5ggg5n2 as /var/task:ro,delegated inside runtime container
START RequestId: 4a6c256a-db61-4701-8088-fffa22dd58b9 Version: $LATEST
....
END RequestId: 4a6c256a-db61-4701-8088-fffa22dd58b9
REPORT RequestId: 4a6c256a-db61-4701-8088-fffa22dd58b9  Init Duration: 1.00 ms  Duration: 5190.54 ms    Billed Duration: 5200 ms        Memory Size: 256 MB     Max Memory Used: 256 MB

Extra information:

OS: Ubuntu 18.04 on WSL2
holy-lambda version: 0.2.1
bb stack:doctor output:

[holy-lambda] Command <stack:doctor>

[holy-lambda] ---------------------------------------
[holy-lambda]  Checking health of holy-lambda stack
[holy-lambda]  Home directory is:        /home/miguel
[holy-lambda]  Project directory is:     /home/miguel/git/onetech/normon-monitor
[holy-lambda]  AWS SAM version:          SAM CLI, version 1.23.0
[holy-lambda]  AWS CLI version:          aws-cli/2.0.42 Python/3.7.3 Linux/4.19.128-microsoft-standard exe/x86_64.ubuntu.18
[holy-lambda]  AWS directory is:         /home/miguel/.aws
[holy-lambda]  AWS directory exists?:    true
[holy-lambda]  Docker version:           Docker version 20.10.6, build 370c289
[holy-lambda]  Babashka tasks sha:       50c055b1b9addf6bfc25bca8a1deecfd3af5c36f
[holy-lambda]  Babashka tasks version:   0.1.49
[holy-lambda]  Babashka version:         babashka v0.4.4
[holy-lambda]  Runtime:                  :native
[holy-lambda]  Runtime entrypoint:       com.nubizzi.normon-monitor.core
[holy-lambda]  Stack name:               normon-monitor-96661897a56449f1954de8a4627c40d1-stack
[holy-lambda]  S3 Bucket name:           normon-monitor-96661897a56449f1954de8a4627c
[holy-lambda]  S3 Bucket prefix:         holy-lambda
[holy-lambda]  S3 Bucket exists?:        true
[holy-lambda] ---------------------------------------

[holy-lambda] :runtime looks good
[holy-lambda] :runtime:entrypoint looks good
[holy-lambda] :stack:capabilities looks good
[holy-lambda] :runtime:bootstrap-file does not exists. Default bootstrap file for :native runtime will be used!
[holy-lambda] :runtime:version is supported only for :babashka runtime
[holy-lambda] :runtime:pods are supported only for :babashka runtime
[holy-lambda] :stack:name looks good
[holy-lambda] property :mvn/local-repo in file deps.edn is correct
[holy-lambda] property :mvn/local-repo in file bb.edn is correct
[holy-lambda] Syncing stack is not required
[holy-lambda] :infra:region definition looks good
[holy-lambda] :infra:bucket-prefix looks good
[holy-lambda] :stack:template looks good
[holy-lambda] :infra:bucket-name looks good
[holy-lambda] Required commands ["aws" "sam" "bb" "docker" "clojure" "zip" "id" "clj-kondo" "bash"] installed!

[holy-lambda] Validating template.yml
2021-06-22 16:47:51 Loading policies from IAM...
2021-06-22 16:47:55 Finished loading policies from IAM.
/home/miguel/git/onetech/normon-monitor/template.yml is a valid SAM Template

deps.edn:

{:deps           {org.clojure/clojure            {:mvn/version "1.10.3"}
                  org.clojure/spec.alpha         {:mvn/version "0.2.194"}
                  org.clojure/tools.logging      {:mvn/version "1.1.0"}
                  io.github.FieryCod/holy-lambda {:mvn/version "0.2.1"}
                  clj-commons/clj-ssh            {:mvn/version "0.5.15"}
                  http-kit/http-kit              {:mvn/version "2.5.3"}
                  org.clojure/data.json          {:mvn/version "2.3.1"}
                  mount/mount                    {:mvn/version "0.1.16"}
                  com.cognitect.aws/api          {:mvn/version "0.8.505"},
                  com.cognitect.aws/endpoints    {:mvn/version "1.1.11.1001"}
                  com.cognitect.aws/ec2          {:mvn/version "811.2.889.0", :aws/serviceFullName "Amazon Elastic Compute Cloud"}
                  com.cognitect.aws/s3           {:mvn/version "811.2.889.0"}}

 ;; Local repositories should be downloaded to `.holy-lambda` directory
 ;; This way project artifacts are separated from global ~/.m2 and only project artifacts
 ;; are deployed to AWS Lambda
 :mvn/local-repo ".holy-lambda/.m2"
 :paths          ["src" "resources"]
 :aliases        {:dev {:extra-paths ["src/dev"]
                        :extra-deps  {org.clojure/tools.namespace {:mvn/version "1.1.0"}
                                      expound/expound             {:mvn/version "0.8.7"}}}}}
```

Add AWS lambda layer for babashka

Publish layer on aws sam repo https://docs.aws.amazon.com/serverlessrepo/latest/devguide/serverlessrepo-publishing-applications.html
Get a verified author badge
Roll out a new release.

Bucket creation should fail early when error occurs and propagate it

Provide test framework for examples

Find missing properties for client context (native)

Hint in source code.
We need to find those:

{:client nil :custom nil :environment nil}

Provide getting-started.md in documentation

[BUG] bb commands run slow, when running under M1 MacOS

Describe the bug
HL uses docker under the hood to make cross platform development environment. Unfortunately Docker runs slowly on M1, most probably, because GraalVM did not yet released official GraalVM CE Docker images that target arm64.

Example warning from running bb stack:compile:

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

Upstream issue to track: oracle/graal#2666

Extra information:

OS: [e.g. MacOS] Mac OS M1
holy-lambda version: irrelevant
bb stack:doctor output: irrelevant

[FEATURE] Add built in support for for application/edn

...

Why the requestId in requestContext is different than the one taken from "Lambda-Runtime-Aws-Request-Id" header?

Tutorial expects creation of a REST API, but template.yml creates an HTTP API

It looks as though the tutorial (specifically End to End: Calling the Lambda from API Gateway) was never updated to match this commit.

:infra:region should support nil value. In such case region should be derived from the AWS profile

Check if docker is running before stack:sync

For MacOS:

pgrep -f Docker.app > /dev/null || echo "Docker is not running!"

For Linux:

pgrep -f docker > /dev/null || echo "Docker is not running!"

Find missing properties for identity (native)

Hint in source code.

  {:identityId "????"
   :identityPoolId (-> event :requestContext :identity :cognitoIdentityPoolId)}

tools.deps template

This would allow using deps.edn as a Lambda project.

We'd probably need a Make target that does AOT compilation at least. Also, the JVM Runtime needs a way to generate a zip file.

To select between Leiningen and deps.edn, maybe provide an option to the lein new command than generates different scaffolding, e.g.:

lein new holy-lambda <my-lambda> :+tool.deps

A separate template project could be possible but would probably have a lot of duplication with the Leinigen template. Probably good to test that stuff works with clj-new.

Add support for node.js runtime

Holy lambda may support arbitrary Clojure alike runtime via conditional read. For Node.js runtime we can choose one of two paths:

Create custom runtime with LTS Node.js version embedded. Use native/entrypoint to create a proxy-like function that takes the event and context, then translates it to a holy-lambda request shape.

Pros of this approach are:

handler name in template.yml does not need to be changed

Cons:

we would have to create a custom runtime layer and publish it to SAM application repository
one would have to deploy a layer first before one could use it locally

Rely on Node.js runtime. Use deflambda to create an exported function that takes holy-lambda like a request and returns a promise.

Pros of this approach are:

creating such runtime takes 2-3 hours of work
we rely on the official Node.js runtime which means that the user can switch to any Node.js version user wants.

Cons:

User would have to specify the proper handler name

Rethink how to pack the code. Maybe ncc?

HL should check if docker image is pulled after docker running check

In case image is pulled then do nothing. Otherwise inform about image pulling and pull the image.

fierycod / holy-lambda Goto Github PK

holy-lambda's Introduction

Goals

Non-Goals

Companies & Inviduals using Holy Lambda?

Extensions

Resources

Current Version

Getting Help

License

holy-lambda's People

Contributors

Stargazers

Watchers

Forkers

holy-lambda's Issues

Recommend Projects

Recommend Topics

Recommend Org