Giter Site home page Giter Site logo

middyjs / middy Goto Github PK

View Code? Open in Web Editor NEW
3.6K 35.0 372.0 45.58 MB

๐Ÿ›ต The stylish Node.js middleware engine for AWS Lambda ๐Ÿ›ต

Home Page: https://middy.js.org

License: MIT License

JavaScript 87.85% TypeScript 11.61% Shell 0.02% CSS 0.52%
serverless aws aws-lambda framework middleware lambda-handler lambda nodejs javascript typescript

middy's People

Contributors

carlnordenfelt avatar chris-heathwood-uoy avatar chrisandrews7 avatar cjbt avatar denovodavid avatar dependabot[bot] avatar dkatavic avatar dnicolson avatar dreamorosi avatar greenkeeper[bot] avatar hsz avatar jimipedros avatar josesantacruz avatar leog avatar lmammino avatar metrue avatar mju-spyrosoft avatar munierujp avatar naorpeled avatar ohjann avatar ossareh avatar puppo avatar randomhash avatar sdomagala avatar theburningmonk avatar thejuan avatar tulsidas avatar vicary avatar vladgolubev avatar willfarrell avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

middy's Issues

Update documentation

  • Explain how error handling works
  • Update list of middlewares
  • Provide documentation for middlewares

Move 'doNotWaitForEmptyEventLoop' middleware from 'after' step to 'before'?

Today I added this middleware โฌ‡๏ธ and I discovered my Lambdas were timing out when they didn't pass validation

module.exports = () => ({
after: (handler, next) => {
handler.context.callbackWaitsForEmptyEventLoop = false

It sets false in after step, meaning after handler execution. But if validation fails, handler never runs, and I get a timeout.

I changed after to before and it works as expected now.

Any risk making the same change in the framework itself? Was there a reason to do so specifically?

I mean it's a tiny change, it's faster to submit a PR then writing down this ๐Ÿ˜… but who knows, was it by design?

EISDIR: illegal operation on a directory when doing a build

Got this error when trying to do a build. Tried simple googling but couldnt find a suitable solution.

By first guess is that you try to copy a file to fonts which is a folder and cant be overriden.
I think we need to copy like this instead:
node_modules/jsdoc/templates/default/static/fonts/OpenSans-Bold-webfont.eot' -> 'docs/middy/0.2.11/fonts/OpenSans-Bold-webfont.eot'

Node Version: 8.5.0
NPM Version: 5.3.0

OS: macOS Sierra

Terminal Output:

$ npm run build

> [email protected] build /Users/philipandersson/Desktop/middy
> npm run build:docs && npm run build:readme


> [email protected] build:docs /Users/philipandersson/Desktop/middy
> jsdoc --readme README.md --package package.json --destination docs src

fs.js:1919
  binding.copyFile(src, dest, flags);
          ^

Error: EISDIR: illegal operation on a directory, copyfile '/Users/philipandersson/Desktop/middy/node_modules/jsdoc/templates/default/static/fonts/OpenSans-Bold-webfont.eot' -> 'docs/middy/0.2.11/fonts'
    at Object.fs.copyFileSync (fs.js:1919:11)
    at /Users/philipandersson/Desktop/middy/node_modules/jsdoc/templates/default/publish.js:516:12
    at Array.forEach (<anonymous>)
    at Object.exports.publish (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/templates/default/publish.js:512:17)
    at Object.module.exports.cli.generateDocs (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/cli.js:448:35)
    at Object.module.exports.cli.processParseResults (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/cli.js:399:20)
    at module.exports.cli.main (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/cli.js:240:14)
    at Object.module.exports.cli.runCommand (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/cli.js:189:5)
    at /Users/philipandersson/Desktop/middy/node_modules/jsdoc/jsdoc.js:105:9
    at Object.<anonymous> (/Users/philipandersson/Desktop/middy/node_modules/jsdoc/jsdoc.js:106:3)
    at Module._compile (module.js:624:30)
    at Object.Module._extensions..js (module.js:635:10)
    at Module.load (module.js:545:32)
    at tryModuleLoad (module.js:508:12)
    at Function.Module._load (module.js:500:3)
    at Function.Module.runMain (module.js:665:10)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build:docs: `jsdoc --readme README.md --package package.json --destination docs src`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build:docs script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/philipandersson/.npm/_logs/2017-10-02T22_22_28_445Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] build: `npm run build:docs && npm run build:readme`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/philipandersson/.npm/_logs/2017-10-02T22_22_28_462Z-debug.log

Add missing typescript definitions

I noticed this package doesn't have TypeScript definitions for the middy/middlewares submodule which makes it a little bit painful to use with TypeScript. Can I interest you in a pull request?

Api Gateway proxy event normalizer middleware

A common issue with Api gateway proxy event is that the keys:

pathParameters and queryStringParameters are created in the event if (and only if) at least one parameter exists.

this will lead to have to do the following tedious checks in the code:

console.log(event.queryStringParameters ? event.queryStringParameters.foo : undefined)

I'd love to have a middleware that takes care to normalize the event in this case and make sure we always have those key (mapped to an empty object) so that we don't need to have ternary conditions or a bunch of if statements every time we want to access one of those values in the collection.

Integration with other FaaS solutions

This is a really great tool for getting HTTP based functions up and running on AWS.

I am curious if there is any interest/demand to support other FaaS solutions such as kubeless, fission, Azure, etc?

Validation error with async handler results in timeout

When using an async handler and during the validation, an error occurs the function times out. The code below results in a timeout for me

const createHandler: Handler = async (event: any, context: Context, callback: Callback) => {
    await exampleAsyncMethod
    const { name } = event.body
    return okResponse
}

export const handler = middy(createHandler)
    .use(eventLoopMiddleware())
    .use(validator({ inputSchema }))
    .use(httpErrorHandler())

Changing it to a none async handler it works fine.


const createHandler: Handler = (event: any, context: Context, callback: Callback) => {
    exampleAsyncMethod.then(
          // code
    })
}

export const handler = middy(createHandler)
    .use(eventLoopMiddleware())
    .use(validator({ inputSchema }))
    .use(httpErrorHandler())

I am using my own middleware for the event loop because of this issue. Is there any reason why async handlers are not working with the middleware?

Interrupting middlewares running progress

Hello guys. Is there a way to interrupt middleware in the middle of execution and execute callback to return something as result as in example

module.exports = (config) => {
    return ({
        before: (handler, next) => {
            if(true) {
                console.log('Lambda is warm!')
                return handler.callback(null, 'Lambda is warm!')
            }

            next()
        }
    })
  }

Or is there any other way to do it?

Best regards

Various Middlewares

Hey folks! I came across Middy a few days ago, and I've been slowly reorganizing one of my applications to take advantage of it. I've also generated a small library of middleware that someone else may be interested in. Take a look and feel free to give me some feedback.

https://github.com/i-am-kenny/remiddy

validation error broken? maybe related to serverless-offline / serverless-webpack plugin?

Just getting started and can't seem to get validation error messages working based on the README. We're using the es7 webpack plugin + serverless-offline, but I don't think that is why.

No matter what we do, we always get a strange Unexpected 'E' response body.

Tried debugging, going as far to the actual source and making sure all our middleware is hit in the correct order and everything, but no dice.

Anyone able to take a look at / spot a stupid mistake we're making? Thanks in advance

handler.js

const middy = require('middy')
const { urlEncodeBodyParser, validator, httpErrorHandler, cors, jsonBodyParser } = require('middy/middlewares')

const dummy = (event, context, callback) => callback(
  { statusCode: 200, body: JSON.stringify({message: 'hi'})}
)
const demo = middy( dummy )
  .use(urlEncodeBodyParser()) // parses the request body when it's a JSON and converts it to an object
  .use(validator({inputSchema})) // validates the input
  .use(httpErrorHandler()) // handles common http errors and returns proper responses

module.exports = { demo }

serverless.yml

# NOTE: update this with your service name
service: demo

# Use the serverless-webpack plugin to transpile ES6
plugins:
  - serverless-webpack
  - serverless-offline


# Enable auto-packing of external modules
custom:
  webpackIncludeModules: true
  stage: "${opt:stage, self:provider.stage}"

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1

functions:
  demo:    
    handler: handler.demo
    events:
      - http:
          cors: true
          path: demo/
          method: POST    

NormalizeHTTPHeaders middleware

Following several discussions with @vladgolubev and @dkatavic (see #85 and #75) I think we should have a middleware that makes sure that the headers in the lambda-proxy integration from API Gateway are always exposed in canonical format (dash-separated-camelcasing).

The idea of this middleware would be to go over all the available headers and reconstruct the object with all the keys properly normalized.

Thoughts?

[Discussion] Move middy into dedicated organisation

What do you think about moving middy to a dedicated middy organisation on GitHub?

I am personally up for it and I can provide few good reasons for it:

  1. An independent organisation might seem more attractive to other contributors and the project could potentially reach a broader adoption (which, as a side effect, will also have a positive impact on Planet 9 in terms of validation and support)
  2. We might use the new middy organisation to host related middy projects (external middlewares, benchmarks, website source, etc.)
  3. We might be encouraged to spend some time even outside Planet 9

As a downside, of course, we might lose the support from Planet 9 which might not want to keep investing (a bit of) our time in developing it.

Let me know your view and vote ๐Ÿ‘ or ๐Ÿ‘Ž to this main comment if you are in favour or not to this idea.

CC: @padraigobrien @dkatavic @acambas @peterjcaulfield @techfort @codecrunchers @augeva @daveanderson-ie

Support for async/await

Coming out from a discussion in #49, it might be very useful to support async functions as middlewares.

The idea is that we might support middlewares with the following syntax:

export default function() {
  return {
    before: async (handler) => {
       // await async stuff
       return 'somevalue'  // equivalent to calling next(null, 'somevalue')
    }
  };
}

Add @DavidWells to the core team

Hello all,
after a PR and a private discussion i would like to propose @DavidWells from the Serverless framework to be part of the core team of this project so that he can contribute more and, most importantly, help on voting features and getting them merged.

Please add your ๐Ÿ‘ or ๐Ÿ‘Ž or ๐Ÿ˜ . If we get 3 ๐Ÿ‘ i will consider this to be commonly agreed. If you are against this or don't like this way of voting, please provide some rationale so that we can try to create a better framework for these kind of decisions in the future ๐Ÿ˜‰

CC @acambas @dkatavic @peterjcaulfield @padraigobrien @techfort

Improve Http Error messages

I altered the httpErrorHandler to follow the JSON spec http://jsonapi.org/examples/#error-objects-basics.

Should this change get applied across the board? Or should I just use this as my own custom middleware?

Whats the default lambda integration you guys are using? (we typically use lambda-proxy as the serverless framework default)

Here is my altered error handler for nice errors =)

const { HttpError } = require('http-errors')

module.exports = () => ({
  onError: (handler, next) => {
    if (handler.error instanceof HttpError) {
        // as per JSON spec http://jsonapi.org/examples/#error-objects-basics
        handler.response = {
          body: JSON.stringify({
            errors: [{
              status: handler.error.statusCode,
              message: handler.error.message,
              detail: handler.error.details
            }]
        })
      }
      return next()
    }

    return next(handler.error)
  }
})

It looks like:

image

Before with body: handler.error.message (no json response)

image

Love this freakin project! Great work ๐ŸŽ‰

jsonBodyParser shouldn't be case-sensitive

A bunch of tools encode the content-type header using different casing, and this causes the endpoint to fail as the body is not JSON decoded before handing it over to my function code.

Error handlers not catching properly with async middlewares

An async middleware with async keyword or just a middleware that implements asynchronous work, doesn't seem to properly bubble up the error object for middlewares that implement the onError method.

This doesn't seem to be properly caught:

export default function() {
  return {
    before: async (handler, next) => {
      throw new Error('wow');
    }
  };
}

Or even something like this:

export default function() {
  return {
    before: (handler, next) => {
      setTimeout(() => {
        throw new Error('wow');
      }, 10);
    }
  };
}

Any ideas?

Create validator middleware

Expected implementation:

module.exports = (inputSchema, outputSchema, options) => ({
  before: (handler, next) => {
    // validate the input using the input schema
    // if it fails throws a BadRequest using http-errors library (check json body parser for an example)
  },
  after: (handler, next) => {
    // if there is an output schema, do the same validation logic with the response
  }
})

XML Body Parser

I have a lambda running which receives XML in the body, parses it and does a few things with it.
I think it would be nice to have an xmlBodyParser middleware (like the json one).

Im using jindw/xmldom for parsing the xml and it work great on lambda as well.

But instead of this repo holding the dependency of xmldom maybe require the user to install it by themselves if they ever use the xmlBodyParser? (so, kind of peerDependency)

I would love to create a PR on this if you feel its appropriate to have as a middleware.

CORS middleware

It would be great to have a middleware that automatically adds the headers needed for enabling CORS in the response object. For example:

{
  headers: {
    'Access-Control-Allow-Origin': '*'
  }
}

The middleware needs to be configurable with an option to specify different origins (it will default to *).

Also, it will be good to have in place the following checks:

  • Add the header only if the current response looks like an API Gateway response
  • Don't override the header if it already exists

"middify" express/koa middleware

Not an issue as much as something we could think of to re-use some of the fantastic work that already exists in the OSS world.
We could create a middify() function that takes any express/koa middleware and makes it lambda compatible, not sure about implementation details, but it should just be a matter of translating the (req, res, next) => {} format into (handler, next) => {} or something.. cc @lmammino

Support for async/await in handler

Since version 0.3.0 we have support for async/await in middlewares.

We could potentially apply the same approach to the handler. Since the handler is invoked and managed by the middy wrapper, we could support different syntaxes including handlers returning promises and consequently async/await handlers.

These are the two syntaxes I envision might be cool to support:

const handler = middy((event, context) => {
  return Promise.resolve({
    statusCode: 200,
    body: 'hello'
  })
})

and, with async/await:

const handler = middy(async (event, context) => {
  return {
    statusCode: 200,
    body: 'hello'
  }
})

Any thoughts?

I think this is feasible by addressing the handler runtime logic here.

httpErrorHandler not working correctly

Hey @middyjs. Love the project. I'll keep an eye on tickets and chip in where I can help.

It appears to me that httpErrorHandler isn't working correctly. Serverless is returning a 200 when I throw an Error. Consider the code in the middleware:

https://github.com/middyjs/middy/blob/master/src/middlewares/httpErrorHandler.js#L5

And this test code:

import HttpError from 'http-errors';
import createError from 'http-errors';

console.log( new createError.UnprocessableEntity() instanceof HttpError );
// false
console.log( new createError.UnprocessableEntity() instanceof createError.HttpError );
// true

Which to me suggests a bug.. But I may just be misusing the library. (Warning: I am not a node guy).

Cheers!

Rename error middlewares?

Currently, for adding error middleware we are calling error method middy.error(() ->). Should we rename it to onError middy.onError(() ->) to be more expressive?

Postgres connection pool example

Here is an example of middy middleware connecting to postgres by @dschep

const pg = require('pg-promise')();
const {getParameter} = require('./ssmParameterStore');

const getDb = () => getParameter('db_password')
  .then((password) => pg({
    host: process.env.DB_HOST,
    port: Number(process.env.DB_PORT),
    database: process.env.DB_DATABASE,
    user: process.env.DB_USER,
    password,
  }));

const dbMiddleware = () => ({
  before: (handler) => getDb().then((db) => {
    handler.context.db = db; 
  }),
  after: (handler) => {
    handler.context.db.$pool.end();
  },
  onError: (handler) => {
    handler.context.db.$pool.end();
    throw handler.error;
  },
})


module.exports = {getDb, dbMiddleware};

We should start collecting examples in the wild and housing them somewhere =)

props to @alexdebrie for the find

HTTP Content negotiation parser

I am currently working on a middleware that will simplify HTTP content negotiation parsing and easily allow an API gateway Lambda function to support multiple charsets, encoding (e.g. compression algorithms), languages or content-types.

My current approach is to use the module negotiator (used also by express and koa) to parse the content negotiation headers and expand the current event.

For example, the middleware will parse the incoming Accept header and expose the resulting information in something like event.preferredMediaTypes.

A usage example might look like the following:

const middy = require('middy')
const { httpContentNegotiation } = require('middy/middlewares')

const handler = middy((event, context, callback) => {
  if (event.preferredMediaTypes[0] === 'application/json') {
    return callback(null, {stastusCode: 200, body: JSON.stringify({foo: 'bar'})})
  }

  if (event.preferredMediaTypes[0] === 'application/xml') {
    return callback(null, {stastusCode: 200, body: '<foo>bar</foo>'})
  }

  return callback(null, {statusCode: 415, body: 'Unsupported Media Type'})
})

handler.use(httpContentNegotiation())

module.exports = { handler }

This middleware could be the basis to create another middleware that allows to automatically serialize the API Gateway response body into the user preferred format (e.g. JSON vs XML). The combination of both middlewares should be enough to close issue number #64 (Improve HTTP error messages), as it will be easy to automatically serialize the error in the expected format.

I hope I will have a PR ready soon, meanwhile feel free to throw here suggestions or ideas ๐Ÿ˜‰

Perfomance issue

I found, that such code:
const { doNotWaitForEmptyEventLoop } = require('middy/middlewares')
or actually importing any middleware, and any amount of it adds aditional 110 +-5 ms to code execution

My lambda without adding any middlewares running approximately 35-40 seconds. And if I want to add some middleware from you library, it becomes 110 ms seconds more. In case of high availability microservices it's not really good.

BTW middy itself without attaching any existed in library middlewares works fine.

Do you have any idea, how to solve this?

Best regards

P.S: the way you can test it:

console.time('test')

const middy = require('middy')
//const { doNotWaitForEmptyEventLoop } = require('middy/middlewares')
const test = require('tape')
const utils = require('aws-lambda-test-utils')
const mockContextCreator = utils.mockContextCreator;

let handler = function(event, context) {
    context.callbackWaitsForEmptyEventLoop = false
    context.succeed("succeed")
}

handler = middy(handler)
//    .use(doNotWaitForEmptyEventLoop())

test('LambdaTest', function(t){
    t.test("test", function(st) {
      function test(result){
        console.log(result)
        st.end()
        console.timeEnd('test')
        process.exit()
      };
      const context = mockContextCreator({}, test); // no options and test as the callback
      handler({}, context);
    });
    t.end();
  });

Warmup middleware

In a discussion in #72, it seemed that middy users would benefit from having an implementation of a warm-up middleware.

The idea of warm-up helps with addressing the cold start issue by just executing the lambda on a schedule every 5 or 10 minutes to make sure there is always at least one container bootstrapped.

When the lambda runs because of the warm-up schedule, it is not supposed to run business logic code (as that would possibly lead to undesired side effects), so it has to exit immediately.

The warmupMiddleware should run a before function that does basically 2 things:

  1. detect if the lambda is running because of the warm-up schedule by checking the event data
  2. if that's the case the middleware has to stop the execution of the lambda by completing the execution successfully and avoiding to run all the other middlewares.

Regarding point 1, we can have an option to define the check as a function that accepts the event as an argument and returns true if the execution is caused by the warm-up schedule and false otherwise. The default check can be compatible with serverless-plugin-warmup.

Regarding point 2, we might want to wait for #72 to be resolved so that we have an official "clean" way to stop the middleware execution and return early.

In any case, this is a potential implementation adapted from the one proposed by @DavidWells:

module.exports = (config) => {
  const isWarmingUp = config.isWarmingUp || (event) => {
    return event.source === 'serverless-warmup-plugin'
  }
  return ({
    before: (handler, next) => {
      if (isWarmingUp(event)) {
          console.log('๐Ÿ‘‹ Exiting early via warmUpMiddleware')
          return handler.context.succeed() // this will probably change when #72 is closed
      }
      next()
    }
  })
}

Thoughts?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.