nytlabs/github-s3-deploy

AWS Lambda function, triggered by Github/SNS webhook, to sync new commits in an S3 bucket

github-s3-deploy's Issues

Support a push with multiple commits

Current project does not behave as expected:

• Create a file test.txt, commit it (commit sha aaa111), push it

• File exists on S3, logs show "creating test.txt"

• Now delete the file, commit again (sha bbb222), don't push yet

• Now make another commit (git add other-test-file.txt), commit has sha ccc333, push both commits bbb222 and ccc333

• Lambda function fires once, "new head" is seen as ccc333, but test.txt still exists on S3, along with other-test-file.txt... logs show that commit ccc333 was processed, but bbb222 was not

I tested this multiple times and was quite confused by the behavior. In reality, I first noticed it when I made a new commit bbb222 and then merged in a branch to get to sha ccc333 - I realized that the head_commit object in the github eventObj notification was only including the merged file changes, and did not include my changes from bbb222.

Expected behavior: Handler should be smart enough to iterate all commits in eventObj.commits array (which will include up to 20 commits at once... supporting a huge push with more than 20 commits will be much more involved, but just supporting 20 commits should be a good bandaid)

I have a full working fix for this, I am ready to submit a PR except I don't want to make it public until issue #5 is resolved

Send an SNS notification to a different topic when sync completed

It'd be really sweet if the logging could be send to another SNS topic so non-AWS console users could have confidence that the sync completed successfully.

Better Lambda resource handling

Extremely large sites may have a risk of overrunning Lambda memory and timeout limits. Large sites can consume a large amount of resources and take a long time to run.

You can mitigate this by testing. If the Lambda process "exits before completion" and reports the memory usage to be the same as the limit, then you likely need to raise the limit on the function. You may also need to expand the timeout.

A pretty large rewrite will have to occur to make this better, but for the current time, just be aware of the limits if it dies for unknown reasons.

Not open source?

Hi there,

It seems this project isn't open source. There's currently no license file and within the index.js file there is this:

Copyright: The New York Times Company

Can you please advise?

Thanks,
Steve

Use an encrypted JSON file to store configuration variables

We should be able to use a config.json file that can be encrypted/decrypted with KMS to store configuration items.

Files are sometimes not modified

Sometimes, when uploading a new version of a file, a newly changed file will not be uploaded properly. No error codes are issued when this happens, so you can never be quite sure that your work "takes". This is probably due to some async timing issue and should be easy to fix, because as the code stands now, it can't be trusted.

Add to awesome-functions

It's a good candidate for https://github.com/plutov/awesome-functions