jorgebastida/gordon

λ Gordon is a tool to create, wire and deploy AWS Lambdas using CloudFormation

gordon's Issues

Add support for SNS as Event Source

Hi

Thanks for gordon! It really rocks! This is a feature request to add support for creating an SNS Topic as an Event Source for Lambda, much like the CloudWatch Events is working currently.

The idea would be that you'd be able to define a Topic name and that it would be linked to the Lambda function specified.

Disclaimer: I've read this FAQ and this issue (#58) but I don't think they are contrary to this idea.

Telegram Bot Example

Seems to be a bit of an issue with the Telegram bot example. Tried getting it going as is (Just updating my tokens) and upon apply:

$ gordon apply
Applying project...
  0001_p.json (cloudformation)
    CREATE_FAILED waiting... |
  0002_pr_r.json (custom)
Traceback (most recent call last):
  File "C:\Python27\Scripts\gordon-script.py", line 9, in <module>
    load_entry_point('gordon==0.2.0', 'console_scripts', 'gordon')()
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\bin.py", line 97, in main
    getattr(obj, options.func)()
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\core.py", line 432, in apply
    getattr(self, 'apply_{}_template'.format(template_type))(name, filename, context)
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\core.py", line 476, in apply_custom_template
    outputs = template.apply(context, self)
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\actions.py", line 131, in apply
    action_outputs[action.name] = action.apply(context, project)
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\actions.py", line 181, in apply
    self.bucket = self._get('bucket', self.context)
  File "C:\Python27\lib\site-packages\gordon-0.2.0-py2.7.egg\gordon\actions.py", line 156, in _get
    value = context[value.name]
KeyError: u'CodeBucket'

Any idea whats going on here? I tried adding another option to settings.yml as a duplicate of "code-bucket" renamed as "CodeBucket" and still no luck.

Enable more protocols for build phase

Why BASE_BUILD_PROTOCOLS has only ref protocol? Is it possible to add env protocol as well? In my use case I run gondon as part of Makefile target and I need to use pip from virtualenv, not global one. I see the only way to pass right absolute pip-path setting is through environmental variable, but env protocol is not supported in the build phase.

Do stages/contexts work with `gordon run`?

I'm trying to use gordon run to test a lambda but it is failing trying read the context json from the path .context. It is hard to tell from the docs whether this is expected or not.

Error building lambda 'version'!

Running into this problem:

gordon build --debug
Loading project resources
Loading installed applications
  contrib_lambdas:
    ✓ lambdas:version
  firstapp:
    ✓ lambdas:helloworld
Building project...
  0001_p.json
      cp -Rf * /Users/raj/.gordon/tmp7a1e6o
      echo "[install]
      prefix=" > /Users/raj/.gordon/tmp7a1e6o/setup.cfg
      pip install -r requirements.txt -q -t /Users/raj/.gordon/tmp7a1e6o

Error building lambda 'version'!
cmd: pip install -r requirements.txt -q -t /Users/raj/.gordon/tmp7a1e6o

Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-pLvUZf/cfn-response/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-0t4MeU-record/install-record.txt --single-version-externally-managed --compile --home=/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/tmpoClcpV" failed with error code 1 in /private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-pLvUZf/cfn-response/

I had to comment out the line that deletes the requirements (maybe don't cleanup if using --debug or have a second flag?) which is unexpectedly stored in ~/.gordon (could this go in _build as well?) and then ran the install command with maximum verbosity (pip install -r requirements.txt -q -t /Users/raj/.gordon/tmpMcrEwn --log test.txt):

Collecting cfn-response (from -r requirements.txt (line 1))
  1 location(s) to search for versions of cfn-response:
  * https://pypi.python.org/simple/cfn-response/
  Getting page https://pypi.python.org/simple/cfn-response/
  Looking up "https://pypi.python.org/simple/cfn-response/" in the cache
  Current age based on date: 427
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 427
  Analyzing links from page https://pypi.python.org/simple/cfn-response/
    Found link https://pypi.python.org/packages/4a/43/c36e466a91de38cc7e6b274e00535a7b35ae68730017848dfb167f4703d8/cfn-response-0.0.4.tar.gz#md5=a78d734fe2551a400a87d4ce8749ad05 (from https://pypi.python.org/simple/cfn-response/), version: 0.0.4
    Found link https://pypi.python.org/packages/86/bf/cff7c0a4308b9cff5a73d75d9858a3a2a1ea16a614bb934ba5aeb96d933f/cfn-response-0.0.2.tar.gz#md5=3d2b13e730e6e8b1541b2cf32e61f53b (from https://pypi.python.org/simple/cfn-response/), version: 0.0.2
    Found link https://pypi.python.org/packages/ce/1a/49e98a3c6af11284e7c408e74ac0e57a1b83e87df77db7910569aa9e7e0a/cfn-response-0.0.1.tar.gz#md5=23f3922cde8254b515212ae4a2be4b5f (from https://pypi.python.org/simple/cfn-response/), version: 0.0.1
    Found link https://pypi.python.org/packages/f3/ff/4f00f954e4772ed81f598fd1151825b05cdab515c4211ca4880dd0eebaa2/cfn-response-0.0.3.tar.gz#md5=9cc0f3cf33a7282af2284fa633b0619a (from https://pypi.python.org/simple/cfn-response/), version: 0.0.3
  Using version 0.0.4 (newest of versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4)
  Looking up "https://pypi.python.org/packages/4a/43/c36e466a91de38cc7e6b274e00535a7b35ae68730017848dfb167f4703d8/cfn-response-0.0.4.tar.gz" in the cache
  Current age based on date: 1952
  Freshness lifetime from max-age: 31557600
  The response is "fresh", returning cached response
  31557600 > 1952
  Using cached cfn-response-0.0.4.tar.gz
  Downloading from URL https://pypi.python.org/packages/4a/43/c36e466a91de38cc7e6b274e00535a7b35ae68730017848dfb167f4703d8/cfn-response-0.0.4.tar.gz#md5=a78d734fe2551a400a87d4ce8749ad05 (from https://pypi.python.org/simple/cfn-response/)
  Running setup.py (path:/private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/setup.py) egg_info for package cfn-response
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/cfn_response.egg-info
    writing pip-egg-info/cfn_response.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/cfn_response.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/cfn_response.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/cfn_response.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/cfn_response.egg-info/SOURCES.txt'
    writing manifest file 'pip-egg-info/cfn_response.egg-info/SOURCES.txt'
  Source in /private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response has version 0.0.4, which satisfies requirement cfn-response from https://pypi.python.org/packages/4a/43/c36e466a91de38cc7e6b274e00535a7b35ae68730017848dfb167f4703d8/cfn-response-0.0.4.tar.gz#md5=a78d734fe2551a400a87d4ce8749ad05 (from -r requirements.txt (line 1))
Installing collected packages: cfn-response
    Running command /usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-UOPjPW-record/install-record.txt --single-version-externally-managed --compile --home=/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/tmpPioLFm
    running install
    error: must supply either home or prefix/exec-prefix -- not both
Cleaning up...
  Removing source in /private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response
Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-UOPjPW-record/install-record.txt --single-version-externally-managed --compile --home=/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/tmpPioLFm" failed with error code 1 in /private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/
Exception information:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/site-packages/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_set.py", line 742, in install
    **kwargs
  File "/usr/local/lib/python2.7/site-packages/pip/req/req_install.py", line 880, in install
    spinner=spinner,
  File "/usr/local/lib/python2.7/site-packages/pip/utils/__init__.py", line 718, in call_subprocess
    % (command_desc, proc.returncode, cwd))
InstallationError: Command "/usr/local/opt/python/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-UOPjPW-record/install-record.txt --single-version-externally-managed --compile --home=/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/tmpPioLFm" failed with error code 1 in /private/var/folders/10/frng74qn5dj4vpmr8x76x7fw0000gn/T/pip-build-YZZvSq/cfn-response/

key line being error: must supply either home or prefix/exec-prefix -- not both which caused me to try: pip install -r requirements.txt -q --log test.txt which worked fine.

Because of the multiple directories being involved I'm not sure what your recommendation is to fix it, but removing -t {target} from the pip command in lambdas.py did seem to work

Anyway, hope this isn't just something dumb on my end because I haven't seen others complaining and I tried to explain all the steps involved.

Also, great work, was really into "serverless framework" for a while until I switched gears. I'm switching back pretty soon here and this project has a lot of promise, although it'll take a couple of days before I wrap my head around everything with this project.

Secrets management

How secrets should be managed using gordon? Can we imagine that gordon would build a file similar to .context called .secrets and leverage KMS to encrypt it?

Files not copied in the tmp directory for gordon run

Run the telegram api example with gordon run

fs.js:549
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: ENOENT: no such file or directory, open '.context'

This happens because the .context file is not copied to the temporary directory created by gordon to run locally the script and

var gordon_context = JSON.parse(fs.readFileSync('.context', 'utf8'));

fails.

I modified the bot not to read the token from .context. I pass the token directly in the new TelegramBot(token) constructor and it works, both locally and on AWS.

Packaging binaries efficiently

Hi. Gordon looks great.

Sorry if this is covered somewhere else.

I'm particularly interested in approaches like those taken to run sci-kit learn on lamda or with or larger packages with sci-kit like serapis.

Will the following commands for zipping up a virtual environment be sufficient for gordon to include these packages in the deploy?

$ find "$VIRTUAL_ENV/lib64/python2.7/site-packages/" -name "*.so" | xargs strip
$ pushd "$VIRTUAL_ENV/lib64/python2.7/site-packages/"
$ zip -r -9 -q ~/venv.zip *
$ popd

Will zip -9 be used to ensure proper size?

Thank you!

Gordon apply error in 0.3.0 (works in 0.2.1)

$ gordon apply --stage=dev
Applying project...
Enter MFA code: [...enter code...]
Traceback (most recent call last):
  File "/usr/local/bin/gordon", line 9, in <module>
    load_entry_point('gordon==0.3.0', 'console_scripts', 'gordon')()
  File "/usr/local/lib/python2.7/dist-packages/gordon/bin.py", line 106, in main
    getattr(obj, options.func)()
  File "/usr/local/lib/python2.7/dist-packages/gordon/core.py", line 426, in apply
    context.update(self.collect_parameters())
  File "/usr/local/lib/python2.7/dist-packages/gordon/core.py", line 475, in collect_parameters
    context=context
  File "/usr/local/lib/python2.7/dist-packages/gordon/utils.py", line 190, in convert_cloudformation_types
    cf_data[k] = ', '.join(v)
TypeError: sequence item 0: expected string, dict found

The same project/config work fine in 0.2.1.
NOT using bucket event sources at all.
I'm not into python so just tell me what you need to debug the issue there... :)

Use existing S3 bucket for deployment

We run in an environment where S3 bucket creation is restricted (due to the limited number of allowed buckets easily being reached). Is there a way to provide gordon a specific (already existing) bucket to be used for deployment instead of having it create a bucket or buckets each time as needed?

The build.gradle file isn't useful outside of gordon

Both the gradle command line tool and IDEs should be able to work with it, but it references target which must be something include outside of the file.

$ gradle build

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/sam/Projects/gordon/nfstest/connect/build.gradle' line: 22

* What went wrong:
A problem occurred evaluating root project 'connect'.
> Could not find property 'target' on task ':buildLambda'.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 3.978 secs

is the env Isolation by AWS Accounts a pre-requisite to use Gordon?

I like the feel/simplicity of this tool (vs. Serverless which feels more complicated). However, before I invest time, I saw section entitled - Isolation between stages?:

...Putting that to the extreme, and following AWS best practices that means using different AWS accounts. This completely clash with the suggested approach that AWS suggest you to follow with services such as apigateway, where they emphasize to have several "stages" for the same apigateway resource. We disagree and completely ignore that functionality.

Gordon keeps reproducibility and isolation at it's core. When you apply gordon projects in different stages or regions, you'll deploy completely isolated Cloudformation stacks which will contain an exact and isolated copy of all the resources you have defined.

So, what if I don't have multiple AWS sub-accounts to achieve separation of envs (dev, staging, prod)? Can I effectively still use the concepts of multiple envs with Gordon but have it make use of just one AWS account?

Support for ApiGateway Integration "PassthroughBehavior" parameter

It would help alot to have the "PassthroughBehavior" of ApiGateway available in gordon config:
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apitgateway-method-integration.html#cfn-apigateway-method-integration-passthroughbehavior

Should have been recently implemented in the latest troposphere, if i'm not mistaken.
It's a bit annoying having to change it from aws panel at every deploy, and having the wrong value can lead to some serious security implications for lambdas too...

And ofc, thanks for the amazing work!

edit: after another check, it looks like the parameter has been merged into master of troposphere but not yet tagged for release, but i assume it will be within a few days.

Gordon doesn't work with temporary AWS credentials

Hi there,

gordon seems to be a good tool so far but when I try to use it I run into an issue with temporary credentials which we're using all the time. Gordon seems to try to get the AWS account id by calling boto3.client('iam').get_user()['User']['Arn'].split(':')[4]. This is fine for static credentials but doesn't work if one uses temporary ones. Gordon fails with:

me$ gordon apply

Applying project...
Traceback (most recent call last):
  File "/usr/local/bin/gordon", line 9, in <module>
    load_entry_point('gordon==0.2.0', 'console_scripts', 'gordon')()
  File "/usr/local/lib/python2.7/site-packages/gordon/bin.py", line 97, in main
    getattr(obj, options.func)()
  File "/usr/local/lib/python2.7/site-packages/gordon/core.py", line 424, in apply
    context.update(self.collect_parameters())
  File "/usr/local/lib/python2.7/site-packages/gordon/core.py", line 455, in collect_parameters
    'aws_account_id': boto3.client('iam').get_user()['User']['Arn'].split(':')[4],
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 258, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 548, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the GetUser operation: Must specify userName when calling with non-User credentials

'pip search gordon' yields versions not on this repo

Installation docs mentions using pip. But doing a pip search on gordon yields a version not mentioned on your Github:

$ pip search gordon
JoeLogTools (0.4.0)  - Joseph Gordon's decorating logging metaclass.
gordon (0.0.1)       - gordon is a simple framework for creating server-less applications using AWS services

https://github.com/jorgebastida/gordon/releases
https://github.com/jorgebastida/gordon/releases/tag/0.1
https://github.com/jorgebastida/gordon/releases/tag/0.2.0

So, it version 0.0.1 for pip pointing to tag/0.1 or tag/0.2.0? Is it always the latest or have you just forgot to update pip releases to current recent releases that have been tagged above?

Mock LambdaContext in Node loader missing context.fail()

Our context.fail() commands are bombing when we run failing node lambdas locally via gordon run. For now we're polyfilling locally to work around this. We'd like to PR a better fix when we have some time, but for now we're opening this issue.

Relevant Neck of the Woods

https://github.com/jorgebastida/gordon/blob/master/gordon/loaders/node.js#L24-L26

How to reuse AWS policy or role between lambdas and/or projects?

Hi,

What would be best way to reuse AWS policies or roles between lambdas and / or projects?

Currently we have two options:

• define policy inline under policies for each lambda
• override role for each lambda

None of the two is optimal:

• the first does not support reuse
• in second we need to include policies generated by gordon (documentation does not state which)
• also using parameter like ref://RoleArn is currently not supported (gordon throws exception)

I am looking for a solution to have a custom role attached to lambdas or to whole project.

Hmm, maybe I need just to prepare PR ;-)

Query parameters are not configured in the API Gateway

I have this in the settings.yml for an application that should only demonstrate how to read parameters from a GET query string or a POST body:

greetingsapi:
  description: Greetings
  resources:
    /hello/{name}:
      methods: GET
      integration:
        lambda: greetings.index

code.js is

exports.handler = function(event, context) {
  var data = "Hello " + event.name + "!\n";
  data += "Event\n" + JSON.stringify(event, null, 2) + "\n";
  data += "Context\n" + JSON.stringify(context, null, 2) + "\n";
  context.succeed(data);
};

echo '{"name": "Paolo"}' | gordon run greetings.index works as expected. However it seems that the events object is always empty when I call through the API gateway:

curl 'https://...eu-west-1.amazonaws.com/dev/hello/Paolo'
"Hello undefined!\nEvent\n{}\n etc"

Actually looking into the AWS API Gateway there are no parameters defined for that method:

I'm probably missing some bits of configuration. I looked into the examples and the documentation, checked several alternatives (also POST instead of GET) but the parameter never gets to the function.

Running a Python-based Lambda locally forbids non-JSON input

When running a Lambda written in Python on my local machine, only JSON input is accepted. The operative line is in gordon/loaders/python.py.

My current workaround is to enclose the input in both single and double quotes, like:

echo '"I am a string!"' | gordon run myapplication.mylambda

It'd be nice to have it interpret strings without extra quotes.

Lambda settings

Shouldn't lambda settings sit inside their own folder?
It's a pain to maintain a large setting file with every individual settings in it.

Unable to use parameter reference for memory setting

If I have a parameters file called dev.yml with:

---
stageMemory: 128

and then a settings file for an application with

---
lambdas:
  helloworld:
    code: helloworld
    handler: code.handler
    runtime: python
    memory: ref://stageMemory

Then I get

user@machine:~/code/test4/demo$ gordon build
Loading project resources
Loading installed applications
  contrib_lambdas:
    ✓ lambdas:version
  firstapp:
    ✓ lambdas:helloworld
Building project...
  0001_p.json
  0002_pr_r.json
Traceback (most recent call last):
  File "/usr/local/bin/gordon", line 9, in <module>
    load_entry_point('gordon==0.2.1', 'console_scripts', 'gordon')()
  File "/usr/local/lib/python2.7/dist-packages/gordon/bin.py", line 106, in main
    getattr(obj, options.func)()
  File "/usr/local/lib/python2.7/dist-packages/gordon/core.py", line 243, in build
    self._build_resources_template()
  File "/usr/local/lib/python2.7/dist-packages/gordon/core.py", line 334, in _build_resources_template
    r.register_resources_template(template)
  File "/usr/local/lib/python2.7/dist-packages/gordon/resources/lambdas.py", line 275, in register_resources_template
    MemorySize=self.get_memory(),
  File "/usr/local/lib/python2.7/dist-packages/gordon/resources/lambdas.py", line 68, in get_memory
    memory = int(self.settings.get('memory', 128))
TypeError: int() argument must be a string or a number, not 'Ref'

This is because int() is being called at build time but references are set as parameters at deploy time, and you can't call int() on the reference type.

The documentation here seems to indicate it works as a reference: https://gordon.readthedocs.io/en/latest/lambdas.html#memory

Either the doc needs updating if this isn't supported or the int() needs to be called at deploy time as a check.

Confusing error message when gordon.contrib.helpers is not present

It looks like contrib.helpers is missing in your settings.yml file apps list should be It looks like gordon.contrib.helpers is missing in your settings.yml file apps list.

Name of the deploy command

First of all, thanks for the great project!

However, is there a reason the name of the deploy command not being "deploy"?

Support for API Gateway IntegrationResponses ResponseParameters

Currently the only keys supported for the API Gateway under IntegrationResponses are pattern, status, and template:

gordon/gordon/resources/apigateway.py

Line 160 in 43d7bd0

if 'template' in response:

ResponseParameters (parameters) is missing: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apitgateway-method-integration-integrationresponse.html#cfn-apigateway-method-integration-integrationresponse-responseparameters

I could create a PR which adds this support, but it doesn't appear that any PRs are being merged currently. Is it worth the effort?

when using babel for es6, what's the recommended way to get node_modules into _build?

In the simple es6 example, the settings.yml

build: TARGET={target} npm run build

and the build script is as follows:

"build": "babel *.js --out-dir $TARGET"

This doesn't take into account node_modules. I've added some additional dependencies that I need to be included in node_modules in the _build directory. What's the recommended approach to do this?

Thanks!

S3 bucket cannot be ref://

When trying to do

s3:
  upload_lambda_trigger:
    bucket: ref://InputBucket

Gordon fails with

Building project...
  0001_p.json
      cp -Rf * /Users/tcosta/.gordon/tmpIxTWCd
      pip install --install-option="--prefix=" -r requirements.txt -q -t /Users/tcosta/.gordon/tmpIxTWCd
      /Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/commands/install.py:180: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
        cmdoptions.check_install_build_global(options)

      cd /Users/tcosta/.gordon/tmpIxTWCd && find . -name "*.pyc" -delete
      cp -Rf * /Users/tcosta/.gordon/tmpWuqh_1
      pip install --install-option="--prefix=" -r requirements.txt -q -t /Users/tcosta/.gordon/tmpWuqh_1
      /Library/Python/2.7/site-packages/pip-8.1.2-py2.7.egg/pip/commands/install.py:180: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
        cmdoptions.check_install_build_global(options)

      cd /Users/tcosta/.gordon/tmpWuqh_1 && find . -name "*.pyc" -delete
      cp -Rf * /Users/tcosta/.gordon/tmpViyHe0
      cp -Rf * /Users/tcosta/.gordon/tmpJovOg2
  0002_pr_r.json
Traceback (most recent call last):
  File "/usr/local/bin/gordon", line 9, in <module>
    load_entry_point('gordon==0.2.1', 'console_scripts', 'gordon')()
  File "/Library/Python/2.7/site-packages/gordon/bin.py", line 106, in main
    getattr(obj, options.func)()
  File "/Library/Python/2.7/site-packages/gordon/core.py", line 243, in build
    self._build_resources_template()
  File "/Library/Python/2.7/site-packages/gordon/core.py", line 334, in _build_resources_template
    r.register_resources_template(template)
  File "/Library/Python/2.7/site-packages/gordon/resources/s3.py", line 337, in register_resources_template
    Bucket=self.get_bucket_name(),
  File "/Library/Python/2.7/site-packages/gordon/resources/s3.py", line 274, in get_bucket_name
    if bucket.startswith('arn:aws:'):
AttributeError: 'Ref' object has no attribute 'startswith'

This is with debug mode on. The ref is defined in dev.yml as

---
InputBucket: my-test-bucket

The build succeeds if the ref is used in the context property and the ref is removed from the s3 key.

'RequestType': KeyError Traceback

I ran through the steps outlined in the Quickstart: Python. I was able to get the CloudFormation to complete, the Lambda function is created, etc.

$ gordon apply
Applying project...
  0001_p.json (cloudformation)
    CREATE_COMPLETE waiting... |                                                    
  0002_pr_r.json (custom)
    ✓ code/contrib_lambdas_version.zip (2b2c3828)
  0003_r.json (cloudformation)
    CREATE_COMPLETE waiting... |                                                    
Project Outputs:

However, when I try to test the function using the "Configure test event" (in the AWS Lambda console) with the sample "Hello world" template:

{
  "key3": "value3",
  "key2": "value2",
  "key1": "value1"
}

I get the following stack trace error:

{
  "stackTrace": [
    [
      "/var/task/version.py",
      22,
      "handler",
      "if event['RequestType'] == 'Delete':"
    ]
  ],
  "errorType": "KeyError",
  "errorMessage": "'RequestType'"
}

Same error in CloudWatch:

Traceback (most recent call last): File "/var/task/version.py", line 22, in handler if event['RequestType'] == 'Delete': KeyError: 'RequestType'

Which is a normal Python error for a key (here "RequestType") not residing in the event dictionary.

That error originates in the handler function found in the version.py file starting on line 22:

if event['RequestType'] == 'Delete':
        send(event, context, SUCCESS)
        return

I am probably missing something obvious. Just posting here to see if anyone knows what I am doing wrong. Thanks!

Requirements.txt being ignored.

I'm building a python lambda and have a structure like this:

ls

--project:
settings.yml
--app:
code.py
requirements.txt
settings.yml

running gordon build;gordon apply successfully uploads the lambda function, but it seems to have totally ignored requirements.txt, causing the lambda function to not run.
Is there a way to force it to use requirements or otherwise troubleshoot why it's skipping it?

Docker - pip support is not available at the moment

https://github.com/jorgebastida/gordon/blob/master/examples/docker/helloworld/settings.yml

Just wondering if you have a issue opened already regarding still needing pip support?

I'm interested in using this tool and having ability to build the Python lambdas in environment that is near identical to the lambda env that it runs (especially if a pip install requires some dependent machine specific install which may need machine dependent library support outside of pip - for example, the pymssql package (so lambda can read/write to SQL Server based RDS database) is good use case that depends on FreeTDS which is pre-requisite machine specific installation needed before doing pip install pymssql - http://pymssql.org/en/latest/intro.html.

Stages not working

When using --stage=??? to apply command, it seems to be ignored.

➜  realist-gordon gordon apply --stage=prod
Applying project...
  0001_p.json (cloudformation)
    ✓ No updates are to be performed.
  0002_pr_r.json (custom)
    ✓ code/contrib_lambdas_version.zip (2a16cb60)
    ✓ code/api_faqslist.zip (995f9a9f)
  0003_r.json (cloudformation)
    UPDATE_COMPLETE_CLEANUP_IN_PROGRESS waiting... |                                
Project Outputs:
  LambdaApiFaqslist
    arn:aws:lambda:us-east-1:xxxxxxxxx:function:dev-realist-gordon-r-ApiFaqslist-1S4WC563WH6W9:current
  ApigatewayReaslistapi
    https://45ry8uwz9l.execute-api.us-east-1.amazonaws.com/dev
➜  realist-gordon 

Provide support for SBT when using Scala instead of Gradle

SBT is the standard build tool for Scala projects, but understand why Gradle is used here.

As an additional option, it would be great to be able to use a different build tool.

Installing binaries for container

Example ffmpeg or any other apt-get type package

Misleading error message when the bucket name is not valid

I typed an invalid bucket name (with an upper case letter) and this is the output of gordon:

 $ gordon apply
Applying project...
  0001_p.json (cloudformation)
    UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS waiting... |                       
  0002_pr_r.json (custom)
Traceback (most recent call last):
  File "/home/me/.local/bin/gordon", line 11, in <module>
    sys.exit(main())
  File "/home/me/.local/lib/python2.7/site-packages/gordon/bin.py", line 97, in main
    getattr(obj, options.func)()
  File "/home/me/.local/lib/python2.7/site-packages/gordon/core.py", line 432, in apply
    getattr(self, 'apply_{}_template'.format(template_type))(name, filename, context)
  File "/home/me/.local/lib/python2.7/site-packages/gordon/core.py", line 476, in apply_custom_template
    outputs = template.apply(context, self)
  File "/home/me/.local/lib/python2.7/site-packages/gordon/actions.py", line 131, in apply
    action_outputs[action.name] = action.apply(context, project)
  File "/home/me/.local/lib/python2.7/site-packages/gordon/actions.py", line 217, in apply
    obj.upload_file(self.filename, ExtraArgs={'Metadata': {'sha1': file_hash}})
  File "/home/me/.local/lib/python2.7/site-packages/boto3/s3/inject.py", line 161, in object_upload_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "/home/me/.local/lib/python2.7/site-packages/boto3/s3/inject.py", line 84, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/me/.local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 641, in upload_file
    self._put_object(filename, bucket, key, callback, extra_args)
  File "/home/me/.local/lib/python2.7/site-packages/boto3/s3/transfer.py", line 651, in _put_object
    **extra_args)
  File "/home/me/.local/lib/python2.7/site-packages/botocore/client.py", line 258, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/me/.local/lib/python2.7/site-packages/botocore/client.py", line 548, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

I expected an error on the creation of the bucket. I realized what happened when I tried to manually create the bucket: the AWS console told me the name was invalid. Apparently gordon silently ignored the error on creation and failed only on the first access to the bucket.

Using reference in settings.yml policy fails

Using a reference parameter in the following manner results in Template format error: Unresolved resource dependencies [arn] in the Resources block of the template

lambdas:
  get:
    code: get
    handler: code.handler
    runtime: nodejs4.3
    policies:
      example_bucket_policy:
        Version: "2012-10-17"
        Statement:
          -
            Action:
              - "dynamodb:GetItem"
            Resource: ref://arn
            Effect: "Allow"

There references in defined in the parameter directory in a file call dev.yml

---
secret: secret
table: metaTags-dev
arn: arn:aws:dynamodb:us-east-1:123456789:table/metaTags-dev

Docker issue

I pulled the recommended Docker image down with Kitematic and it booted with the following error.
Kind of important for my workflow since I compile a lot of C code for Lambda, but feel free to close if you think this out of scope.

{"errorMessage":"Cannot find module 'index'","errorType":"Error","stackTrace":["Function.Module._load (module.js:276:25)","Module.require (module.js:353:17)","require (internal/module.js:12:17)"]}
START RequestId: 87511eb4-1ea8-15aa-9a36-d93f338f827e Version: $LATEST
Unable to import module 'index': Error
at Function.Module._resolveFilename (module.js:325:15)
at Function.Module._load (module.js:276:25)
at Module.require (module.js:353:17)
at require (internal/module.js:12:17)
END RequestId: 87511eb4-1ea8-15aa-9a36-d93f338f827e
REPORT RequestId: 87511eb4-1ea8-15aa-9a36-d93f338f827e Duration: 14.25 ms Billed Duration: 100 ms Memory Size: 1536 MB Max Memory Used: 24 MB

Missing .context when running locally

I'm using Contexts as explained here. When using gordon run I get

with open(\'.context\', \'r\') as f:\nIOError: [Errno 2] No such file or directory: \'.context\'\n'

It works fine when invoking the Lambda function on AWS.

`gordon build` hanging on "Building project..."

I recently set up a new laptop and am now having trouble running gordon build on a project that previously worked fine. Not being a python developer, I'm not sure how I should go about troubleshooting this. Can you make a suggestion? My output is as follows:

➜  gordon build --stage prod
Loading project resources
  ✓ contexts:default
  ✓ apigateway:slackdeployslashcommand
Loading installed applications
  contrib_lambdas:
    ✓ lambdas:version
  ember-deployers:
    ✓ lambdas:travis-ember-deploy
Building project...
  0001_p.json

VCS Aware: Deploy Last Commit

Given gordon the ability to detect git/mercurial and deploy files commit in the most recent commit instead of the current state of the directory. Additionally provide a warning when there are uncommitted changes that are not being deployed.

Make it possible for Lambda VPCs to be a reference

I have the following use case, which seems to be currently unsupported based on reading through the docs and source:

• One set of lambdas (one app) that essentially call each other in a chain
  • e.g. lambda_one -> lambda_two -> lambda_three
• Two environments which I need to deploy immutable copies of these lambdas to (dev + prod)
  • Each environment lives in it's own VPC (Dev-VPC/Prod-VPC) and has it's own subnets/databases

The problem here is that I can't see any way to conditionally specify a VPC per stage on a per-lambda basis. I've looked through the docs and source and it seems that vpc is just hardcoded to be grabbed from the root settings.yaml file by string match.

Ideally, I could use a ref and the config could look something like this:
settings.yaml:

...
vpcs:
  Dev-VPC:
     security-groups;
       ...
    subnet-ids:
      ...

app/settings.yml:

lambdas:
  lambda_one:
    vpc: ref://MyVPC

parameters/dev.yaml:

MyVpc: "Dev-VPC"

and then the ref would somehow auto-resolve the vpc Dev-VPV to the mapping present in settings.yaml. Looking at the code, I'm not sure how this could be done, so I'm putting this in an issue instead of a pull request. I do have an experimental patch at glaurungg/gordon@2e16877, but this requires you to re-specify the entire VPC mapping in your parameter .yaml files.

Any support here would be much appreciated.

Add Windows Support to Gordon

I've had a go with gordon since it looks like exactly what I've been looking for - something to shove lambdas into the cloud and allow me to wire them together (crucially, contract first with swagger).

I'm having a problem when I build:

> gordon build
Loading project resources
Loading installed applications
  contrib_lambdas:
    Ô£ô lambdas:version
  firstapp:
    Ô£ô lambdas:helloworld
Building project...
  0001_p.json

Error building lambda 'version'!
cmd: cd C:\Users\peter.mounce\.gordon\tmp6oo2vu && find . -name "*.pyc" -delete

Access denied - .

File not found - -NAME

File not found - -DELETE

I'm using Windows 8.1, python 2.7.10, gordon 0.2.1 (I think - I pip installed it moments ago).

I've done, in a fresh directory

gordon start project demo
cd demo
gordon startapp firstapp --runtime=js
gordon build # and got the output above
# edited /settings.yml to uncomment the firstapp array element inside the apps array
gordon build # and got the output above again

I'd expected to see the output in the quickstart.

The directory mentioned in the output does not exist.

I've tried both an admin powershell prompt and a regular one. Then a cmd for good measure; no differences.

No trigger from s3:ObjectCreated:*

Hi there! Thanks for creating this awesome tool.

I was trying to test it out by having a lambda function respond to an object being created in an existing S3 bucket test-bucket. However, adding a file to that bucket is not triggering the function, nor am I seeing any addition to the event sources for that lambda function in the AWS console. In my project settings.yml:

---
project: test
default-region: us-east-1
code-bucket: gordon-test-326a588b
apps:
  - gordon.contrib.helpers
  - gordon.contrib.lambdas
  - gordon.contrib.s3
  - scouting_locations

s3:
  scouting_s3_integration:
    bucket: arn:aws:s3:::test-bucket
    notifications:
      on_create:
        lambda: scouting_locations.helloworld
        events:
          - s3:ObjectCreated:*

I'm able to test the lambda function successfully in the AWS console, just not when the test bucket changes. Did I miss a step?

Use profile for getting aws credentials when applying

Like the aws cli, i would like to use a different profile for the aws credentials.

SES or direct support for CloudFormation

@jorgebastida I was wondering if there's currently a way to specify other CloudFormation resources (like SES). If not, is it something you'd consider in the future?

How to make use of common re-usable code with Gordon

See attached images below. I'm trying to make use of a common sibling directory, "group" for my 3 different lambda functions: post, get, get_detail. I'm just testing post now just trying to get simple example of importing of sibling directory references in Python. Looking at build of post by Gordon, it appears its not sucking in the relative references that would be needed to be part of build to make this code work in AWS Lambda. Error I get by AWS is "module initialization error: Attempted relative import in non-package". You can see in the screenshot that I unzipped the group_post.zip file and its missing the relative reference as it seems that Gordon only builds out one directory that encapsulates the code.py lambda file.

Support for CORS

Is it currently possible to set up CORS for an API deployed through gordon?

The gordon documentation is very vague when it comes to API Gateway and I wasn't able to set up the required headers on the response (namely Access-Control-Allow-Origin, Access-Control-Allow-Headers and Access-Control-Allow-Methods).

UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS is not a fatal state

Remove it from the list of fatal states and don't make it show the super scary "Oh Oh!!" message.

Cannot use double quotes in apigateway responses pattern

Hi,

I'm trying to create an apigateway using Gordon, and I want to use double quotes (") in the regex pattern used to decide what response code to return. Unfortunately, Gordon either over-escapes or under-escapes the pattern before it gives it to AWS, causing the stack creation to fail.

Here's the part of the created template that looks wrong to me:

                    "IntegrationResponses": [
                        {
                            "SelectionPattern": "",
                            "StatusCode": "201"
                        },
                        {
                            "SelectionPattern": ".*\"status\":400.*",
                            "StatusCode": "400"
                        },
                        {
                            "SelectionPattern": "*.\\\"errorMessage\\\".*",
                            "StatusCode": "500"
                        }
                    ],

You can see the full template if you want to, and the relevant settings.yml. I've also created a repo which replicates the problem at https://github.com/jakul/gordon-regex-error.

The Amazon docs suggest that the correct format for regexes should be:

"SelectionPattern": ".*\\"status\\":400.*",
 "SelectionPattern": "*.\\"errorMessage\\".*",

Can you take a look @jorgebastida ?

Cheers,
Craig

Lambda functions nonexistent after deletion and re-apply

I ran gordon apply to publish my lambdas, which worked fine. Then, I deleted those Lambdas manually. Now, when I attempt to run gorgon apply, it says that there is nothing to change, when really those Lambdas are not there any more.

Perhaps I am missing something but it seems like this should republish.

S3 Notifications and Parameter References

When defining S3 Events notifications in the project settings.yml Can parameters be used? E.g.:

s3:
  object_created_events:
    bucket: my-bucket
    notifications:
      my_lambda_consumer:
        lambda: s3consumer.firstapp
        events:
          - s3:ObjectCreated:*
        key_filters:
          prefix: ref://MyS3Prefix

When I run Gordon locally via cat s3event.json | dordon run s3consumer.firstapp It fails with AttributeError: 'Ref' object has no attribute 'startswith'.

Was Gordon ever designed to do this?

Or how do I ensure that the proper S3 prefix sends events to the correct lambda stage? When I run gordon apply will gordon create all S3 event notifications or will different ones be created per stage?

Following quickstart gave me error

Hi there,
I'm following quickstart (http://gordon.readthedocs.io/en/latest/quickstart_py.html) and at the end when I do gordon apply, I get this:

Applying project...
  0001_p.json (cloudformation)
    ✓ No updates are to be performed.
  0002_pr_r.json (custom)
Traceback (most recent call last):
  File "/usr/local/bin/gordon", line 9, in <module>
    load_entry_point('gordon==0.2.0', 'console_scripts', 'gordon')()
  File "/usr/local/lib/python3.4/dist-packages/gordon/bin.py", line 97, in main
    getattr(obj, options.func)()
  File "/usr/local/lib/python3.4/dist-packages/gordon/core.py", line 432, in apply
    getattr(self, 'apply_{}_template'.format(template_type))(name, filename, context)
  File "/usr/local/lib/python3.4/dist-packages/gordon/core.py", line 476, in apply_custom_template
    outputs = template.apply(context, self)
  File "/usr/local/lib/python3.4/dist-packages/gordon/actions.py", line 131, in apply
    action_outputs[action.name] = action.apply(context, project)
  File "/usr/local/lib/python3.4/dist-packages/gordon/actions.py", line 198, in apply
    file_hash = utils.get_file_hash(self.filename)
  File "/usr/local/lib/python3.4/dist-packages/gordon/utils.py", line 77, in get_file_hash
    return get_zip_hash(filename)
  File "/usr/local/lib/python3.4/dist-packages/gordon/utils.py", line 70, in get_zip_hash
    digest.update(path)
TypeError: Unicode-objects must be encoded before hashing

Thanks