scout24/pybuilder_aws_plugin

PyBuilder plugin to handle packaging and uploading Python AWS Lambda code.

pybuilder_aws_plugin's Issues

Move plugin to pybuilder org

Can we move this to the pybuilder org and maintain it from there? Would make it easier for external people to merge pull-requests.

cc @mriehl @snordhausen

exclude boto3 from dependencies

Since it is already installed in the AWS-Lambda execution context.

Exclude *.pyc and *.pyo files from packaging

I think we should not deliver *.pyc and *.pyo files within the zip by default to prevent incompatibility issues with the version of the runtime python in regards to the packaging python. If desired (eg for performance concerns) we can add a switch do allow delivery.

more freedom in directory structure

It would be nice to have a little more freedom in the directory structure. For instance the "scripts" directory is mandatory, even though I could directly call the module in AWS LAMDA.

[INFO] Going to assemble the lambda-zip.
BUILD FAILED - [Errno 2] No such file or directory: 'XXX/lambda-manage-elasticsearch/src/main/scripts'

write S3 keyname to file

As a developer setting up CI/CLD I need to know the exact bucket key name that is genearted during upload. Suggest to write this to a file in target or target/reports.

Cant install or use

Hello,

I am new to the python world so please bare with me. I found your plugin and its exactly what i am looking for. I have another POC python AWS Lambda and the build is a mess. I was trying to adopt PyBuilder and Your plugin to bring it to the next level

When i try to add

use_plugin('pypi:pybuilder_aws_plugin')

i get the following error

←[1mPyBuilder version 0.11.8
←[0;0mBuild started at 2016-07-22 11:16:25
------------------------------------------------------------
←[1;33m[WARN] ←[0;0m Missing plugin pypi:pybuilder_aws_plugin
←[1m[INFO] ←[0;0m Downloading plugin pypi:pybuilder_aws_plugin
------------------------------------------------------------
←[1;31mBUILD FAILED - [Errno 13] Permission denied: 'c:\\users\\jcorson\\appdata\\local\\temp\\tmpoak5pe'
←[0;0m------------------------------------------------------------
Build finished at 2016-07-22 11:16:26
Build took 0 seconds (72 ms)
(venv)

Each attempt is a different temp folder. If i did a pip install:

pip install pybuilder_aws_plugin

i get an error about pyymal

...
running build_ext
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
checking if libyaml is compilable
error: [Error 2] The system cannot find the file specified

----------------------------------------
Failed building wheel for pyyaml
....

I also tried to install pyyaml directly and also have a issue.

Is there anything i can do to get your plugin up and running?

documentation

• explain how the plugin works
• document what can be configured
• any limitations?

We may want to merge in pybuilder_aws_upload_json_plugin

As we share some code and logics with this plugin it might me desirable to merge the two and let this plugin reside as pybuilder_aws_plugin.

Missing Task Dependency

Hi,

Trying to use this plugin and failing both on windows and linux with the same error

Traceback (most recent call last):
  File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/cli.py", line 405, in main
    exclude_all_optional=options.exclude_all_optional
  File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/reactor.py", line 117, in prepare_build
    self.execution_manager.resolve_dependencies(exclude_optional_tasks, exclude_tasks, exclude_all_optional)
  File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/execution.py", line 470, in resolve_dependencies
    raise MissingTaskDependencyException(task.name, d)
MissingTaskDependencyException: Missing task 'install_build_dependencies' required for task 'package_lambda_code'
------------------------------------------------------------
BUILD FAILED - Missing task 'install_build_dependencies' required for task 'package_lambda_code'
------------------------------------------------------------
Build finished at 2017-03-23 11:30:49
Build took 0 seconds (49 ms)

build.py contains

from pybuilder.core import use_plugin

use_plugin('pypi:pybuilder_aws_plugin')

default_task = "package_lambda_code"

Any idea how to resolve?

Thanks

Why is boto/boto3 excluded from package_lambda_code?

I'm trying to package a pybuilder project for lambda and saw the following in the codebase: https://github.com/ImmobilienScout24/pybuilder_aws_plugin/blob/master/src/main/python/pybuilder_aws_plugin/lambda_tasks.py#L85

Is there a specific reason why boto/boto3 are being excluded from the lambda dependencies?

create project folder in S3 bucket

We need to use one S3 bucket for different LAMDA functions. Currently the plugin only creates a flat hierarchy of version folders.

URL dependencies are not included in lambda dist

When adding and URL dependency to project as runtime dependency, e.g.
in build.py init phase: project.depends_on(modulename, url="file://" + .tar.gz)
so it is expected to be installed to target/lambda_dependencies dir, but this won't work.

It seems that in lambda_tasks.prepare_dependencies_dir the dependencies
are fetch by build_install_dependencies_string(project).
and the build_install_dependencies_string(project) excludes URL dependencies.

How this should be resolved, either to workaround build_install_dependencies_string in lambda_tasks
or modify the actual build_install_dependencies_string?

We resolved this by adding additional function to our build which installs all URL dependencies:

def include_url_dependencies(project, logger, target_directory):
    """ Installs url dependencies to target_directory.  """
    dependencies = [
        dependency for dependency in project.dependencies if dependency.url]
    if not dependencies:
        return ""

    pip_cmd = 'pip install --target {0} {1}'
    for dependency in dependencies:

        cmd = pip_cmd.format(target_directory, dependency.url)
        logger.info("Installing url-dependency {0}: '{1}'".format(dependency.url, cmd))

        process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
        process.communicate()
        if process.returncode != 0:
            msg = "Command '{0}' failed to install dependency: {1}".format(
                    cmd, process.returncode)
            raise Exception(msg)

Allow the plugin to use a different directory than src/main/python when creating the package

Currently, the package_lambda_code task takes the files from src/main/{python,scripts} which means that using the filter_resources plugin will not affect files in the produced zip file.

Please add a way to include files that have been filtered.

I have a simple backwards-compatibile solution that I've proposed in a PR.

Include metadata in lambda-zip

This would be nice for provenance: just put a metadata.yaml file in the zip. It would contain, project version, name, summary author, build-date, git-hash of the plugin, etc...

write some tests

some tests would be nice