scout24 / pybuilder_aws_plugin Goto Github PK
View Code? Open in Web Editor NEWPyBuilder plugin to handle packaging and uploading Python AWS Lambda code.
License: Apache License 2.0
PyBuilder plugin to handle packaging and uploading Python AWS Lambda code.
License: Apache License 2.0
Hello,
I am new to the python world so please bare with me. I found your plugin and its exactly what i am looking for. I have another POC python AWS Lambda and the build is a mess. I was trying to adopt PyBuilder and Your plugin to bring it to the next level
When i try to add
use_plugin('pypi:pybuilder_aws_plugin')
i get the following error
←[1mPyBuilder version 0.11.8
←[0;0mBuild started at 2016-07-22 11:16:25
------------------------------------------------------------
←[1;33m[WARN] ←[0;0m Missing plugin pypi:pybuilder_aws_plugin
←[1m[INFO] ←[0;0m Downloading plugin pypi:pybuilder_aws_plugin
------------------------------------------------------------
←[1;31mBUILD FAILED - [Errno 13] Permission denied: 'c:\\users\\jcorson\\appdata\\local\\temp\\tmpoak5pe'
←[0;0m------------------------------------------------------------
Build finished at 2016-07-22 11:16:26
Build took 0 seconds (72 ms)
(venv)
Each attempt is a different temp folder. If i did a pip install:
pip install pybuilder_aws_plugin
i get an error about pyymal
...
running build_ext
creating build\temp.win32-2.7
creating build\temp.win32-2.7\Release
checking if libyaml is compilable
error: [Error 2] The system cannot find the file specified
----------------------------------------
Failed building wheel for pyyaml
....
I also tried to install pyyaml directly and also have a issue.
Is there anything i can do to get your plugin up and running?
some tests would be nice
Currently, the package_lambda_code
task takes the files from src/main/{python,scripts}
which means that using the filter_resources
plugin will not affect files in the produced zip file.
Please add a way to include files that have been filtered.
I have a simple backwards-compatibile solution that I've proposed in a PR.
I'm trying to package a pybuilder project for lambda and saw the following in the codebase: https://github.com/ImmobilienScout24/pybuilder_aws_plugin/blob/master/src/main/python/pybuilder_aws_plugin/lambda_tasks.py#L85
Is there a specific reason why boto/boto3 are being excluded from the lambda dependencies?
Since it is already installed in the AWS-Lambda execution context.
When adding and URL dependency to project as runtime dependency, e.g.
in build.py init phase: project.depends_on(modulename, url="file://" + .tar.gz)
so it is expected to be installed to target/lambda_dependencies dir, but this won't work.
It seems that in lambda_tasks.prepare_dependencies_dir the dependencies
are fetch by build_install_dependencies_string(project).
and the build_install_dependencies_string(project) excludes URL dependencies.
How this should be resolved, either to workaround build_install_dependencies_string in lambda_tasks
or modify the actual build_install_dependencies_string?
We resolved this by adding additional function to our build which installs all URL dependencies:
def include_url_dependencies(project, logger, target_directory):
""" Installs url dependencies to target_directory. """
dependencies = [
dependency for dependency in project.dependencies if dependency.url]
if not dependencies:
return ""
pip_cmd = 'pip install --target {0} {1}'
for dependency in dependencies:
cmd = pip_cmd.format(target_directory, dependency.url)
logger.info("Installing url-dependency {0}: '{1}'".format(dependency.url, cmd))
process = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
process.communicate()
if process.returncode != 0:
msg = "Command '{0}' failed to install dependency: {1}".format(
cmd, process.returncode)
raise Exception(msg)
It would be nice to have a little more freedom in the directory structure. For instance the "scripts" directory is mandatory, even though I could directly call the module in AWS LAMDA.
[INFO] Going to assemble the lambda-zip.
BUILD FAILED - [Errno 2] No such file or directory: 'XXX/lambda-manage-elasticsearch/src/main/scripts'
I think we should not deliver *.pyc and *.pyo files within the zip by default to prevent incompatibility issues with the version of the runtime python in regards to the packaging python. If desired (eg for performance concerns) we can add a switch do allow delivery.
We need to use one S3 bucket for different LAMDA functions. Currently the plugin only creates a flat hierarchy of version folders.
As we share some code and logics with this plugin it might me desirable to merge the two and let this plugin reside as pybuilder_aws_plugin.
Hi,
Trying to use this plugin and failing both on windows and linux with the same error
Traceback (most recent call last):
File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/cli.py", line 405, in main
exclude_all_optional=options.exclude_all_optional
File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/reactor.py", line 117, in prepare_build
self.execution_manager.resolve_dependencies(exclude_optional_tasks, exclude_tasks, exclude_all_optional)
File "/home/ec2-user/tmp/venv/lib/python2.7/site-packages/pybuilder/execution.py", line 470, in resolve_dependencies
raise MissingTaskDependencyException(task.name, d)
MissingTaskDependencyException: Missing task 'install_build_dependencies' required for task 'package_lambda_code'
------------------------------------------------------------
BUILD FAILED - Missing task 'install_build_dependencies' required for task 'package_lambda_code'
------------------------------------------------------------
Build finished at 2017-03-23 11:30:49
Build took 0 seconds (49 ms)
build.py contains
from pybuilder.core import use_plugin
use_plugin('pypi:pybuilder_aws_plugin')
default_task = "package_lambda_code"
Any idea how to resolve?
Thanks
As a developer setting up CI/CLD I need to know the exact bucket key name that is genearted during upload. Suggest to write this to a file in target
or target/reports
.
Can we move this to the pybuilder org and maintain it from there? Would make it easier for external people to merge pull-requests.
This would be nice for provenance: just put a metadata.yaml file in the zip. It would contain, project version, name, summary author, build-date, git-hash of the plugin, etc...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.