Comments (3)
Hello @lennynyktyk I think you are experiencing few different issues.
- Because gordon applies some validations,
prefix
can't be a reference at the moment, only a string. In future versions we'll probably remove most of these validations on build time and add a validation entry point which can do static analysis of the template to find potential errors (like this one). - When running
$ gordon run xxx
, it fails to you because of prefix been a reference (which is not possible at the moment) - It is a current limitation that you can't have different prefixes for the same notification in different stages.
In order to solve 3 I would need to know a bit more about your use case... Is this because you have only one bucket for both stages... and you don't want to consume "production" files with the test lambda?
from gordon.
I was hoping to import the application logs that Elastic Beanstalk publishes to S3 every hour into Redshift with a Lambda function. EB stores the logs in a s3://elasticbeanstalk-<region>-<aws_account_id>/resources/environments/logs/publish/<eb_environment_id>/
. This causes all logs from different stages/environments of an EB application to be stored in the same bucket.
I can have the lambda filter based on the key prefix stored in the context gordon generates. Just seemed like this may be an opportunity to make the lambda less complex as gordon seems to try to do this in other areas.
from gordon.
This is an interesting use-case indeed... but I think you are going to hit (yet another limitation, in this case because of aws).
The api to wire together a S3 bucket and a lambda makes impossible to define individual sources, and forces you to define all of them every time you change them. (For more information read this https://github.com/jorgebastida/gordon/blob/master/gordon/contrib/s3/bucket_notification_configuration/bucket_notification_configuration.py#L13-L37)
Because of this limitation, you'll not be able to use stages, because those are deployed into 100% isolated stacks, so each of your stages would try to override the configuration of the same bucket on each deploy.
The only solutions which cross my mind are:
- Use gordon to deploy the lambda, but configure the s3 integration manually using the web-ui (If you don't need extreme reproducibility, this option is probably fine)
- Create one lambda, one s3 integration and deploy it to one stage and do two different things based on the s3 key where the event comes from. (Not super elegant, but would work)
- Create one function which does the Redshift push, and create two lambdas parametrized with your Beanstalk stage. Create two s3 integrations, one for each prefix. (This would work, has reproducibility, but... sadly you can't use stages)
Example of the third solution (python):
def push_to_redshift(event, context, stage):
# do some stuff
def push_to_production_redshift(event, context):
return push_to_redshift(event, context, 'production')
def push_to_staging_redshift(event, context):
return push_to_redshift(event, context, 'staging')
An in your main settings
s3:
object_created_events:
bucket: my-bucket
notifications:
my_production_lambda_consumer:
lambda: s3consumer.push_to_production_redshift
events:
- s3:ObjectCreated:*
key_filters:
prefix: /resources/environments/logs/publish/production/
my_staging_lambda_consumer:
lambda: s3consumer.push_to_staging_redshift
events:
- s3:ObjectCreated:*
key_filters:
prefix: /resources/environments/logs/publish/staging/
Hope this helps!
from gordon.
Related Issues (20)
- Option to fix lambda name in settings.yml
- Add support for configuring DLQ HOT 1
- Issues with nodejs6.10 HOT 7
- question about gordon build
- iam:GetRole still needed
- 'gordon' with no args raises error
- Add python3.6 runtime HOT 5
- Step Functions
- Typo? "context_destinaton"
- Share build settings between lambdas?
- requires troposphere <2.0 HOT 1
- S3 bucket name collision conflict when used across multiple accounts
- gordon appears to ignore npm failures HOT 1
- Can't integrate my kinesis Stream
- version 0.7.0 not published for python3? HOT 3
- python 3 string issue
- EC2 Throttling Issue
- unable to use gordon in aws china
- API Gateway: Headers integration lambda parameters was lost on deply
- wrong project
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gordon.