stijnzanders/limber

Serverless DAGs orchestrated via Python

Create a file called limber.yaml with the following configuration:

cloud:
    provider: google
    project: "<project name>"
    region: "europe-west1"
    key_file: "<path to key_file>"
    default_bucket: "<gcp cloud storage bucket name>"

In the main directory run:

1. python -m limber init
2. python -m limber terraform login
3. python -m limber plan
4. python -m limber apply

Google Cloud APIs enabled

1. CloudFunctions API
2. PubSub API
3. CloudScheduler API
4. Cloud Build API
5. Secret manager API

Other

1. Create app engine in your project: https://console.cloud.google.com/appengine

Service account needed roles:

1. Pub/Sub Editor
2. Service Account User
3. Cloud Scheduler Admin
4. Cloud Functions Developer
5. Storage Admin
6. Secret Manager Admin

limber's Issues

Add terraform remote backend functionality

1. Create a .tf file with the remote backend info
2. Fix issue with key file not being able to be read

Remvove hardcoding of project-id

Each cloud function publishes to a topic. At the moment the project id of the topic is hardcoded.

Write successes and failures of DAGs and Tasks to Slack

Upload the whole plugin folder to Google Cloud functions and centralize requirements

Auto version updating

Automatically update the package version based on PR merges in master and production

Each cloud function gets updated at deployment

Only the ones changing should be updated.

Issue is because the hash is attached to the end of the file name like #1d19745ab496efc96a5e00c4ac7dbc2e, this hash is not stored in GCP cloud, hence an update is always required.