abhishekbhakat/turbine

Simple python script for easy local airflow deployment with docker. Packed with additional components. Will be adding more going forward.

Turbine

Simple python script for local airflow deployment with docker. Added components are

• elasticsearch logging
• smtp server for email alerts
• hashicorp vault for secrets backend
• vscode server for ide
• marquez for lineage
• minio for S3 bucket store
• oittaa/gcp-storage-emulator for GCS
• azurite for blob storage

Will be adding more going forward.

NOTE: Not for Windows! But can work in WSL liunx.

NOTE: Dev branch is for aarch64 architecture.

NOTE: Code is commented for Astro.

NOTE: Turbine astro needs Astro-CLI installed.

Usage

python Turbine.py

Sample output

Creating new cache__φ(。。)
Creating new farm__φ(。。)
New farm created using network as 172.27.0.1/16! (￣▽￣)ノ
Airflow type:
 1. Astro [default]
 2. OSS 
 3. OSS Main branch
-> 1
Project name: abcd
Enable remote logging [yN]: 
Enable vault [yN]: 
Enable code server [yN]: 
Using port 8080 for webserver, 5555 for flower, 7000 for IDE and 0 for redis
Using network: 172.27.2.1
Initializing Astro project
Pulling Airflow development files from Astro Runtime 9.1.0
Initialized empty Astro project in /Users/abhishekbhakat/Codes/Turbine/farm/abcd-astro-airflow
Cache updated! (￣▽￣)ノ

Start Airflow

Change directory inside the folder created. And run start.sh.

Sample output

Deploying...
Cleaning older deployment...
Building image...
Preping db...
Deployed:
Airflow: http://localhost:8082
Airflow Swagger: http://localhost:8082/api/v1/ui/
Flower: http://localhost:5557
Vault: http://localhost:8200
Opensearch: http://localhost:5601/app/home#/
Marquez: http://localhost:3000/

Logs for deployment are collected in start.log.

Vault

For vault to work, we have to manually create a keys to unseal the vault and root_token will be auto generated. After that, the root_token needs to be updated on the Dockerfile with AIRFLOW__SECRETS__BACKEND_KWARGS.

Sample command to add secrets from vault shell

vault secrets enable -path=airflow/ kv-v2
vault kv put airflow/variables/my_var value=hello

turbine's Issues

only lower case names are working as of now