bhargavaganti/apache-airflow

Apache Airflow Scripts

Google (GCP) Cloud Composer (Apache Airflow)

This repo contains helpful Scripts, Configs, and Commands


Create Cloud Composer (Apache Airflow) Environment

gcloud composer environments create z-cloud-composer1 \
  --location us-east1 \
  --zone us-east1-b \
  --machine-type n1-standard-1 \
  --env-variables=gcp_project=project123,gcs_bucket=gs://mybucket,gce_zone=us-east1-b

Delete Cloud Composer (Apache Airflow) Environment

gcloud composer environments delete z-cloud-composer1 

--location us-east1

Upload DAG from source (source could be local, gcs path, url, etc. It could also be a path to directory with multiple DAGs.

gcloud composer environments storage dags import 

--environment z-cloud-composer1 

--location us-east1 

--source /tmp/airflow_dags/airflow_demo1.py

Remove/Delete DAG from Cloud Composer

gcloud composer environments storage dags delete 

--environment z-cloud-composer1 

--location us-east1 

my_dag_name

References:
    • Google (GCP) Cloud Composer
    • Apache Airflow Github
    • Apache Airflow Docs