airflow-gcp-examples's Introduction

airflow-gcp-examples

Repository with examples and smoke tests for the GCP Airflow operators and hooks.

Setup

This Google Cloud Examples does assume you will have a standard Airflow setup up and running. This tutorial does work perfectly locally as in a production setup because the only requirement is that you have a service key, that we'll explain next. But first a quick rundown of what you need:

Running Airflow (as of this writing you need Airflow master branch!!!)
Create a service account (Cloud Console)
Setup a Google Cloud Connection in Airflow
Setup variables that the DAG's will need
Copy the DAG's to your dags folder

Airflow setup

Checkout master of Airflow
pip install google-api-python-client
python setup.py install

Make sure you're running the LocalExecutor and have a decent database setup.

Google Cloud Service Key

Next create a service account where you want your smoke tests and examples to run in. Go to the console:

And create a service key. Provision a JSON private key and give it Editor's rights

Airflow Connection

In Airflow you need to define the gcp_smoke named connection to your project:

Supply the path to the downloaded private key, supply the project_id and define the minimum scope of https://www.googleapis.com/auth/cloud-platform

Airflow Variables

You need to setup variables that are used in the examples. You can tweak them to suite your environment.

variable	example value	note
gc_project	my-project	Project where the examples will run in
gcq_dataset	airflow	BigQuery dataset for examples
gcq_tempset	airflow_temp	BiqQuery dataset with 1 day retentions
gcs_bucket	airflow-gcp-smoke	Storage bucket
gcs_root	data	Storage root path (required, no start and end with slash)

Recommend Projects