pranavpawar3/airflow-tutorial

Apache Airflow tutorial

NOTE: The Docker system is inspired from this repo, though we have customized the dockerfile as well as its config for our custom use case.

Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube

Contents

Part	Title	Git Tag
1	Introduction to Apache Airflow (blog post)	v0.1
2	Set up airflow environment with docker (blog post)	v0.2
3	Set up airflow environment using Google Cloud Composer (blog post)	N/A
4	Writing your first pipeline (blog post)	N/A
5	Airflow concept (blog post)	N/A
6	Build a data pipeline using Google Cloud Bigquery (blog post)	v0.6
7	Airflow variables (blog post)	v0.7

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

• Clone this repo
• Install the prerequisites
• Run the service
• Check http://localhost:8080
• Done! 🎉

Prerequisites

• Install Docker
• Install Docker Compose
• Following the Airflow release from Python Package Index

Usage

Run the web service with docker

docker-compose up -d

# Build the image
# docker-compose up -d --build

Check http://localhost:8080/

• docker-compose logs - Displays log output
• docker-compose ps - List containers
• docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

• docker-compose run --rm webserver airflow list_dags - List dags
• docker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task

If you want to run/test python script, you can do so like this:

• docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:

• Host : postgres
• Schema : airflow
• Login : airflow
• Password : airflow

Credits

• Apache Airflow
• docker-airflow