docker-airflow
This repository contains Dockerfile of airflow for Docker's automated build published to the public Docker Hub Registry.
Informations
- Based on Debian Jessie official Image debian:jessie and uses the official Postgres as backend and RabbitMQ as queue
- Install Docker
- Install Docker Compose
- Following the Airflow release from Python Package Index
Installation
Pull the image from the Docker repository.
docker pull puckel/docker-airflow
Build
For example, if you need to install Extra Packages, edit the Dockerfile and than build-it.
docker build --rm -t puckel/docker-airflow .
Usage
Start the stack (postgres, rabbitmq, airflow-webserver, airflow-scheduler airflow-flower & airflow-worker):
docker-compose up -d
If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "mysql_default" set this values (equivalent to values in airflow.cfg/docker-compose.yml) :
- Host : mysql
- Schema : airflow
- Login : airflow
- Password : airflow
Check Airflow Documentation
UI Links
- Airflow: localhost:8080
- Flower: localhost:5555
- RabbitMQ: localhost:15672
When using OSX with boot2docker, use: open http://$(boot2docker ip):8080
Run the test "tutorial"
docker exec dockerairflow_webserver_1 airflow backfill tutorial -s 2015-05-01 -e 2015-06-01
Scale the number of workers
Easy scaling using docker-compose:
docker-compose scale worker=5
This can be used to scale to a multi node setup using docker swarm.
Wanna help?
Fork, improve and PR. ;-)