airflow-plugins/example-airflow-dags

Example DAGs using hooks and operators from Airflow Plugins

Example DAGs

This repository contains example DAGs that can be used "out-of-the-box" using operators found in the Airflow Plugins organization. These DAGs have a range of use cases and vary from moving data (see ETL) to background system automation that can give your Airflow "super-powers".

Getting Started

The example DAGs found here can be split into three main categories:

ETL

These DAGs focus on pulling data from various systems and putting them into Amazon Redshift, with S3 as a staging store. These represent the simplest implementation of an "ETL" workflow and can either be used "out-of-the-box" or extended to add additional custom logic.

PoC (Proof of Concept)

These DAGs demonstrate simple implementations of custom operators and Airflow setups. They are typically not "copy-and-paste" DAGs but rather walk through how something would work.

System

These DAGs are used on the system administration level and can be thought of as "meta-DAGs" that maintain various states and configurations within Airflow itself. In some cases, these DAGs are used in concert with other custom operators, such as the rate_limit_reset DAG.

Contributions

Contributions of your own DAGs are very welcome. Please see some of the example DAGs for a sense of general formatting guidelines.

License

Apache 2.0

example-airflow-dags's Issues

Issue with hubspot plugin reference.

Example-Airflow-DAGs/etl/hubspot_to_redshift.py

Line 71 in 3cf6b9b

from HubspotPlugin.schemas import hubspot_schema

Shouldn't this be importing from hubspot_plugin, given the name given to the hubspot plug in here (https://github.com/airflow-plugins/hubspot_plugin/blob/7b6e23a0493ac9c32fbd8c3265ea895a12c07ec5/__init__.py#L7)

Can't seem to get this dag to be recognized in airflow, I get no errors upon running the .py file, but it doesn't get listed in the dag list.