marcosmarxm/airflow-testing-ci-workflow

(project & tutorial) dag pipeline tests + ci/cd setup

Airflow DAG development with tests + CI workflow

This code is complementar to the article How to develop data pipeline in Airflow through TDD (test-driven development). I suggest you to read to better understand the code and the way I think how to setup the project.

Step-by-step: How to develop a DAG using TDD (english version)
Passo-a-passo: Como desenvolver uma DAG usando TDD (portuguese version)

The project

Below is a summary of what will be accomplished in this project. We'll simulate the transfer of some fake transaction data from an ecommerce. A simple task transfering data from the otlp-db database to the olap-db database.

To help in the development we use a local development environment to build the pipeline with tests and also a Continuous Integration pipeline with Github Action to ensure that tests are applied at each change.

Containers

• airflow: container running local setup for development;
• oltp-db and olap-db: container that simulate database in a production environment and receive fake data;

In this tutorial we won't developt the dashboard part only the pipeline.

Dependencies?

Docker, docker-compose and makefile.

How to run?

The command below will setup the environment using docker-compose. Wait a few minutes (240s, yeah omg right?) to Airflow initialize its internal configuration, then the command will create credentials and connections.

make setup

By running the above command it is possible to access Airflow at localhost: 8080. A user of test user: admin / password: admin is created. At this stage you can develop your DAGs and test them as you modify them. And finally, the command that calls the pytest to perform tests.

make testing

Some resources about Airflow testing and DataOps:

• Pipelines on pipelines: Agile CI/CD workflows for Airflow DAGs @ Airflow Summit 2020
• Data Testing with Airflow
• Data's Inferno: 7 Circles of Data Testing Hell with Airflow
• Testing and Debugging in Apache Airflow by GoDataDriven
• The Challenge of Testing Data Pipelines
• Automated Testing for Proceting Data Pipeliens from Undocumented Assumptions
• Why Great Data Engineering Needs Automated Testing
• Testing in Airflow Part 1 - DAG validation tests, DAG definition tests and unit tests
• Testing in Airflow Part 2 - Integration Tests and e2e Pipeline Tests

airflow-testing-ci-workflow's Issues

Test failing when it should pass?

Just a note in case this happens for someone else: when I followed the tutorial and made the change which should make the first test to pass for the first time, I was still getting FAILED as a result.

Add resource +

• Add this repository to resource: https://github.com/ninja-van/airflow-boilerplate
• Add black formatting and .pre-commit.yaml to maintain good practices

Grammar correction in english

apply PR #26 in English too.