firstnapat/education-sandbox

Codespace with Airflow and the Astro CLI

Education Sandbox

This repository is meant to help users to do the activities of the Astronomer Academy

Set up

Option 1: Use GitHub Codespaces

Run this Airflow project without installing anything locally.

1. Fork this repository.
2. Create a new GitHub codespaces project on your fork. Make sure it uses at least 4 cores!
3. After creating the codespaces project the Astro CLI will automatically start up all necessary Airflow components. This can take a few minutes.
4. Once the Airflow project has started, access the Airflow UI by clicking on the Ports tab and opening the forward URL for port 8080.

With 4 cores you get 30hrs free/month

Option 2: Use the Astro CLI

Download the Astro CLI to run Airflow locally in Docker. astro is the only package you will need to install.

1. Run git clone https://github.com/astronomer/2-6-example-dags.git on your computer to create a local clone of this repository.
2. Install the Astro CLI by following the steps in the Astro CLI documentation. Docker Desktop/Docker Engine is a prerequisite, but you don't need in-depth Docker knowledge to run Airflow with the Astro CLI.
3. Run astro dev start in your cloned repository.
4. After your Astro project has started. View the Airflow UI at localhost:8080.

Useful links:

• Academy.
• Docs.

Project Structure

This repository contains the following files and folders:

• .astro: files necessary for Astro CLI commands.
• .devcontainer: the GH codespaces configuration.
• dags: all DAGs in your Airflow environment. Files in this folder will be parsed by the Airflow scheduler when looking for DAGs to add to your environment. You can add your own dagfiles in this folder.
• include: supporting files that will be included in the Airflow environment.
• plugins: folder to place Airflow plugins. Empty.
• tests: folder to place pytests running on DAGs in the Airflow instance. Contains default tests.
• .dockerignore: list of files to ignore for Docker.
• .gitignore: list of files to ignore for git. NOTE that airflow_settings.yaml is not ignored in this project.
• airflow_settings.yaml: contains the connection to DuckDB used in the project.
• Dockerfile: the Dockerfile using the Astro CLI.
• packages.txt: system-level packages to be installed in the Airflow environment upon building of the Dockerimage.
• README.md: this Readme.
• requirements.txt: python packages to be installed to be used by DAGs upon building of the Dockerimage.