deji01/airflow-sneakers

Deploy Apache Airflow workflow with Helm Chart

Airflow-ETL

Deploy Apache Airflow workflow with Helm Chart.

Requirements

• Kubectl
• Helm
• Kind
• Docker

Kind Installation

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.15.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

Setup

# Create a kubernetes cluster of 1 control plane and 3 worker nodes
kind create cluster --name airflow-cluster --config kind-cluster.yaml

# Check the cluster info
kubectl cluster-info

# Check the nodes with kubectl
kubectl get nodes -o wide

# Add the official repository of the Airflow Helm Chart
helm repo add apache-airflow https://airflow.apache.org

# Update the repo
helm repo update

# Create namespace airflow
kubectl create namespace airflow

# Check the namespace 
kubectl get namespaces

# Install the Airflow Helm Chart
helm install airflow apache-airflow/airflow --namespace airflow --debug

# Get pods
kubectl get pods -n airflow

# Check release
helm ls -n airflow

# If for some reasons the release is stuck in pending-install or timed out
# Resinstall the chart
helm delete airflow --namespace airflow
helm install airflow apache-airflow/airflow --namespace airflow --debug —timeout 10m0s

# Port forward 8080:8080
kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow --context kind-airflow-cluster

Debug

# 1. Delete the Helm release
helm delete airflow --namespace airflow

# 2. Check your PODs
kubectl get pods -n airflow

# 3. If airflow-migrations is in ContainerCreating state delete it
kubectl get jobs -n airflow
kubectl delete jobs <pods_name_of_airflow_migrations> -n airflow

# 4. Install the chart again
helm install airflow apache-airflow/airflow --namespace airflow --debug --timeout 10m0s

Get Values

Get the configuration file of the Helm chart make changes to Airflow instance in deployment.

# Get the Chart values
helm show values apache-airflow/airflow > values.yaml

Update Airflow deployment

# Check the current revision
helm ls -n airflow

# Upgrade the chart
helm upgrade --install airflow apache-airflow/airflow -n airflow -f values.yaml --debug

# Check after
helm ls -n airflow

Configure airflow instance

in the values.yaml file add

extraEnv: |
	- name: AIRFLOW__CORE__LOAD_EXAMPLES
		value: 'True'

Clear Everything

kind delete cluster --name airflow-cluster

Load Docker image on Kubernetes cluster

kind load docker-image <image name> --name <cluster name>

Create secret key in Kubernetes cluster

kubectl create secret generic <file name> --from-file=gitSshKey=<private ssh key location> -n airflow