rajeshdasi/airflow-chart-astronomer

A Helm chart to install Apache Airflow on Kubernetes

Astronomer's Helm Chart for Apache Airflow

This chart will bootstrap an Airflow deployment on a Kubernetes cluster using the Helm package manager.

Quickstart

To install this helm chart remotely (using helm 3)

kubectl create namespace airflow

helm repo add astronomer https://helm.astronomer.io
helm install airflow --namespace airflow astronomer/airflow

To install airflow with the KEDA autoscaler

helm repo add kedacore https://kedacore.github.io/charts
helm repo add astronomer https://helm.astronomer.io

helm repo update

kubectl create namespace keda
helm install keda \
    --namespace keda kedacore/keda \
    --version "v1.5.0"

kubectl create namespace airflow

helm install airflow \
    --set executor=CeleryExecutor \
    --set workers.keda.enabled=true \
    --set workers.persistence.enabled=false \
    --namespace airflow \
    astronomer/airflow

To install this repository from source

kubectl create namespace airflow
helm install --namespace airflow .

Prerequisites

• Kubernetes 1.12+
• Helm 3.6+
• PV provisioner support in the underlying infrastructure

Installing the Chart

To install the chart with the release name my-release:

helm install --name my-release .

The command deploys Airflow on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Upgrading the Chart

First, look at the updating documentation to identify any backwards-incompatible changes.

To upgrade the chart with the release name my-release:

helm upgrade --name my-release .

Uninstalling the Chart

To uninstall/delete the my-release deployment:

helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

Updating DAGs

Bake DAGs in Docker image

The recommended way to update your DAGs with this chart is to build a new docker image with the latest code (docker build -t my-company/airflow:8a0da78 .), push it to an accessible registry (docker push my-company/airflow:8a0da78), then update the Airflow pods with that image:

helm upgrade my-release . \
  --set images.airflow.repository=my-company/airflow \
  --set images.airflow.tag=8a0da78

Docker Images

• The Airflow image that are referenced as the default values in this chart are generated from this repository: https://github.com/astronomer/ap-airflow.
• Other non-airflow images used in this chart are generated from this repository: https://github.com/astronomer/ap-vendor.

Parameters

The complete list of parameters supported by the community chart can be found on the Parameteres Reference page, and can be set under the airflow key in this chart.

The following tables lists the configurable parameters of the Astronomer chart and their default values.

Parameter	Description	Default
ingress.enabled	Enable Kubernetes Ingress support	false
ingress.acme	Add acme annotations to Ingress object	false
ingress.tlsSecretName	Name of secret that contains a TLS secret	~
ingress.webserverAnnotations	Annotations added to Webserver Ingress object	{}
ingress.flowerAnnotations	Annotations added to Flower Ingress object	{}
ingress.baseDomain	Base domain for VHOSTs	~
ingress.auth.enabled	Enable auth with Astronomer Platform	true
workers.autoscaling.enabled	Traditional HorizontalPodAutoscaler	false
workers.autoscaling.minReplicas	Minimum amount of workers	1
workers.autoscaling.maxReplicas	Maximum amount of workers	10
workers.autoscaling.targetCPUUtilization	Target CPU Utilization of workers	80
workers.autoscaling.targetMemoryUtilization	Target Memory Utilization of workers	80
extraObjects	Extra K8s Objects to deploy (these are passed through tpl). More about Extra Objects.	[]
sccEnabled	Enable security context constraints required for OpenShift	false
authSidecar.enabled	Enable authSidecar	false
authSidecar.repository	The image for the auth sidecar proxy	nginxinc/nginx-unprivileged
authSidecar.tag	The image tag for the auth sidecar proxy	stable
authSidecar.pullPolicy	The K8s pullPolicy for the the auth sidecar proxy image	IfNotPresent
authSidecar.port	The port the auth sidecar exposes	8084
gitSyncRelay.enabled	Enables git sync relay feature.	False
gitSyncRelay.repo.url	Upstream URL to the git repo to clone.	~
gitSyncRelay.repo.branch	Branch of the upstream git repo to checkout.	main
gitSyncRelay.repo.depth	How many revisions to check out. Leave as default 1 except in dev where history is needed.	1
gitSyncRelay.repo.wait	Seconds to wait before pulling from the upstream remote.	60
gitSyncRelay.repo.subPath	Path to the dags directory within the git repository.	~

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

helm install --name my-release \
  --set executor=CeleryExecutor \
  --set enablePodLaunching=false .

Autoscaling with KEDA

KEDA stands for Kubernetes Event Driven Autoscaling. KEDA is a custom controller that allows users to create custom bindings to the Kubernetes Horizontal Pod Autoscaler. We've built an experimental scaler that allows users to create scalers based on postgreSQL queries. For the moment this exists on a separate branch, but will be merged upstream soon. To install our custom version of KEDA on your cluster, please run

helm repo add kedacore https://kedacore.github.io/charts

helm repo update

helm install \
    --set image.keda=docker.io/kedacore/keda:1.2.0 \
    --set image.metricsAdapter=docker.io/kedacore/keda-metrics-adapter:1.2.0 \
    --namespace keda --name keda kedacore/keda

Once KEDA is installed (which should be pretty quick since there is only one pod). You can try out KEDA autoscaling on this chart by setting workers.keda.enabled=true your helm command or in the values.yaml. (Note: KEDA does not support StatefulSets so you need to set worker.persistence.enabled to false)

helm repo add astronomer https://helm.astronomer.io
helm repo update

kubectl create namespace airflow

helm install airflow \
    --set executor=CeleryExecutor \
    --set workers.keda.enabled=true \
    --set workers.persistence.enabled=false \
    --namespace airflow \
    astronomer/airflow

Walkthrough using kind

Install kind, and create a cluster

We recommend testing with Kubernetes 1.21+, example:

kind create cluster --image kindest/node:v1.21.14

Confirm it's up:

kubectl cluster-info --context kind-kind

Add Astronomer's Helm repo

helm repo add astronomer https://helm.astronomer.io
helm repo update

Create namespace + install the chart

kubectl create namespace airflow
helm install airflow -n airflow astronomer/airflow

It may take a few minutes. Confirm the pods are up:

kubectl get pods --all-namespaces
helm list -n airflow

Run kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working. Login as admin and password admin.

Build a Docker image from your DAGs

1. Start a project using astro-cli, which will generate a Dockerfile, and load your DAGs in. You can test locally before pushing to kind with astro airflow start.

   mkdir my-airflow-project && cd my-airflow-project
   astro dev init

2. Then build the image:

   docker build -t my-dags:0.0.1 .

3. Load the image into kind:

   kind load docker-image my-dags:0.0.1

4. Upgrade Helm deployment:

   helm upgrade airflow -n airflow \
       --set images.airflow.repository=my-dags \
       --set images.airflow.tag=0.0.1 \
       astronomer/airflow

Extra Objects

This chart can deploy extra Kubernetes objects (assuming the role used by Helm can manage them). For Astronomer Cloud and Enterprise, the role permissions can be found in the Commander role.

extraObjects:
  - apiVersion: batch/v1beta1
    kind: CronJob
    metadata:
      name: "{{ .Release.Name }}-somejob"
    spec:
      schedule: "*/10 * * * *"
      concurrencyPolicy: Forbid
      jobTemplate:
        spec:
          template:
            spec:
              containers:
                - name: myjob
                  image: ubuntu
                  command:
                    - echo
                  args:
                    - hello
              restartPolicy: OnFailure

Contributing

Check out our contributing guide!

License

Apache 2.0 with Commons Clause