bbuxton93/terraform-databricks-examples

Examples of using Terraform to deploy Databricks resources

terraform-databricks-examples

This repository contains the following:

• Examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.

• Multiple examples of Databricks workspace and resources deployment on Azure, AWS and GCP using Databricks Terraform provider.

There are two ways to use this repository:

1. Use examples as a reference for your own Terraform code: Please refer to examples folder for individual examples.
2. Reuse modules from this repository: Please refer to modules folder.

Repository structure

Code in the repository is organized into following folders:

• modules - implementation of specific Terraform modules:
• examples - specific instances that use Terraform modules.
• cicd-pipelines - Detailed examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or Github Actions.

Repository content

Note
For detailed information about the examples, modules or CICD pipelines, refer to README.md file inside corresponding folder for a detailed guide on how to setup the CICD pipeline.

Examples

The folder examples contains the following Terraform implementation examples :

Cloud	Example	Description
Azure	adb-lakehouse	Lakehouse terraform blueprints
Azure	adb-with-private-link-standard	Provisioning Databricks on Azure with Private Link - Standard deployment
Azure	adb-vnet-injection	A basic example of VNet injected Azure Databricks workspace
Azure	adb-exfiltration-protection	A sample implementation of Data Exfiltration Protection
Azure	adb-external-hive-metastore	Example template to implement external hive metastore
Azure	adb-kafka	ADB - single node kafka template
Azure	adb-private-links	Azure Databricks Private Links
Azure	adb-splunk	ADB workspace with single VM splunk integration
Azure	adb-squid-proxy	ADB clusters with HTTP proxy
Azure	adb-teradata	ADB with single VM Teradata integration
Azure	adb-uc	ADB Unity Catalog Process
Azure	adb-unity-catalog-basic-demo	ADB Unity Catalog end to end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants
AWS	aws-workspace-basic	Provisioning AWS Databricks E2
AWS	aws-workspace-with-firewall	Provisioning AWS Databricks E2 with an AWS Firewall
AWS	aws-exfiltration-protection	An implementation of Data Exfiltration Protection on AWS
AWS	aws-workspace-with-private-link	Coming soon
AWS	aws-databricks-flat	AWS Databricks simple example
AWS	aws-databricks-modular-privatelink	Deploy multiple AWS Databricks workspaces
AWS	aws-databricks-uc	AWS UC
AWS	aws-databricks-uc-bootstrap	AWS UC
AWS	aws-remote-backend-infra	Simple example on remote backend
AWS	aws-workspace-config	Configure workspace objects
GCP	gcp-sa-provisionning	Provisionning of the identity with the permissions required to deploy on GCP.
GCP	gcp-basic	Workspace Deployment with managed vpc
GCP	gcp-byovpc	Workspace Deployment with customer-managed vpc

Modules

The folder modules contains the following Terraform modules :

Cloud	Module	Description
All	databricks-department-clusters	Terraform module that creates Databricks resources for a team
Azure	adb-lakehouse	Lakehouse terraform blueprints
Azure	adb-lakehouse-uc	Provisioning Unity Catalog resources and accounts principals
Azure	adb-with-private-link-standard	Provisioning Databricks on Azure with Private Link - Standard deployment
Azure	adb-exfiltration-protection	A sample implementation of Data Exfiltration Protection
Azure	adb-with-private-links-exfiltration-protection	Provisioning Databricks on Azure with Private Link and Data Exfiltration Protection
AWS	aws-workspace-basic	Provisioning AWS Databricks E2
AWS	aws-workspace-with-firewall	Provisioning AWS Databricks E2 with an AWS Firewall
AWS	aws-exfiltration-protection	An implementation of Data Exfiltration Protection on AWS
AWS	aws-workspace-with-private-link	Coming soon
GCP	gcp-sa-provisionning	Provisions the identity (SA) with the correct permissions
GCP	gcp-workspace-basic	Provisions a workspace with managed VPC
GCP	gcp-workspace-byovpc	Workspace with customer-managed VPC.

CICD pipelines

The folder cicd-pipelines contains the following implementation examples of pipeline:

Tool	CICD Pipeline
Github Actions	manual-approve-with-github-actions
Azure DevOps	manual-approve-with-azure-devops