azure/azure-data-labs-modules

A list of Terraform modules to build your Azure Data IaC templates.

Azure Data Labs - Modules

This repository includes a library of curated Terraform modules as well as a Continuous Integration (CI) environment using GitHub Actions for modules' validation.

All modules follows the naming convention outlined in the Cloud Adoption Framework. Resources will be prefixed with the appropriate abbreviations.

Looking for built-in templates? Check out Azure Data Labs templates repository to deploy built-in templates with Terraform and GitHub Actions.

Ideas? Contributions? Bugs? If you'd like to add more resources or you run into any issues, feel free to create a pull request or open an issue in this repository.

Available Modules

Name	Status
Azure Kubernetes Service
Analysis Services Server
Application Insights
Batch Services / Account
Bastion Host
Cognitive Search
Cognitive Services / Account
Cognitive Services / Deployment
Container Registry
Cosmos DB / Account
Cosmos DB / Cassandra Keyspace
Cosmos DB / Gremlin Database
Cosmos DB / Mongo Database
Cosmos DB / SQL Database
Cosmos DB / Table
Data Explorer / Kusto Cluster
Data Explorer / Kusto Database
Data Factory / Data Factory
Data Factory / Self-Hosted Integration Runtime
Data Share / Data Share
Data Share / Data Share Account
Databricks / Workspace
Databricks / Access Connector
Event Grid / Domain
Event Grid / Topic
Event Hubs / Event Hubs
Event Hubs / Namespace
Fabric / Capacity
Firewall
Function App / Linux Function App
HDInsight / Hadoop Cluster
HDInsight / HBase Cluster
HDInsight / Interactive Query Cluster
HDInsight / Kafka Cluster
HDInsight / Spark Cluster
Key Vault
Load Balancer
Log Analytics / Log Analytics Cluster
Log Analytics / Log Analytics Workspace
Machine Learning / Computer Cluster
Machine Learning / Compute Instance
Machine Learning / Inference Cluster
Machine Learning / Synapse Spark
Machine Learning / Workspace
MySQL Database
Network Security Group
Network Security Rule
PostgreSQL Database
Power BI Embedded
Private DNS Zone
Private Endpoint
Purview
Redis Cache
Redis Enterprise
Resource Group
Route Table
Route
Route Table
Service Bus / Namespace
Service Bus / Queue
Service Bus / Subscription
Service Bus / Topic
Service Plan
SQL Database
SQL Database Server
SQL Managed Instance
Storage Account
Stream Analytics Job
Subnet
Subnet Network Security Group Association
Subnet Route Table Association
Synapse / Kusto Pool
Synapse / Private Link Hub
Synapse / Spark Pool
Synapse / SQL Pool
Synapse / Workspace
User Assigned Identity
Virtual Machine
Virtual Network
Virtual Network Peering

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Learn More

• What is Infrastructure as Code?
• Overview of Terraform on Azure
• Terraform Modules
• GitHub Actions

azure-data-labs-modules's Issues

Add support for module_enabled to the cognitive-search module

Add module enablement to CosmosDB Databases

Currently only module enablement is supported for the cosmosdb-account module through the module-enable variable.
Following the same principle, module enablement should be available for the rest of the cosmosdb modules: cosmosdb-cassandra-keyspace, cosmosdb-gremlin-database, cosmosdb-mongo-database and cosmosdb-sql-database

Add support for CMK in Synapse module

The Synapse module doesn't have the option to use a CMK to encrypt data at rest.

https://learn.microsoft.com/en-us/azure/synapse-analytics/security/workspaces-encryption#encryption-of-data-at-rest

The example of that in the azurerm Terraform provider is:
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/synapse_workspace#example-usage---creating-a-workspace-with-customer-managed-key-and-azure-ad-admin

Thank you.

[new-module] Add Azure Functions module

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/function_app

[new module] Stream Analytics cluster

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/stream_analytics_cluster

Add support for module_enabled to the event-grid-domain module

Add support for module_enabled to the log-analytics-cluster module

[new module] Log Analytics Cluster

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/log_analytics_cluster

[new module] Databricks Workspace configuration & IP access list

https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/workspace_conf
https://registry.terraform.io/providers/databrickslabs/databricks/latest/docs/resources/ip_access_list

[new module] Self Hosted Integration Runtime

Includes:

• VM
• VM extension to install the SHIR

[new module] Cosmos DB Cassandra Cluster

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cosmosdb_cassandra_cluster

Add CMK support to Postgres module

Improve validations of variables

Add support for module_enabled to the private-dns-zone module

Add support for module_enabled to the purview module

Add dns-zone-vnet-link dependency in Databricks module tests

When deploying a Databricks workspace with private endpoints, there should be an explicit dependency with the private dns zone vnet link.

This is a race condition which will happen on rare occasions when the deployment of the private dns zone link takes more time than the workspace in the module test.

Standardize output variables and make sure they are tested

[new module] Azure Batch

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/batch_account

Includes:

• Private endpoints with respective private dns zones

Add support for module_enabled to the resource-group module

[new module] Data Share

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/data_share

Includes:

• Data share account
• Data share

[new module] Virtual Network Peering

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/virtual_network_peering

Synapse Kusto Pools module test should get WorkspaceUID from the workspace it creates

Add support for module_enabled to the log-analytics-workspace module

Add support for module_enabled to the route module

[new module] Databricks Unity Catalog Metastore

https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/metastore

[new module] Cosmos DB Cassandra Data Center

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/cosmosdb_cassandra_datacenter

[new module] Stream Analytics job

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/stream_analytics_job

[new module] Redis Enterprise

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/redis_enterprise_database

Add support for module_enabled to the kusto-database module

Allow multiple capabilities in CosmosDB Account

Right now only one value for enable_capabilities is allowed, and if leave default to empty, it gives an error:
Error: expected capabilities.0.name to be one of [EnableAggregationPipeline EnableCassandra EnableGremlin EnableTable EnableServerless EnableMongo MongoDBv3.4 mongoEnableDocLevelTTL DisableRateLimitingResponses AllowSelfServeUpgradeToMongo36], got

The parameter should change to enable_capabilities and be a list of strings, defaulting to an empty list

[new module] Azure Firewall

AzureRM https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/firewall

Upgrade to greatest and latest AzureRM

Add support for module_enabled to the event-grid-topic module

[new module] Synapse Kusto pool

AzAPI: https://learn.microsoft.com/en-us/azure/templates/microsoft.synapse/workspaces/kustopools?pivots=deployment-language-terraform

This modules still is not available on https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs

Add support for module_enabled to the kusto-cluster module

[new-module] Add Power BI embedded module

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/powerbi_embedded

Add support for module_enabled to the cognitive-services module

Add support for module_enabled to the linux-function-app module

Add support for module_enabled to the network-security-rule module

Refactor adb module to use other module dependencies

[new module] Add Event Hub module

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/eventhub

Includes:

• Remove Event Hub from https://github.com/Azure/azure-data-labs-modules/tree/main/terraform/event-hubs/event-hubs-namespace, just keeping the name space.
• Create a separated Event Hub module

Add support for module_enabled to the key-vault module

Add support for module_enabled to the aks-cluster module

Add support for module_enabled to the application-insights module

Add support for module_enabled to the container-registry module

Secure aks cluster, including private endpoints

Add support for module_enabled to the network-security-group module

Upgrade to the latest and greatest AzureRM

Add support for module_enabled to the databricks module

[new module] Add SQL Managed Instance module

AzureRM: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/powerbi_embedded