azure/data-product-analytics

Template to deploy a Data Product for analytics and data science use-cases into a Data Landing Zone of the Data Management & Analytics Scenario (former Enterprise-Scale Analytics). The Data Product template can be used by cross-functional teams to create insights and products for external users.

Cloud-scale Analytics Scenario - Data Product Analytics

Objective

The Cloud-scale Analytics Scenario provides a prescriptive data platform design coupled with Azure best practices and design principles. These principles serve as a compass for subsequent design decisions across critical technical domains. The architecture will continue to evolve alongside the Azure platform and is ultimately driven by the various design decisions that organizations must make to define their Azure data journey.

The Cloud-scale Analytics architecture consists of two core building blocks:

1. Data Management Landing Zone which provides all data management and data governance capabilities for the data platform of an organization.
2. Data Landing Zone which is a logical construct and a unit of scale in the Cloud-scale Analytics architecture that enables data retention and execution of data workloads for generating insights and value with data.

The architecture is modular by design and allows organizations to start small with a single Data Management Landing Zone and Data Landing Zone, but also allows to scale to a multi-subscription data platform environment by adding more Data Landing Zones to the architecture. Thereby, the reference design allows to implement different modern data platform patterns like data-mesh, data-fabric as well as traditional datalake architectures. Cloud-scale Analytics Scenario has been very well aligned with the data-mesh approach, and is ideally suited to help organizations build data products and share these across business units of an organization. If core recommendations are followed, the resulting target architecture will put the customer on a path to sustainable scale.

---

The Cloud-scale Analytics Scenario represents the strategic design path and target technical state for your Azure data platform.

---

This repository describes a Data Product template for Data Analytics and Data Science. Data Products are another unit of scale inside a Data Landing Zone through the means of Resource Groups. Resource Groups inside the Data Landing Zone subscription are created and handed over to cross-functional teams to provide them an environment in which they can work on their own data use-cases. The ownership of this resource group and operation of services within is handed over to the Data Product teams. In order to enable self-service, the owning teams are free to deploy their own services within the guardrails set by Azure Policy. Repository templates can be used for these teams to more quickly scale within an organization and rollout common data analysis patterns not just once but multiple times across various use-cases. The ownership of templates is also handed over, which ultimately gives these teams a starting point while allowing them to enhance the template based on their specific requirements. This Data Product template deploys a set of services, which can be used for data analytics and data science. The template includes services such as Azure Machine Learning, Cognitive Services and Azure Search. The Data Product teams can then leverage these tools to generate insights and value with data.

Note: Before getting started with the deployment, please make sure you are familiar with the complementary documentation in the Cloud Adoption Framework. Also, before deploying your first Data Product, please make sure that you have deployed a Data Management Landing Zone and at least one Data Landing Zone. The minimal recommended setup consists of a single Data Management Landing Zone and a single Data Landing Zone.

Deploy Cloud-scale Analytics Scenario

The Cloud-scale Analytics architecture is modular by design and allows customers to start with a small footprint and grow over time. In order to not end up in a migration project, customers should decide upfront how they want to organize data domains across Data Landing Zones. All Cloud-scale Analytics architecture building blocks can be deployed through the Azure Portal as well as through GitHub Actions workflows and Azure DevOps Pipelines. The template repositories contain sample YAML pipelines to more quickly get started with the setup of the environments.

Reference implementation	Description	Deploy to Azure	Link
Cloud-scale Analytics Scenario	Deploys a Data Management Landing Zone and one or multiple Data Landing Zones all at once. Provides less options than the the individual Data Management Landing Zone and Data Landing Zone deployment options. Helps you to quickly get started and make yourself familiar with the reference design. For more advanced scenarios, please deploy the artifacts individually.
Data Management Landing Zone	Deploys a single Data Management Landing Zone to a subscription.		Repository
Data Landing Zone	Deploys a single Data Landing Zone to a subscription. Please deploy a Data Management Landing Zone first.		Repository
Data Product Batch	Deploys a Data Workload template for Data Batch Analysis to a resource group inside a Data Landing Zone. Please deploy a Data Management Landing Zone and Data Landing Zone first.		Repository
Data Product Streaming	Deploys a Data Workload template for Data Streaming Analysis to a resource group inside a Data Landing Zone. Please deploy a Data Management Landing Zone and Data Landing Zone first.		Repository
Data Product Analytics	Deploys a Data Workload template for Data Analytics and Data Science to a resource group inside a Data Landing Zone. Please deploy a Data Management Landing Zone and Data Landing Zone first.		Repository

Deploy Data Product

To deploy the Data Product into your Data Landing Zone, please follow the step-by-step instructions:

1. Prerequisites
2. Create repository
3. Setting up Service Principal
4. Template Deployment
   1. GitHub Action Deployment
   2. Azure DevOps Deployment
5. Known Issues

Contributing

Please review the Contributor's Guide for more information on how to contribute to this project via Issue Reports and Pull Requests.

data-product-analytics's Issues

AML Compute Deployments fail

Describe the bug
The AML Compute Cluster and Compute Instance deployments fail, when deploying to a region, where the 'no public IP' feature is not available. Hence, we have to change the configuration depending on the selected region of the deployment.

Steps to reproduce

1. Deploy Data Product via ADO, GitHub or Portal
2. Select a region like "UK South" where the feature is not supported
3. Deployment fails.

Screenshots

Feature Request: Add Information about Service Endpoint Requirements

Describe the solution you'd like
Please add to the docs as well as the Portal experience information about the service endpoint requirements of Azure Machine Learning. Azure Machine Learning and the computes require the Storage Service Endpoint to be enabled on the subnet that is used.

Bug: Alerts.bicep not alerting

Deployment Mode

Azure DevOps

Steps to reproduce

When deployed the new logging elements, for example Synapse pipeline errors did not trigger any alerts.

In the script, the 'static threshold' operator is "GreaterThan 1" when that likely should be "GreaterThanOrEqual 1".

AFAIK, this exists in other products' Alerts.bicep files as well.

Error Message

No response

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Documentations needed for 2 env variables in updateParameters.yml

These two env variables in the updateParameters.yml are not covered in the documentation:

  ML_COMPUTE_INSTANCE_USER_OBJECT_ID: '{mlComputeInstanceObjectId}'
  DATABRICKS_WORKSPACE_ID: '{databricksWorkspaceId}'

It would be nice to list them in the parameter update process and also the details on where we can find these two.

Feature Request: Select Multiple Azure Cognitive Services

Describe the solution you'd like
Marvin it will be great if we have a multi choice deployment for cog services given that some customers may use more than one.
Thanks Sudha

Update ARM templates and add new features

New features need to be added to be in line with latest features:

• Private Link for AML
• Connection to Synapse
• Connection to ADB
• Datastores
• Datasets
• ...

Wrong Name for Container Build Compute

Describe the bug
The bicep templates reference the wrong name for the container build compute. This needs to be fixed to be able to build images in the customer environment.

Steps to reproduce

1. Deploy Data Product Analytics through any of the of the available options
2. Try to build a container image in AML
3. Wrong compute name is referenced.

Screenshots
No Screenshots

Compute Cluster Deployment Fails for Policy DNS A-record Deployment

Describe the bug
If the user deploys this template, the CPU Cluster and GPU Cluster deployments fails, with the following error message:

{
    "status": "Failed",
    "error": {
        "code": "UserError",
        "message": "{\n  \"error\": {\n    \"code\": \"UserError\",\n    \"severity\": null,\n    \"message\": \"Provided EnableNodePublicIp 'False' requires workspace private endpoint.\",\n    \"messageFormat\": null,\n    \"messageParameters\": null,\n    \"referenceCode\": null,\n    \"detailsUri\": null,\n    \"target\": null,\n    \"details\": [],\n    \"innerError\": null,\n    \"debugInfo\": null,\n    \"additionalInfo\": null\n  },\n  \"correlation\": {\n    \"operation\": \"\",\n    \"request\": \"\"\n  },\n  \"environment\": \"eastus2\",\n  \"location\": \"eastus2\",\n  \"time\": \"2021-08-02T12:17:49.9687945+00:00\",\n  \"componentName\": \"machinelearningcompute\"\n}",
        "details": []
    }
}

Steps to reproduce

1. Deploy this template.
2. See how it fails.

Screenshots

"Deployment 'synapse001' could not be found" error

Describe the bug
When deploying this solution with the processingService parameter set to "dataFactory", you receive a deployment error:
Deployment 'synapse001' could not be found.

Steps to reproduce

1. Set processingService to "dataFactory"
2. Run the deployment
3. Get the error
4. ???
5. Profit!!!

Screenshots

(Pull request to fix this is forthcoming.)

Update Portal Experience

• Remove VS Code reference for CIs
• Add option to connect Container Registry
• Change default option for DNS A-records (deployed through Policy)

Name change

Please change all instances of "Enterprise-scale analytics and ai" to "Data Management and Analytics Scenario" to align to marketing ask. This should happen for one-clicks and all documentation.

Feature Request: Add Azure Cognitive Service for Language

Describe the solution you'd like
Add new Azure Cognitive Service for Language to the list of supported Cognitive Services.
Also add potential support for this service: https://azure.microsoft.com/en-us/services/openai-service/#overview

Feature Request: Make SQL Pool Optional

Describe the solution you'd like
Make SQL Pool deployment optional when choosing Synapse as deployment option.

Documentation: about section

Documentation Issue

There doesn't seem to be a way to submit a change in the 'about' section. See below changes.

Template to deploy a Data Product for analytics and data science use-cases into a Data Landing Zone of the Cloud-Scale Analytics Scenario, (former Enterprise-Scale Analytics, Data Management & Analytics Scenario). The Data Product template can be used by cross-functional teams to create insights and products for external users.

Code of Conduct

• I agree to follow this project's Code of Conduct

ACTION REQUIRED: Microsoft needs this private repository to complete compliance info

There are open compliance tasks that need to be reviewed for your data-product-analytics repo.

Action required: 4 compliance tasks

To bring this repository to the standard required for 2021, we require administrators of this and all Microsoft GitHub repositories to complete a small set of tasks within the next 60 days. This is critical work to ensure the compliance and security of your Azure GitHub organization.

Please take a few minutes to complete the tasks at: https://repos.opensource.microsoft.com/orgs/Azure/repos/data-product-analytics/compliance

• The GitHub AE (GitHub inside Microsoft) migration survey has not been completed for this private repository
• No Service Tree mapping has been set for this repo. If this team does not use Service Tree, they can also opt-out of providing Service Tree data in the Compliance tab.
• No repository maintainers are set. The Open Source Maintainers are the decision-makers and actionable owners of the repository, irrespective of administrator permission grants on GitHub.
• Classification of the repository as production/non-production is missing in the Compliance tab.

You can close this work item once you have completed the compliance tasks, or it will automatically close within a day of taking action.

If you no longer need this repository, it might be quickest to delete the repo, too.

GitHub inside Microsoft program information

More information about GitHub inside Microsoft and the new GitHub AE product can be found at https://aka.ms/gim or by contacting [email protected]

FYI: current admins at Microsoft include @marvinbuss, @daltondhcp, @esbran

Feature Request: set up diagnostic logging for Azure ML workspace

As part of the provisioning of the Azure ML workspace, we should set up diagnostic logging into a central log analytics workspace.

Feature: Pull Base Image from private ACR

Feature or Idea - What?

See here: https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-managed-identities?tabs=python#build-azure-machine-learning-managed-environment-into-base-image-from-private-acr-for-training-or-inference

Feature or Idea - Why?

Required for private base images.

Code of Conduct

• I agree to follow this project's Code of Conduct

Feature Request

Describe the solution you'd like
In the prereqs we need more clarity on enterprise landing zone for data being optional to deploy the asset and also need clarity on the min number of assets that needs to be deployed for deploying AML. For instance we just need data factory to deploy along with AML and do not need synapse, cog services and so on..

Bug: AzureML cannot access images in Azure Container Registry with Quarantine flag

Deployment Mode

Deploy To Azure

Steps to reproduce

1. Deploy solution
2. Run AzureML job or Deployment online-endpoint - samples: Samples/SDK v2/sdk/python/endpoints/online/managed/online-endpoints-simple-deployment.ipynb

Error Message

(None) ResourceNotFound: Deployment failed due to timeout while waiting for Environment Image to become available. Check Environment Build Log in Workspace or Workspace storage for potential failures. Environment info: Name: CliV2AnonymousEnvironment, Version: f53727d1df8137354bc0f2c69d2619ee, Build log may be available at (log Please see troubleshooting guide, available here: https://aka.ms/oe-tsg#error-resourcenotfound Code: None

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Feature Request: Add Docs about Security of the Solution pattern

Hi Marvin,
Please outline what are the features that are getting deployed in terms of security, network and baseline features for the workload esp AML.
Thanks
Sudha

Improvement: Optional resources cause errors in the deployment

Deployment Mode

Azure DevOps

Steps to reproduce

Optional resources deployed causing error as the references to optional resources is present in the deployment template

Error Message

resource not found

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

Add Private Link Connectivity between Data Factory and Purview

Describe the solution you'd like
Automate the creation of managed private endpoints on the Data Factory managed vnet. This will allow private communication with Purview to collect Lineage Data.

Add Infobox for RG Selection & Improve Network selection

Describe the solution you'd like

• Add an infobox when selecting the RG for the product deployment
• Improve network selection and only show correctly configured subnets