rogercli/azuredatafactory_etl

Azure Data Factory ETL Pipeline

Overview

Step 1 - Explore, transform, and load data into Data Warehouse using Apache Spark

Step 2 - Transform data with Azure Data Factory

Step 3 - Optimise query performance with dedicated SQL pools in Azure Synapse

Key Areas to Address

• Why should one use Azure Key Vault when working in the Azure environment? What are the alternatives to using Azure Key Vault? What are the pros and cons of using Azure Key Vault?

• How do you achieve the loop functionality within an Azure Data Factory pipeline? Why would you need to use this functionality in a data pipeline?

• What are expressions in Azure Data Factory? How are they helpful when designing a data pipeline?

• What are the pros and cons of parametrizing a dataset in Azure Data Factory pipeline’s activity?

• What are the different supported file formats and compression codecs in Azure Data Factory? When will you use a Parquet file over an ORC file? Why would you choose an AVRO file format over a Parquet file format?