zoinertejada/azure-data-guide

Azure Data Architecture Guide

azure-data-guide's Issues

Provide bread crumbs to articles

Provide bread crumbs to articles so that you can see what this section is about.

Example:
https://github.com/ZoinerTejada/azure-data-guide/blob/master/data-guide/common-architectures/transactional-data.md

Proposal:
Common Architectures > Relational > Transactional Data

Remarks:
This is probably done via the Azure documentation header but wanted to make sure that it's there as this can be interesting to see where you are.

Guidance for data storage without business use-case

In some scenarios there's a need to start storing potential interesting information, without knowing how it will be analyzed or processed further.

Maybe this is an interesting topic that we can tackle somewhere to provide guidance on.
Do we just dump it in Azure Data Lake Store or are there any other relevant locations?

Guidance on design patterns

Guidance on design patterns such as sharding, caching, etc or having a link to the existing material at least and how to apply them.

Although this might already be covered.

Azure Logic Apps as an orchestrator

Any plans on adding Azure Logic Apps as an orchestrator for "Pipeline Orchestration, Control Flow and Data Movement"?

This works very well with other orchestrators such as Azure Data Factory where one can trigger the other.
I'm working on a blog post on this topic (aiming for beginning Dec), feel free to let me know if you're interested in this topic.

Guidance on operational experience

Guidance on operational experience

Guidance for data integration scenarios

Any plans on adding guidance for data integration scenarios?

As an example - Every hour we need to fetch data with an external data source and process this.

Maybe this can be part of "Data Integration" section or the existing "Pipeline Orchestration, Control Flow and Data Movement" section.

Guidance on Distaster Recovery

Any plans on providing guidance on disaster recovery?

ELT schematic in Data Pipelines has wrong headers

The image in Extract, Transform & Load (ETL) has the wrong headers in the schematic and should be swapped:

Data Pipelines - How are they triggered?

Would it be worth mentioning the scenarios on how a data pipeline can be triggered?

I'm thinking of the following:

• On-Demand
• Time slicing ie. every hour, day, month
• Scheduled ie every day at 12 AM
• Triggered from within another process such as Azure Logic App or another pipeline

I can contribute to this if you'd like.

Guidance on database vs data warehousing

For people whom are new to data analytics/processing it's sometimes hard to determine when a SQL database is good enough, if they need a data warehouse instead or need a mixture of both.

A scenario for this is where you have your API & backend processing using a database. Later on you need to provide reporting on a subset of that data. Do you run these on top of that database (pref. a replica for performance) or do you move that subset to a SQL data warehouse and run reports on top of that?

Interesting topics:

• Impact on pricing and how this can be optimized
• Design for this workload (Can be linked back to "Pipeline Orchestration, Control Flow and Data Movement")

Flow chart in Common Data Architectures does not include ELT

Flow chart in Common Data Architectures does not include ELT

Guidance on manageability of Azure services

Guidance on manageability of Azure services.

• Clusters offerings (ie Azure Data Lake HDI) vs hosting
• Serverless offerings such as (ie Azure Data Lake Analytics)

Not sure if that is the best word but mainly looking at

Introduce Azure Time Series Insights

Any plans on introducing Azure TIme Series Insights?

Guidance on resource & cost optimizations

Guidance on resource & cost optimizations.

The following should be mentioned at least:

• Elastic Pools vs Single Databases

Relates to #9

Provide high-level overview of all services

It might be interesting to provide a high-level overview of all services that are discussed and what purpose they accomplish, to a certain degree.

Example from this blog post:

Guidance on scalability

Guidance on scalability in terms of auto scaling based on time or workload.

Scenarios:

• Auto-scaling based on time
  • Scale up database at 9 AM
  • Scale down at 6 PM
• Auto-scaling based on workload
  • Big spike in traffic, can we autoscale on that?

Extend "Options for Pipeline Orchestration, Control Flow, and Data Movement" with ADF 2.0

Does it make sense to extend "Options for Pipeline Orchestration, Control Flow, and Data Movement" now that Azure Data Factory is in preview and allows you to run SSIS in the cloud & on premises as part of a pipeline?

More information here.