josephmachado/data_engineering_best_practices

Sample project to demonstrate data engineering best practices

Hi 👋

I write about data engineering | data modeling | software engineering | system design

• 🔭 I write data engineering articles on Start data engineering

• 👨‍💻 All of my projects are available at https://github.com/josephmachado

• 📝 I regularly write articles on https://www.startdataengineering.com/

• 💬 Ask me about Data engineering, SQL, Databases, Data pipelines, Data infrastructure

• 📫 How to reach me https://www.startdataengineering.com/contact

• 📄 Know about my experiences https://www.linkedin.com/in/josephmachado1991/

Connect with me:

Languages and Tools:

data_engineering_best_practices's Issues

Build cloud infrastructure

Description

Currently the project is only meant to be run locally, implement a solution to deploy this to the cloud.

Areas to cover

1. Platform to use
2. CI/CD
3. Env variable management (w Secret managements)
4. Cloud storage & processing.
5. Scheduling & Orchestration.
6. Logging, metadata & debugging

Choice of platform

Please leave a comment with your choices for areas 1 - 5 above.