athivvat/data-engineer-roadmap

A Roadmap to Becoming a Data Engineer ⚙️👨🏻‍🔧

Data Engineer Roadmap

Overview

This is resources for going from software engineer/non-software engineer to profressional in AI engineer.

About Me

I'm a Thai Software Engineer who is really passionate to become entrepreneur, investor, and AI engineer. Was found Humaan.ai to bringing the AI to everyone to do incredible ways.

Currently, I'm studying for a master's degree in Business Analytics and Data Sciences, where majoring in Artificial Intelligence and Machine Learning at National Institute of Development Administration (NIDA), Thailand.

Who is Data Engineer

“A scientist can discover a new star, but he cannot make one. He would have to ask an engineer to do it for him." — Gordon Lindsay Glegg

Data Engineers are the data professionals who collect, move & store, data preparation to be analyzed by Data Scientists.

Data engineering is all about the movement, manipulation, and management of data.

Contents

Step 1: Introduction

• What is Data Engineering?
• Who is Data Engineer?
• What Does a Data Engineer Do?

Step 2: Big Data

Step 3: Data Life Cycle

• Generation
• Collection
• Processing
• Storage
• Management
• Analysis
• Interpretation

Step 4: Batch, Microbatch and Streaming

4.1 Batch Processing

4.2 Streaming Processing

• Reliability, Throughput, and Flexibility
• Message Queue Technologies
  • Apache Kafka
  • ActiveMQ
  • RabbitMQ
  • Amazon’s Simple Queue Service (SQS)

Step 5: Distributed Computing

Step 6: Cloud Computing

Certifications

• Google Cloud Data Engineer Certification
• Cloudera Certified Professional Data Engineer
• Microsoft Certified Solutions Associate in Data Engineering with Azure

Resources

Books

• Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing
• Stream Processing with Apache Spark: Mastering Structured Streaming and Spark Streaming

Working at Humaan.ai

Our mission is to "solve intelligence" and enable AI technologies to unleash everyone capabilites and to address humanity's grand challenges.

[Jobs at Humaan.ai]