vdn-curated/data-engineering-howto

A list of useful resources to learn Data Engineering from scratch

How To Become a Data Engineer

Useful articles

• The AI Hierarchy of Needs
• The Rise of Data Engineer
• The Downfall of the Data Engineer
• A Beginner’s Guide to Data Engineering
  • Part I
  • Part II
  • Part III
• Functional Data Engineering — a modern paradigm for batch data processing
• How to become a Data Engineer (in Russian)

Algorithms & Data Structures

• Algorithmic Toolbox in Russian
• Data Structures in Russian
• Data Structures & Algorithms Specialization on Coursera
• Algorithms Specialization from Stanford on Coursera

SQL

• Comprehensive SQL Tutorial by Mode Analytics
• SQL Practice on Leetcode
• Modern SQL a website about modern SQL syntax

Programming

• Scala School by Twitter
• Fluent Python intermediate level book about Python
• Intro to Scala in Russian on Stepik by Tinkoff Bank
• The Hitchhiker’s Guide to Python by Kenneth Reitz & Tanya Schlusser

Databases

• Intro to Database Systems by Carnegie Mellon University
• Advanced Database Systems by Carnegie Mellon University
• On Disk IO
  • I. Flavors of IO
  • II. More Flavours of IO
  • III. LSM Trees
  • IV. B-Trees and RUM Conjecture
  • V. Access Patterns in LSM Trees

Distributed Systems

• Distributed systems for fun and profit by Mikito Takada
• Distributed Systems by by Maarten van Steen & Andrew S. Tanenbaum
• CS 436: Distributed Computer Systems by University of Waterloo

Books

• Design Data-Intensive Applications by Martin Kleppmann
• Introduction to Algorithms by Thomas Cormen
• The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling
• Star Schema The Complete Reference
• The Data Engineering Cookbook by Andreas Kretz

Courses

• Big Data for Data Engineers Specialization by Yandex
• Data Engineering on Google Cloud Platform Specialization by Google
• Data Engineer Nanodegree by Udacity

Blogs

• Martin Kleppmann author of Designing Data-Intensive Application
• BaseDS by Vaidehi Joshi about Distributed Systems

Tools

• Apache Airflow is a platform to programmatically author, schedule and monitor workflows in Python
• Apache Spark is a unified analytics engine for large-scale data processing
• Apache Kafka is a distributed streaming platform
• Luigi is a Python package that helps you build complex pipelines of batch jobs.

Cloud Platforms

• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure
• Yandex Cloud
• DigitalOcean
• IBM Cloud

Other

• Data Engineering Podcast
• Insight Data Engineering Ecosystem Map

Newsletters & Digests

• Data Eng Weekly - Your weekly Data Engineering news
• SF Data Weekly - A weekly email of useful links for people interested in building data platforms
• Data Elixir - Data Elixir is an email newsletter that keeps you on top of the tools and trends in Data Science.