weiter10/data-engineer-beginner-resources

data-engineer-beginner-resources

The objective of this post is to group some of the most important thing that you need to start your career as data engineering.

Concerning programming languajes, I would say that python is the most used one. Libraries such as pandas or pyspark is a must for you. You also need to have a very basic knowledge in numpy and matplotlib IMO. To quickly practise your Python code, you would start by using Anaconda with Jupyter Notebooks. Anaconda provides to you a Python virtual environment manager.

After you have a basic-intermediate knowledge of Python, you can go throught this Udemy course: https://www.udemy.com/course/data-engineering-essentials-sql-python-and-spark/ where you will learn very important technologies that you will use every day in your work, such as SQL and Spark. You also need to have a basic knowledge of NoSQL.

I recommend you that you use git for control versioning and practising it while you learn data engineering programming.

You also need to have a basic knowledge of Linux Shell. Practise it while you are doing your programming exercises.

Terms such as data quality, ETL, Data Lake, Data Warehouse, Data Streamming (Kafka), Data Lineage is a must for you.

Continuous integration (CI) and continuous delivery (CD), GitHub Actions are terms that need to be familiar for you, although this term is for DevOps jobs. You also need to know what Docker and Kubernetes is for.

Cloud technologies is a must for you, you need to research in some Cloud technologies that are used for data engineering tasks.

Last but not least is that you need to know why your work is important, use cases. Read about this and how your work is critical to data scientist and to make available the data to the organization.

Finally, I recommend to you following people in Twitter related to Data Engineer. Reddit is very good to stay updated in this field. Newsletters such as https://www.dataengineeringweekly.com/ could be interesting.

I would like to share with you this good landscape by Lakerfs

Miscellany: https://insights.stackoverflow.com/survey/2021