data-engineering-community/data-engineering-wiki

The best place to learn data engineering. Built and maintained by the data engineering community.

data-engineering-wiki's Issues

Creating a SQL "bible" of sorts

I've been learning SQL over the past 1/2 year and the concepts aren't sticking to me like they are with Python. I've been making a dictionary of SQL concepts/tools but I realized that the wiki could benefit from one. Here is an example from my Notion page.

What do you think about building a section/page for SQL related concepts? I know there is the SQL page already but maybe a quick reference guide like this could be useful since SQL is a huge part of the data world.

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Index.md

• [ERR] https://twitter.com/intent/tweet?text=Check%20out%20this%20awesome%20resource%20for%20data%20engineering!%20https%3A//dataengineering.wiki/ | Failed: Network error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921: (certificate has expired)
  Full Github Actions output

Tools/Apache Spark

https://github.com/JPHaus/data-engineering-wiki/blob/main/Tools/Apache%20Spark.md

This note is a seedling and is a great place to make your first contribution!

Link Checker Report

Summary

Status	Count
🔍 Total	974
✅ Successful	970
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	3

Errors per input

Errors in Tools/Metricflow.md

• [404] https://transform.co/metricflow/ | Failed: Network error: Not Found
• [404] https://docs.transform.co/docs/overview/metricflow-overview | Failed: Network error: Not Found

Errors in Tools/Databases/ClickHouse.md

• [404] https://clickhouse.com/docs/en/home/ | Failed: Network error: Not Found
  Full Github Actions output

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Concepts/Graph Database.md

• [500] https://orientdb.com/ | Failed: Network error: Internal Server Error
  Full Github Actions output

Side Pane Tool category not organized || Need more subfolders

When you expand the Tools sections on the sidebar you get options for Data Ingestion ....... Workflow Orchestrators and after that, all the tools are floating outside without any category:

Can't we add some more folders like the AWS Cloud tool with subfolders like storage and put S3 inside that and the same case with all other tools?

Or to make it simple Put all AWS tools inside one AWS folder, all Azure-related stuff inside the Azure folder, and so on.

To recreate this issue:

• Go to https://dataengineering.wiki/Index and expand the Tools sidebar

OR

• Click on this link

If possible you can assign me this task and I can help you guys to organize this.

Requesting access on the repo

Hi @JPHaus ,
I am new to github and open source. I was checking your repo and wanted to contribute in it. Could you please grant me access on it?

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1031
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	5

Errors per input

Errors in FAQ/FAQ.md

• [503] https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/FAQ/FAQ.md | Failed: Network error: Service Unavailable

Errors in FAQ/What is the difference between a Data Engineer and X.md

• [503] https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/FAQ/What%20is%20the%20difference%20between%20a%20Data%20Engineer%20and%20X.md | Failed: Network error: Service Unavailable

Errors in Guides/Testing Your Data Pipeline.md

• [503] https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Guides/Testing%20Your%20Data%20Pipeline.md | Failed: Network error: Service Unavailable

Errors in Guides/Data Governance Guide.md

• [503] https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Guides/Data%20Governance%20Guide.md | Failed: Network error: Service Unavailable

Errors in Guides/SQL Guide.md

• [503] https://raw.githubusercontent.com/data-engineering-community/data-engineering-wiki/main/Guides/SQL%20Guide.md | Failed: Network error: Service Unavailable
  Full Github Actions output

Link Checker Report

Summary

Status	Count
🔍 Total	993
✅ Successful	991
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Concepts/Graph Database.md

• [ERR] https://titan.thinkaurelius.com/ | Failed: Network error: dns error: no record found for Query { name: Name("titan.thinkaurelius.com.e1fplksw42iexmyiu2m33ibpoe.ex.internal.cloudapp.net."), query_type: AAAA, query_class: IN }
  Full Github Actions output

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Tools/Databases/Rockset.md

• [404] https://rockset.com/docs/ | Failed: Network error: Not Found
  Full Github Actions output

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Tools/Databases/Rockset.md

• [404] https://rockset.com/docs/ | Failed: Network error: Not Found
  Full Github Actions output

Changing order of folders

@JPHaus. Thank you for sharing the 'publish.js' file that you are using to change the order in which files appear in the navigation bar.
Do you know if/how it is possible to change the publish.js file to also change the order of folders in the navigation bar rather than simply of notes?

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Concepts/Graph Database.md

• [500] https://orientdb.com/ | Failed: Network error: Internal Server Error
  Full Github Actions output

Add spot instances to cost optimization guide

Under general compute there should be a section about using spot instances to save money if your workload can tolerate it.

https://github.com/data-engineering-community/data-engineering-wiki/blob/main/Guides/Cost%20Optimization%20in%20the%20Cloud.md

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Concepts/Graph Database.md

• [ERR] https://orientdb.com/ | Failed: Network error: tcp connect error: Connection refused (os error 111)
  Full Github Actions output

Learning DE Roadmap

A FAQ in the community is a structured roadmap for learning Data Engineering and it's about time we start addressing it. We currently have a getting started guide but it's not detailed enough and was meant to be improved on anyways.

It can be a complex question to answer but we can simplify it by adding a few constraints. Since the majority of folks asking are those who are new to DE or trying to transition we should focus on skills for junior/entry level and mid level roles. While there aren't many jr roles at the moment it can still be useful to make the distinction for foundational skills. To make it as general as possible, I believe we should exclude tools/requirements that only apply to FAANG-like companies since they are more niche and oftentimes FAANG companies have developed their own internal tooling to solve their unique problems. Finally, the focus should be on core concepts instead of tooling. While we can include specific tools, we should try to avoid directly recommending specific tools and instead point learners to pages that have lists of the current popular tools to keep this resource as evergreen as possible (example: workflow orchestration popular tools).

While I don't believe a diagram is a requirement, I do think it could be helpful if we can get it to render nicely in mermaid because we can then make it interactive and link to other notes in the wiki like we do with other diagrams. The canvas feature for Obsidian publish is not yet supported so we would probably use a mermaid flowchart for now.

Existing popular roadmap shared in the community:

• Data Engineer Roadmap 2021 by datastacktv
  • Visually appealing
  • I like how they distinguished nice to have skills
  • Comments under major topics explaining why it's necessary is helpful
  • Just found that there's a text version as well: https://github.com/datastacktv/data-engineer-roadmap/blob/master/text/roadmap.md
  • Overall I really like it but it could probably be broken up by role or skill level to make it more manageable
  • Looks like the company is now defunct so the roadmap hasn't been updated (maybe we can adopt the repo?)

For V1, please share any thoughts/ideas/constructive criticism on the structure and core concepts. I'll start a new branch after Christmas and start something we can work from.

Feature: Project section?

Hey, I've noticed that the subreddit for r/DataEngineering tends to have the same question over and over about projects and people asking for examples. How about adding a section in the wiki for people's open source data engineering projects?

Stop working on iOS Safari

Hey Joe, thank you for share your JS! I noticed that in my website, your JS stop working on iOS Safari... I guess because Obsidian team works on sidebar some days ago.

Web site: https://lab.marconoris.com

Contributing by adding content - how about outlining potential topics?

Hi everyone, as I've read through the wiki I think it's great, but I also think it lacks some information regarding several concepts/tools which I personally find potentially relevant for data engineers. I would like to ask you what you think about the idea of gathering a list of such topics first? Some sort of a TO DO list. I think it could enable more contributions as potential contributors would be more encouraged if they knew exactly what topics could they focus on, and if their ideas are in a scope of a data engineering wiki.

Some examples of stuff that I think could be added, even if some of them are basic and/or straight-forward:

• entries regarding message brokers/queues - what are they and some sample tools such as Apache Kafka
• infrastructure stuff - virtualization, containerization, infrastructure-as-a-code etc. tools like Docker, Kubernetes, Terraform
• data visualization, including some sample industry standard tools (for instance Power BI) and open-source (such as Superset)
• maybe some more programming stuff (more languages - for example R or Rust, some OOP concepts, Big O Notation etc.)

Content on Data Contracts?

👋 hi folks - just discovered data-engineering-wiki today. What a great resource. With all the noise in the space, I love the idea of a shared, vendor neutral space to create an in-depth handbook for the community.

I'm no expert on Data Contracts, but noticed that it's a hot topic these days, and I would love to have some reference content here. I'm wondering if y'all think that would be valuable and if so where that content might land. I'm not necessarily volunteering to author this (but I could probably be convinced to do some research and start a draft PR for collaboration 🙂 ).

Addition of more Tools || New pages

Adding of Tools Like :

• Power Automate , power Apps
  And some Visualization Tools like:
• Power BI, QlikView, Qlik-sense, Tableau

This will help users to see all the relevent tools at one place.

Dead link on projects page

The NBA Data Modeling project on the projects page is a dead link. Can we update that and/or add in new projects to the page?

Concepts/Full Load

https://github.com/JPHaus/data-engineering-wiki/blob/main/Concepts/Full%20Load.md

This note is a seedling and is a great place to make your first contribution!

Link Checker Report

Summary

Status	Count
🔍 Total	1037
✅ Successful	1035
⏳ Timeouts	0
🔀 Redirected	0
👻 Excluded	1
❓ Unknown	0
🚫 Errors	1

Errors per input

Errors in Index.md

• [ERR] https://twitter.com/intent/tweet?text=Check%20out%20this%20awesome%20resource%20for%20data%20engineering!%20https%3A//dataengineering.wiki/ | Failed: Network error: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1921: (certificate has expired)
  Full Github Actions output