Topic: dataengineering Goto Github

👇 Here are 582 public repositories matching this topic...

514-labs / moose
dataengineering,The developer framework for your data & analytics stack
Organization: 514-labs
Home Page: https://www.moosejs.com
data dataengineering deployment framework insights metrics analytics python rust typescript

aakashnand / trino-ranger-demo
dataengineering,Tutorial on how to setup Trino and Apache Ranger using docker
User: aakashnand
docker ranger trino dataengineering datagovernance tutorial

abhishek-ch / data-machinelearning-the-boring-way
dataengineering,Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.
User: abhishek-ch
data-infrastructure dataengineering datascience kubernetes machine-learning mlops

airscholar / realtimestreamingengineering
dataengineering,This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka and Elasticsearch. It covers each stage from data acquisition, processing, sentiment analysis with ChatGPT, production to kafka topic and connection to elasticsearch.
User: airscholar
Home Page: https://www.youtube.com/watch?v=ETdyFfYZaqU
apache-spark chatgpt dataengineering elasticsearch kafka openai-api tcp-socket

anuran-roy / serpytor
dataengineering,A distributed, low-code, end-to-end data collection and analysis tool for data folks. Take the pain out of data collection from your pipeline!
User: anuran-roy
Home Page: https://anuran-roy.github.io/serpytor
data dataengineering datascience low-code lowcode pipeline python python3 distributed-computing distributed-systems

awslabs / aws-ddk
dataengineering,An open source development framework to help you build data workflows and modern data architecture on AWS.
Organization: awslabs
Home Page: https://awslabs.github.io/aws-ddk/
aws dataops dataengineering python

awslabs / aws-orbit-workbench
dataengineering,A Data Platform built for AWS, powered by Kubernetes.
Organization: awslabs
Home Page: https://awslabs.github.io/aws-orbit-workbench/
kubernetes eks jupyter jupyterhub data-analysis datalake mach orbit-workbench gpu eks-cluster

balajirvp / de-zoomcamp
dataengineering,Code/Notes for the Data Engineering Zoomcamp by DataTalksClub
User: balajirvp
batchprocessing bigquery dataengineering dbt docker gcp prefect streaming pyspark python rdd spark sql terraform datalake datamodeling datawarehouse

chollinger93 / bridgefour
dataengineering,Bridge Four is a simple, functional, effectful, single-leader, multi worker, distributed compute system optimized for embarrassingly parallel workloads.
User: chollinger93
Home Page: https://chollinger.com/blog/2023/06/building-a-functional-effectful-distributed-system-from-scratch-in-scala-3-just-to-avoid-leetcode-part-1/
data dataengineering distributed-systems scala

cynthiakoopman / forecasting-solar-energy
dataengineering,Forecasting Solar Power: Analysis of using a LSTM Neural Network
User: cynthiakoopman
deeplearning machinelearning forecasting lstm neuralnetworks data-science dataengineering solarpower solar-energy renewable-energy

daihuynh / dagster_dbt_metabase_simple_solution
dataengineering,A simple Data Engineering solution for testing or education purposes. You only need to know SQL and Python to understand this project. Dagster is the data orchestration, while DBT is for data transformation. The BI tool is Metabase.
User: daihuynh
dagster dataengineering dbt metabase sqlite

danielsaban / data-scraping-sofascore
dataengineering,Data Engineering/Scraping Project. Creating a detailed Sports Relational Database for the Top European Soccer Leagues.
User: danielsaban
dataengineering datamining webscraping relational-databases parser etl-pipeline rest-api

data-burst / data-engineering-roadmap
dataengineering,
Organization: data-burst
Home Page: https://databurst.tech
data-engineering dataengineering roadmap data-engineering-roadmap

dataexpert-io / data-engineer-handbook
dataengineering,This is a repo with links to everything you'd ever want to learn about data engineering
Organization: dataexpert-io
apachespark awesome bigdata data dataengineering sql

datafold / data-diff
dataengineering,Compare tables within or across databases
Organization: datafold
Home Page: https://docs.datafold.com
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring databricks-sql dataengineering dataquality oracle-database postgres sql dbt python data data-diffing

datavault-uk / automate-dv
dataengineering,A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)
Organization: datavault-uk
Home Page: https://www.automate-dv.com
datavault20 dbt datawarehousing datavault etl snowflake dataengineering datawarehouse datalake elt

eldar1205 / awesome-python-backend
dataengineering,Index for online reading materials in order to learn Python and backend development/engineering concepts from scratch and develop a mastery sufficient for Senior/Principal Backend Engineers and Data Engineers
User: eldar1205
backend backendengineering dataengineering python

erdemozgen / data-engineering-roadmap
dataengineering,Roadmap for Data Engineering
User: erdemozgen
awesome ci-cd cloud data-science dataengineering development devops machine-learning mlops roadmap

finance-and-ml / us-stock-prediction-using-ml-and-spark
dataengineering,Predict stock price based on financial news feeds
Organization: finance-and-ml
stock-price-prediction spark scala tableau webscraping dataengineering

franloza / coches-net-dashboard
dataengineering,Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market
User: franloza
dagster data dataengineering duckdb elt pipeline scraping visualization

grai-io / grai-core
dataengineering,
Organization: grai-io
Home Page: https://www.grai.io
hacktoberfest data datalineage dbt open-source postgresql python mssql mysql snowflake

josephmachado / beginner_de_project_stream
dataengineering,Simple stream processing pipeline
User: josephmachado
Home Page: https://www.startdataengineering.com/post/data-engineering-project-for-beginners-stream-edition/
apache-flink dataengineering datapipeline postgresql graphana prometheus

josephmachado / e2e_datapipeline_test
dataengineering,Example repo to create end to end tests for data pipeline.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/setting-up-e2e-tests/
aws dataengineering moto pytest python3 testing

josephmachado / socialetl
dataengineering,Project for "Data pipeline design patterns" blog.
User: josephmachado
Home Page: https://www.startdataengineering.com/post/code-patterns/
dataengineering design-patterns etl-pipeline makefile python social-media-data sqllite3 reddit

judeleonard / prescriber-etl-data-pipeline
dataengineering,An End-to-End ETL data pipeline that leverages pyspark parallel processing to process about 25 million rows of data coming from a SaaS application using Apache Airflow as an orchestration tool and various data warehouse technologies and finally using Apache Superset to connect to DWH for generating BI dashboards for weekly reports
User: judeleonard
Home Page: https://judeleonard.github.io/Prescriber-ETL-data-pipeline/
airflow apache-superset bi-dashboards dataengineering datawarehouse etl-pipeline pyspark airflow-docker amazon-s3 azure-blob-storage

kevinheavey / modern-polars
dataengineering,Code and data for the Modern Polars book
User: kevinheavey
Home Page: https://kevinheavey.github.io/modern-polars/
data-analytics data-engineering data-science dataengineering pandas polars python

kirilldikalin / myknowledge
dataengineering,Все, о чем меня когда-либо спрашивали на собеседованиях, и другие полезные знания в кратком формате
User: kirilldikalin
Home Page: https://kirilldikalin.github.io/kirilldikalin.io/knowlege_base/iKnowledge_base.html
data-science python dataengineering sql

kislerdm / data-engineering-interviews
dataengineering,Data engineering interviews Q&A for data community by data community
User: kislerdm
Home Page: https://data-engineering-interviews.org
dataengineering interview-questions opensource python sql kafka spark linux

marcosmjd / ghcn-d
dataengineering,Data Pipeline from the Global Historical Climatology Network DataSet
User: marcosmjd
airflow bigquery dataengineering elt etl dbt googlecloudstorage googledatastudio iac terraform

mehd-io / pypi-duck-flow
dataengineering,end-to-end data engineering project to get insights from PyPi using python, duckdb, MotherDuck & Evidence
User: mehd-io
Home Page: https://duckdbstats.com/
dataengineering duckdb etl python

mikestack15 / orangutan-stem
dataengineering,An open-source project dedicated to constructing robust data pipelines and scalable software infrastructure. We leverage industry-standard tools favored by developers to enhance efficiency and reliability. Uniquely, these pipelines are field-tested on farms across Sumatra, Indonesia, ensuring real-world applicability and resilience.
User: mikestack15
Home Page: https://orangutan-stem.com
dataengineering portfolio stem airflow-docker computer-vision django-application software-engineering django-docker traefik ultralytics

minhadona / data_engineer_interview_challenges
dataengineering,Found a data engineering challenge or participated in a selection process ? Share with us!
User: minhadona
beginner beginner-friendly beginner-project beginners code-challenge-practice code-challenges data-engineering data-pipeline data-pipelines dataengineering

noahgift / data-engineering-and-dataops
dataengineering,Duke MIDS: Data Engineering and DataOps Course
User: noahgift
Home Page: https://noahgift.github.io/data-engineering-and-dataops/
duke cloud dataops dataengineering data data-science software-engineering mlops book course

ogbinar / dataengineeringpilipinas
dataengineering,Data Engineering Pilipinas is a community for data engineers, data analysts, data scientists, developers, AI / ML engineers, and users of closed and open source data tools and methods / techniques in the Philippines. Data Engineering Pilipinas is a PyData group.
User: ogbinar
Home Page: https://dataengineering.ph
community data database dataengineering datascience datasets engineering etl python sql

olist / work-at-olist-data
dataengineering,Apply for a job at Olist's Data Team: https://olist.gupy.io/
Organization: olist
Home Page: https://olist.gupy.io/
analytics data dataengineering datascience dataset julia machinelearning pandas python r sql

open-metadata / openmetadata
dataengineering,OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
Organization: open-metadata
Home Page: https://open-metadata.org
metadata datadiscovery data-science dataquality data-profiling metadata-management dataengineering data-catalog data-observability dbt

prodmodel / prodmodel
dataengineering,Build, test, deploy, iterate - Dev and prod tool for data science pipelines
Organization: prodmodel
build-tool build-automation build-system data-science data datascience production productivity dataengineering dataeng

sarahmk125 / airflow-docker-metrics
dataengineering,
User: sarahmk125
airflow-containers airflow-metrics data dataengineering grafana orchestration statsd-format statsd-metrics workflow-orchestration

sbalnojan / run-a-data-team
dataengineering,A guide for leading a data (engineering) team
User: sbalnojan
Home Page: https://www.thdpth.com/
analytics awesome-list data dataengineering datascience

snowflake-labs / snowpark-python-demos
dataengineering,This repository provides various demos/examples of using Snowpark for Python.
Organization: snowflake-labs
python snowpark dataengineering datascience machine-learning

sparsh-ai / recohut
dataengineering,Recohut - Learn data engineering, data science
User: sparsh-ai
Home Page: https://www.recohut.in/docs/introduction
data-engineering dataengineering datascience

spratiher9 / sparkdataset
dataengineering,Instant search for and access to many datasets in Pyspark.
User: spratiher9
Home Page: https://pypi.org/project/sparkdataset/
standard datasets pyspark python3 python spark easy-access-application instantsearch r benchmark-framework

tirendazacademy / awesome-data-science-resources
dataengineering,Resources about data science, machine learning, deep learning, data engineering, and SQL.
User: tirendazacademy
ai artificial-intelligence data-analysis data-engineering data-science dataengineering datascience deep-learning deeplearning machine-learning machinelearning machinelearning-python sql

tobikodata / sqlmesh
dataengineering,Efficient data transformation and modeling framework that is backwards compatible with dbt.
Organization: tobikodata
Home Page: https://sqlmesh.com
dataops elt etl sql python dataengineering transformation dbt

waylonwalker / kedro-static-viz
dataengineering,kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
User: waylonwalker
Home Page: https://static-viz.kedro.dev
data datapipeline dataengineering kedro python kedro-plugin

wittline / apache-spark-docker
dataengineering,Dockerizing an Apache Spark Standalone Cluster
User: wittline
Home Page: https://wittline.github.io/apache-spark-docker/
pyspark apache-spark docker-compose hive-metastore hive hdfs hue dataengineering dataengineer hadoop-cluster

wittline / data-engineer-challenge
dataengineering,Challenge Data Engineer
User: wittline
Home Page: https://wittline.github.io/data-engineer-challenge/
data-engineering data-pipeline dataengineering docker docker-compose fastapi postgresql

wittline / pydag
dataengineering,Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag
User: wittline
Home Page: https://wittline.github.io/pyDag/
big-data bigquery cloud dag data-engineering data-pipeline dataengineering dataproc dataproc-cluster directed-acyclic-graph

wittline / pyspark-on-aws-emr
dataengineering,The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.
User: wittline
Home Page: https://wittline.github.io/pyspark-on-aws-emr/
aws emr-cluster aws-emr python spark pyspark big-data-analytics big-data dataengineering wordcloud-generator

zinggai / zingg
dataengineering,Scalable identity resolution, entity resolution, data mastering and deduplication using ML
Organization: zinggai
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution