Topic: data-engineering Goto Github

👇 Here are 3218 public repositories matching this topic...

adilkhash / data-engineering-howto
data-engineering,A list of useful resources to learn Data Engineering from scratch
User: adilkhash
distributed-systems data-engineering data-pipeline cloud-providers scala

airbytehq / airbyte
data-engineering,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

alexioannides / pyspark-example-project
data-engineering,Implementing best practices for PySpark ETL jobs and applications.
User: alexioannides
data-engineering data-science etl etl-job etl-pipeline pyspark python spark

andkret / cookbook
data-engineering,The Data Engineering Cookbook
User: andkret
Home Page: https://learndataengineering.com/
data-engineer data-engineering big-data best-practices cookbook

apache / airflow
data-engineering,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / incubator-devlake
data-engineering,Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Organization: apache
Home Page: https://devlake.apache.org/
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration

apache / superset
data-engineering,Apache Superset is a Data Visualization and Data Exploration Platform
Organization: apache
Home Page: https://superset.apache.org/
analytics apache apache-superset asf bi business-analytics business-intelligence data-analysis data-analytics data-engineering data-science data-visualization data-viz flask python react sql-editor superset

argoproj / argo-workflows
data-engineering,Workflow Engine for Kubernetes
Organization: argoproj
Home Page: https://argo-workflows.readthedocs.io/
airflow argo argo-workflows batch-processing cloud-native cncf dag data-engineering gitops hacktoberfest k8s knative kubernetes machine-learning mlops pipelines workflow workflow-engine

avaiga / taipy
data-engineering,Turns Data and AI algorithms into production-ready web applications in no time.
Organization: avaiga
Home Page: https://www.taipy.io
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines

aws / aws-sdk-pandas
data-engineering,pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Organization: aws
Home Page: https://aws-sdk-pandas.readthedocs.io
python aws pandas apache-arrow apache-parquet data-engineering etl data-science redshift athena

bytewax / bytewax
data-engineering,Python Stream Processing
Organization: bytewax
Home Page: https://docs.bytewax.io/
python stream-processing rust data-engineering data-processing data-science dataflow machine-learning streaming-data

cloudquery / cloudquery
data-engineering,The open source high performance ELT framework powered by Apache Arrow
Organization: cloudquery
Home Page: https://cloudquery.io
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection

dagster-io / dagster
data-engineering,An orchestration platform for the development, production, and observation of data assets.
Organization: dagster-io
Home Page: https://dagster.io
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

dagworks-inc / hamilton
data-engineering,Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.
Organization: dagworks-inc
Home Page: https://hamilton.dagworks.io/en/latest/
data-science python dag data-engineering dataframe etl etl-framework etl-pipeline feature-engineering featurization

datafold / data-diff
data-engineering,Compare tables within or across databases
Organization: datafold
Home Page: https://docs.datafold.com
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring

datastacktv / data-engineer-roadmap
data-engineering,Roadmap to becoming a data engineer in 2021
Organization: datastacktv
Home Page: https://datastack.tv
data-engineer-roadmap data-engineering cloud roadmap

datatalksclub / data-engineering-zoomcamp
data-engineering,Free Data Engineering course!
Organization: datatalksclub
data-engineering kafka spark dbt docker prefect

dlt-hub / dlt
data-engineering,data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
Organization: dlt-hub
Home Page: https://dlthub.com/docs
data python data-engineering data-lake data-loading data-warehouse elt extract load transform

eugeneyan / applied-ml
data-engineering,📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
User: eugeneyan
applied-machine-learning production applied-data-science machine-learning data-science reinforcement-learning data-engineering recsys search deep-learning

eventual-inc / daft
data-engineering,Distributed DataFrame for Python designed for the cloud, powered by Rust
Organization: eventual-inc
Home Page: https://getdaft.io
big-data data-engineering data-science dataframe distributed-computing machine-learning python rust

evidence-dev / evidence
data-engineering,Business intelligence as code: build fast, interactive data visualizations in pure SQL and markdown
Organization: evidence-dev
Home Page: https://evidence.dev
analytics sql business-intelligence data-visualization dbt duckdb tailwindcss self-hosted open-source svelte

feast-dev / feast
data-engineering,The Open Source Feature Store for Machine Learning
Organization: feast-dev
Home Page: https://feast.dev
big-data data-engineering data-quality data-science feature-store features machine-learning ml mlops python

feathr-ai / feathr
data-engineering,Feathr – A scalable, unified data and AI engineering platform for enterprise
Organization: feathr-ai
Home Page: https://join.slack.com/t/feathrai/shared_invite/zt-1ffva5u6v-voq0Us7bbKAw873cEzHOSg
apache-spark artificial-intelligence azure data-engineering data-quality data-science feature-engineering feature-governance feature-management feature-marketplace feature-metadata feature-platform feature-store machine-learning mlops

gokumohandas / made-with-ml
data-engineering,Learn how to design, develop, deploy and iterate on production-grade ML applications.
User: gokumohandas
Home Page: https://madewithml.com
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml llms ray distributed-training

gokumohandas / mlops-course
data-engineering,Learn how to design, develop, deploy and iterate on production-grade ML applications.
User: gokumohandas
Home Page: https://madewithml.com
machine-learning deep-learning pytorch mlops data-engineering data-quality data-science distributed-ml llms natural-language-processing python ray

great-expectations / great_expectations
data-engineering,Always know what to expect from your data.
Organization: great-expectations
Home Page: https://docs.greatexpectations.io/
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest

growthbook / growthbook
data-engineering,Open Source Feature Flagging and A/B Testing Platform
Organization: growthbook
Home Page: https://www.growthbook.io
abtesting statistics abtest experimentation split-testing mixpanel snowflake bigquery redshift clickhouse

gunnarmorling / awesome-opensource-data-engineering
data-engineering,An Awesome List of Open-Source Data Engineering Projects
User: gunnarmorling
awesome-list data-engineering

hemansnation / god-level-ai
data-engineering,A collection of scientific methods, processes, algorithms, and systems to build stories & models.
User: hemansnation
Home Page: https://www.himanshuramchandani.co/
python datastructures numpy pandas matplotlib scikit-learn machine-learning deep-learning pytorch data-science

jqnatividad / qsv
data-engineering,CSVs sliced, diced & analyzed.
User: jqnatividad
csv data-wrangling tsv cli opendata data-engineering ckan excel luau datapackage

kantord / just-dashboard
data-engineering,:bar_chart: :clipboard: Dashboards using YAML or JSON files
User: kantord
Home Page: https://kantord.github.io/just-dashboard/
data-science data-visualization dashboard data-engineering d3 d3js chart data yaml csv

kestra-io / kestra
data-engineering,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
data data-engineering data-integration data-orchestration data-orchestrator data-pipeline data-quality elt etl low-code orchestration pipeline reverse-etl scheduler workflow workflow-engine

mage-ai / mage-ai
data-engineering,🧙 Build, run, and manage data pipelines for integrating and transforming data.
Organization: mage-ai
Home Page: https://www.mage.ai/
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

meltano / meltano
data-engineering,Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.
Organization: meltano
Home Page: https://meltano.com/
dataops dataops-platform elt open-source opensource data pipelines extract-data meltano meltano-sdk

metarank / metarank
data-engineering,A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine
Organization: metarank
Home Page: https://metarank.ai
ranking scala search personalization machine-learning deep-learning data-engineering feature-engineering feature-extraction kubernetes

mlrun / mlrun
data-engineering,MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
Organization: mlrun
Home Page: https://mlrun.org
mlops python data-science machine-learning data-engineering experiment-tracking model-serving mlops-workflow workflow kubernetes

moataz-elmesmary / data-science-roadmap
data-engineering,Data Science Roadmap from A to Z
User: moataz-elmesmary
Home Page: https://www.linkedin.com/posts/moatazelmesmary_github-moataz-elmesmarydata-science-roadmap-activity-6974293994960769024-yCdi?utm_source=share&utm_medium=member_desktop
data-analysis data-engineering data-science data-visualization deep-learning machine-learning mathematics probability python sql

ploomber / ploomber
data-engineering,The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
Organization: ploomber
Home Page: https://docs.ploomber.io
workflow machine-learning data-science data-engineering mlops papermill jupyter jupyter-notebooks pipelines vscode pycharm notebooks

prefecthq / prefect
data-engineering,Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Organization: prefecthq
Home Page: https://prefect.io
python workflow data-engineering data-science workflow-engine prefect infrastructure ml-ops data-ops automation

quadratichq / quadratic
data-engineering,Quadratic | Technical Spreadsheet with Python, SQL, and AI
Organization: quadratichq
Home Page: https://QuadraticHQ.com
python data-science spreadsheet sql quadratic etl data-engineering data-analysis data wasm

redpanda-data / connect
data-engineering,Fancy stream processing made operationally mundane
Organization: redpanda-data
Home Page: https://docs.redpanda.com/redpanda-connect/about/
message-queue stream-processing streaming-data message-bus logs stream-processor cqrs event-sourcing go golang

risingwavelabs / risingwave
data-engineering,SQL stream processing, analytics, and management. We decouple storage and compute to offer efficient joins, instant failover, dynamic scaling, speedy bootstrapping, and concurrent query serving.
Organization: risingwavelabs
Home Page: https://www.risingwave.com/slack
database stream-processing cloud-native sql distributed-database rust serverless postgresql real-time postgres

rudderlabs / rudder-server
data-engineering,Privacy and Security focused Segment-alternative, in Golang and React
Organization: rudderlabs
Home Page: https://www.rudderstack.com/
privacy warehouse-management data-warehouse customer-data customer-data-pipeline customer-data-platform customer-data-lake segment-alternative data-integration data-synchronization

running-elephant / datart
data-engineering,Datart is a next generation Data Visualization Open Platform
Organization: running-elephant
Home Page: https://running-elephant.github.io/datart-docs/
analytics bi business-analytics business-intelligence chart d3 dashboard data-analysis data-analytics data-engineering data-visualization data-viz datart davinci display echarts react report sql-editor typescript

san089 / udacity-data-engineering-projects
data-engineering,Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.
User: san089
data data-engineering data-engineering-pipeline etl-pipeline cassandra-database postgresql-database data-modeling data-warehouse data-lake airflow

sodadata / soda-core
data-engineering,:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
Organization: sodadata
Home Page: https://go.soda.io/core-docs
python data-engineering data-governance data-monitoring data-observability data-profiling data-quality data-quality-checks data-quality-monitoring data-reliability

superstreamlabs / memphis
data-engineering,Memphis.dev is a highly scalable and effortless data streaming platform
Organization: superstreamlabs
Home Page: https://docs.memphis.dev
data data-stream-processing data-streaming kubernetes messaging-queue data-engineering data-pipeline golang enrichment message-broker message-bus message-queue microservices schema-registry

treeverse / lakefs
data-engineering,lakeFS - Data version control for your data lake | Git for data
Organization: treeverse
Home Page: https://docs.lakefs.io
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang

whoiskatrin / sql-translator
data-engineering,SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.
User: whoiskatrin
Home Page: https://www.sqltranslate.app/
data-analysis data-engineering dataquery datascience dataset openai postgresql query sql

xonsh / xonsh
data-engineering,:shell: Python-powered shell. Full-featured and cross-platform.
Organization: xonsh
Home Page: http://xon.sh
artificial-intelligence bash chatgpt cli command-line console data-engineering data-science devops fish hacktoberfest iterm2 python raspberry-pi security-automation shell windows-terminal xonsh zsh