Topic: data-integration Goto Github

👇 Here are 379 public repositories matching this topic...

airbytehq / airbyte
data-integration,The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Organization: airbytehq
Home Page: https://airbyte.com
data pipeline data-analysis data-engineering java python etl change-data-capture data-collection data-integration

apache / airflow
data-integration,Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Organization: apache
Home Page: https://airflow.apache.org/
airflow apache apache-airflow python scheduler workflow automation dag data-engineering data-integration

apache / flink-cdc
data-integration,Flink CDC is a streaming data integration tool
Organization: apache
Home Page: https://nightlies.apache.org/flink/flink-cdc-docs-stable
change-data-capture cdc batch data-integration data-pipeline distributed elt etl flink kafka

apache / hop
data-integration,Hop Orchestration Platform
Organization: apache
Home Page: https://hop.apache.org/
java streaming hop apache data-integration etl orchestration pipeline workflow

apache / hudi
data-integration,Upserts, Deletes And Incremental Processing on Big Data.
Organization: apache
Home Page: https://hudi.apache.org/
hudi apachehudi datalake bigdata apachespark incremental-processing stream-processing data-integration apacheflink

apache / incubator-devlake
data-integration,Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Organization: apache
Home Page: https://devlake.apache.org/
data data-analysis data-engineering data-integration data-transfers devops domain-layer etl golang integration

apache / seatunnel
data-integration,SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
Organization: apache
Home Page: https://seatunnel.apache.org/
data-integration high-performance offline real-time apache batch cdc change-data-capture data-ingestion elt

apache / seatunnel-web
data-integration,SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).
Organization: apache
Home Page: https://seatunnel.apache.org/
apache data-integration data-pipeline etl-framework high-performance offline real-time seatunnel sql-engine

artie-labs / transfer
data-integration,Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.
Organization: artie-labs
Home Page: https://artie.so
snowflake cdc change-data-capture golang bigquery kafka apache-kafka data-integration data-pipelines database

avaiga / taipy
data-integration,Turns Data and AI algorithms into production-ready web applications in no time.
Organization: avaiga
Home Page: https://www.taipy.io
automation data-engineering data-ops data-visualization datascience developer-tools mlops orchestration pipeline pipelines

bruin-data / ingestr
data-integration,ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Organization: bruin-data
Home Page: https://bruin-data.github.io/ingestr/
bigquery copy-database data-ingestion data-integration data-pipeline duckdb ingestion-pipeline mssql postgresql snowflake

bytedance / bitsail
data-integration,BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.
Organization: bytedance
Home Page: https://bytedance.github.io/bitsail/
flink big-data data-integration data-lake data-pipeline data-synchronization high-performance real-time

categoricaldata / cql
data-integration,Categorical Query Language IDE
Organization: categoricaldata
Home Page: http://categoricaldata.net
cql category-theory functional-programming data-migratation etl data-integration data-science data-exchange

ceumicrodata / metl
data-integration,mito ETL tool
Organization: ceumicrodata
python data-integration etl etl-framework pipeline

cloudquery / cloudquery
data-integration,The open source high performance ELT framework powered by Apache Arrow
Organization: cloudquery
Home Page: https://cloudquery.io
aws gcp azure sql data-integration elt etl etl-framework bigquery data-collection

commoncoreontology / commoncoreontologies
data-integration,The Common Core Ontology Repository holds the current released version of the Common Core Ontology suite.
Organization: commoncoreontology
cco applied-ontology bfo semantics ontologies ontology-suite owl-ontology interoperability data-integration semantic-consistency

conduitio / conduit
data-integration,Conduit streams data between data stores. Kafka Connect replacement. No JVM required.
Organization: conduitio
Home Page: https://conduit.io
go data-integration etl data-pipeline data-engineering data-stream conduit kafka kafkaconnect

cuebook / cuelake
data-integration,Use SQL to build ELT pipelines on a data lakehouse.
Organization: cuebook
Home Page: https://cuelake.cuebook.ai
apache-iceberg delta lakehouse datalake data-lake elt etl data-engineering data-integration data-ingestion

dagster-io / dagster
data-integration,An orchestration platform for the development, production, and observation of data assets.
Organization: dagster-io
Home Page: https://dagster.io
data-pipelines dagster workflow data-science workflow-automation python scheduler data-orchestrator etl analytics

dataplane-app / dataplane
data-integration,Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.
Organization: dataplane-app
Home Page: https://dataplane.app
golang airflow kubernetes data scheduler workflows workflow-automation data-pipelines data-science data-engineering

dtstack / chunjun
data-integration,A data integration framework
Organization: dtstack
Home Page: https://dtstack.github.io/chunjun/
flink bigdata data-integration framework java

genular / pandora
data-integration,PANDORA - Predictive Analytics aNd Data Oriented Research Applications :computer:
Organization: genular
Home Page: https://genular.org
machine-learning systems-biology data-science data-visualization data-mining pandora bioinformatics biomarkers clinical-data clustering

google / megalista
data-integration,First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).
Organization: google
google googleads data-integration googleanalytics dataflow python bigquery customermatch conversions audience-targeting

hetio / hetionet
data-integration,Hetionet: an integrative network of disease
Organization: hetio
Home Page: https://neo4j.het.io
hetionet rephetio network neo4j hetnet drug-repurposing data-integration

immunogenomics / harmony
data-integration,Fast, sensitive and accurate integration of single-cell data with Harmony
Organization: immunogenomics
Home Page: https://portals.broadinstitute.org/harmony/
scrna-seq algorithm data-integration r

infinyon / fluvio
data-integration,Lean and mean distributed stream processing system written in rust and web assembly.
Organization: infinyon
Home Page: https://www.fluvio.io/
cloud-native streaming rust real-time serverless stateful stream-processing webassembly data-integration data-flow

jitsucom / jitsu
data-integration,Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days
Organization: jitsucom
Home Page: https://jitsu.com
data-integration clickhouse golang bigquery data-collection data-connectors redshift snowflake postgres

jupyter-naas / drivers
data-integration,Low-code Python library enabling access to APIs, tools, data sources in seconds.
Organization: jupyter-naas
data data-science etl integration lowcode open-source opensource pipeline pipelines data-analysis

kestra-io / kestra
data-integration,Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
Organization: kestra-io
Home Page: https://kestra.io
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration

kuwala-io / kuwala
data-integration,Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times
Organization: kuwala-io
Home Page: https://kuwala.io
data data-integration data-science open-data spatial-analysis elt kuwala open-source scraping dbt

leesf / hudi-resources
data-integration,汇总Apache Hudi相关资料
User: leesf
hudi apachehudi apache hudi-resources datalake bigdata stream-processing incremental-processing data-integration

mage-ai / mage-ai
data-integration,🧙 Build, run, and manage data pipelines for integrating and transforming data.
Organization: mage-ai
Home Page: https://www.mage.ai/
machine-learning artificial-intelligence data data-engineering data-science python elt etl pipelines data-pipelines

mara / mara-example-project-2
data-integration,An example mini data warehouse for python project stats, template for new projects
Organization: mara
etl pypi bigquery data-integration sql

mara / mara-pipelines
data-integration,A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
Organization: mara
etl data-integration python postgresql pipeline data

mims-harvard / scikit-fusion
data-integration,scikit-fusion: Data fusion via collective latent factor models
Organization: mims-harvard
matrix-factorization data-fusion latent-features data-integration embeddings knowledge-graphs

morph-kgc / morph-kgc
data-integration,Powerful RDF Knowledge Graph Generation with RML Mappings
Organization: morph-kgc
Home Page: https://morph-kgc.readthedocs.io
knowledge-graph data-integration rml r2rml rdf database python etl data-engineering rdf-star

olehmberg / winter
data-integration,WInte.r is a Java framework for end-to-end data integration. The WInte.r framework implements well-known methods for data pre-processing, schema matching, identity resolution, data fusion, and result evaluation.
User: olehmberg
data-fusion identity-resolution schema-matching tabular-data data-integration

opensanctions / nomenklatura
data-integration,Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
Organization: opensanctions
data-integration deduplication record-link

recap-build / recap
data-integration,Work with your web service, database, and streaming schemas in a single format.
Organization: recap-build
Home Page: https://recap.build
data-catalog metadata data-discovery recap data-engineering data-integration data-pipelines etl

rudderlabs / rudder-server
data-integration,Privacy and Security focused Segment-alternative, in Golang and React
Organization: rudderlabs
Home Page: https://www.rudderstack.com/
golang hybrid-cloud privacy security warehouse-management data-warehouse rudderstack customer-data customer-data-pipeline customer-data-platform

runprism / prism
data-integration,Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.
Organization: runprism
Home Page: https://runprism.com
data data-engineering data-science etl etl-pipeline machine-learning orchestration pipeline python data-orc

saeyslab / nichenetr
data-integration,NicheNet: predict active ligand-target links between interacting cells
Organization: saeyslab
single-cell-omics intercellular-communication ligand-receptor ligand-target data-integration single-cell-rna-seq rna-seq gene-expression network-inference cell-cell-communication

saezlab / cosmosr
data-integration,COSMOS (Causal Oriented Search of Multi-Omic Space) is a method that integrates phosphoproteomics, transcriptomics, and metabolomics data sets.
Organization: saezlab
Home Page: https://saezlab.github.io/cosmosR/
metabolomic-data proteomics phosphoproteomics data-integration transcriptomics network-modelling

sdm-tib / sdm-rdfizer
data-integration,An Efficient RML-Compliant Engine for Knowledge Graph Construction
Organization: sdm-tib
Home Page: https://doi.org/10.5281/zenodo.3872103
knowledge-graph rml data-integration

seandavi / awesome-single-cell
data-integration,Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.
User: seandavi
rna-seq-data gene-expression scrna-seq-data bioinformatics awesome-list dimensionality-reduction cell-cycle atac-seq analysis cell-differentiation python clustering cell-populations gene-expression-profiles rna-seq-experiments cell-clusters analysis-pipeline data-integration data-visualization single-cell

slowkow / harmonypy
data-integration,🎼 Integrate multiple high-dimensional datasets with fuzzy k-means and locally linear adjustments.
User: slowkow
Home Page: https://portals.broadinstitute.org/harmony/
data-science data-integration bioinformatics single-cell-analysis

sysbiochalmers / gecko
data-integration,Toolbox for including enzyme constraints on a genome-scale model.
Organization: sysbiochalmers
Home Page: http://sysbiochalmers.github.io/GECKO/
proteomics data-integration toolbox matlab kinetics enzyme-constraints systems-biology

teichlab / cellhint
data-integration,A tool for semi-automatic cell type harmonization and integration
Organization: teichlab
cell-type-harmonization data-integration machine-learning python scrna-seq single-cell

thedataengineeringbook / thedataengineeringbook
data-integration,The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Organization: thedataengineeringbook
Home Page: https://thedataengineeringbook.online
data-engineering data hacktoberfest book data-engineer data-pipeline data-integration data-ingestion data-infrastructure

theislab / scarches
data-integration,Reference mapping for single-cell genomics
Organization: theislab
Home Page: https://docs.scarches.org/en/latest/
single-cell deep-learning rna-seq-analysis data-integration batch-correction multimodal-deep-learning multiomics single-cell-genomics scrna-seq human-cell-atlas