manuzhang/awesome-lakehouse

a curated list of awesome lakehouse frameworks, applications, etc

awesome-lakehouse

a curated list of awesome Lakehouse frameworks, applications, etc

Table of Contents

• Table Format
• Lakehouse System
• Metadata Service
• Machine Learning
• Benchmark

Table Format

• Apache Iceberg [Java] - a high-performance format for huge analytic tables, bringing the reliability and simplicity of SQL tables to big data.
• Apache Hudi [Java] - a transactional data lake platform that brings database and data warehouse capabilities to the data lake.
• Apache Paimon (incubating) [Java] - a streaming data lake platform with high-speed data ingestion, changelog tracking and efficient real-time analytics.
• Delta [Scala] - an open-source storage framework that enables building a Lakehouse architecture with various compute engines and languages.
• OneTable [Java] - an omni-directional converter for table formats that facilitates interoperability across data processing systems and query engines.

Lakehouse System

• Amoro [Java] - a management system built on open data lake formats, bringing pluggable and self-managed features for Lakehouse.
• GeoLake [Java] - Universal solution for geospatial data tailored to data lakehouse systems.
• LakeSoul [Rust] - a cloud-native Lakehouse framework that supports scalable metadata management, ACID transactions, efficient and flexible upsert operation, schema evolution, and unified streaming & batch processing.
• Lakehouse Engine [Python] - a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
• Smart Data Lake [Scala] - a data lake automation framework that makes loading and transforming data a breeze.

Metadata Service

• DeltaCAT [Python] - a Pythonic Data Catalog powered by Ray.
• Gravitino [Java] - a high-performance, geo-distributed, and federated metadata lake.
• lakeFS [Go] - data version control for data lake.
• Metacat [Java] - a unified metadata exploration API service.
• Nessie [Java] - a Transactional Catalog for Data Lakes with Git-like semantics.
• OpenHouse [Java] - an open source control plane designed for efficient management of tables within open data lakehouse deployments.

Machine Learning

• Space [Python] - Unified storage framework for the entire machine learning lifecycle.

Benchmark

• LHBench [Scala] - a benchmark for Lakehouse storage systems.
• LST-Bench [Java] - a framework that allows users to run benchmarks specifically designed for evaluating the performance, efficiency, and stability of Log-Structured Tables (LSTs).