The mabel from fossabot

mabel is a Data Engineering platform designed to run in serverless environments.

mabel just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.

Documentation GitHub Wiki
Bug Reports GitHub Issues
Feature Requests GitHub Issues
Source Code GitHub
Discussions GitHub Discussions

Focus on What Matters

We've built mabel to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.

from mabel import Reader

data = Reader(dataset="test_data")
print(data.count())

Key Features

On-the-fly compression
Low-memory requirements, even with terabytes of data
Indexing and partitioning of data for fast reads
Cursors for tracking reading position between processes
Partial SQL DQL (Data Query Language) support
Schema and data_expectations validation

Installation

From PyPI (recommended)

pip install --upgrade mabel

From GitHub

pip install --upgrade git+https://github.com/mabel-dev/mabel

A preview release of mabel is available from PyPI

pip install --upgrade mabelbeta

You may need to manually uninstall mabel before the test version will install.

These versions are usually labelled with an a (signifying alpha status) in the library version. Alpha versions are more likely to have functional issues.

Guides

How to Read Data

Dependencies

orjson for JSON (de)serialization

bitarray for handling high density boolean data

siphashc for non-cryptographic hashing

pydantic to define internal data models

zstandard for real-time on disk compression

LZ4 for real-time in memory compression

simdjson for fast JSON deserialization

cython for precompilation

There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in tests/requirements.txt.