Better ETL like "beatle"
An intial take on writing the skeleton of a small "Extract-Transform-Load" (ETL) application that is able to:
- Retrieve data from various locations using standard custom workflows
- Perform some amount of processing on that data
- Load the processed data to various locations using standard custom workflows
Our interfaces for defining the workflows of extraction, transforming, and loading data are:
Extractor
Transformer
Loader
Writing a new ETL job will require the following:
- A config file specifying what kind of extraction and loader processes will be used
- A callable (e.g function, method, etc.) that will accept all the extracted datasources and return one or more datasets
We'll interact with our tooling using a simple command line interface (CLI)
betl -j <name-of-job>
The configuration file will define the kind of extraction process used and the names of the datasets at each step in the process
extract:
iris:
type: CsvExtractor
location: https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv
load:
processed_data:
type: CsvLoader
location: output/iris_melt.csv
The dataset names (e.g. "iris" above) will be the argument names passed to the next step of an ETL job.
So the related Transformer
of this ETL job should be ready to accept iris
as an argument.
Data being passed between steps should one of the following:
Path
Iterable[Path]
pandas.DataFrame
Iterable[pandas.DataFrame]
This supports common ways of processing data either from files or loading data into memory immediately using pandas
.
Using Iterables
supports processing data in a pipeline where all extracted data does not need to be loaded into memory at once.