samuel-njoroge/data-load-tool

data load tool (dlt)

Data Load Tool (dlt)

dlt - An open-source library added in Python scripts to load data from various and often messy data sources into well-structured, live datasets.

Overview

Installation.

pip install dlt

Run the pipeline.

python pipeline_file_name.py

This is the name of your python file.

Inspect the created dataset using a Streamlit app.

dlt pipeline pipeline_name show

This is the name of your pipeline defined in your python file.

Example : The pipeline african_countries loads data from a defined source to DuckDB.

# Create a pipeline.
pipeline = dlt.pipeline(
    pipeline_name="african_countries",
    destination="duckdb",
    dataset_name="countries" # Database schema "countries".
)