dyvenia/elt_workshop

ELT workshop materials.

dyvenia

Public dyvenia content

Use https://diataxis.fr/ as a reference

and MK Docs as help for the structure

Build the Docs with mkdocs

Poetry

First you will need to create a python virtual environment. Make sure to install poetry pip install poetry and then run inside the kb folder:

cd kb
poetry install

Build your docs

The above will create a python environment for you with your required dependencies to build these docs. Once you have your virtual environment you can test it by serving these docs doing:

cd on_docs
poetry run mkdocs serve

If everything worked correctly you should be able to see the docs site at http://127.0.0.1:8000.

elt_workshop's Issues

create set up guide

README.md done for Docker, VSCode, Git, Prefect, and Airbyte set up

TODO

ELT workshops - TODO

0. Intro

• #7

1. Set up

• #15
  • extensions: Python, Python Docstring Generator, Live Share
• #14
• #13
• #11
• #12

2. Part 1 - Prefect Intro

• #2
• #9

3. Part 2 - Build a pipeline - existing Airbyte source

Build a full pipeline, eg. API source -> local filesystem -> Metabase.

• #6
  Idea for now: source from Strava using Airbyte, sink to Postgres, use DBT to transform, and set up a simple viz in Metabase.
  See https://github.com/desertaxle/airbyte-prefect-recipe for Prefect + Airbyte + dbt.
  See https://airbyte.io/recipes/build-a-github-activity-dashboard-for-your-project for Airbyte + Metabase.
• #18

4. Part 3 - Build a pipeline - new Airbyte source

Create a new API source and build a full pipeline using it, eg. New API source -> local filesystem -> Metabase using.

• #5
• #10

5. General

• #16
• #4
• #3
• #8
• #17

6. Next iterations

• add PR template
• add mkdocs
• add CI/CD
• write a cleanup script
• add prod tips
• VSCode set up script (settings, extensions)
• add part 4 - "Build a library for self-service analytics"
• use custom app to enter data and serve API
• consider Dagster

create intro presentation

• write summary in text
• create presentation

create set up script

test VSCode Live with another person

• tested once, looks decent, although the set up was a bit messy (took ~10 minutes to get it running due to freezes and permission errors)
• repeat this process at least once more to see whether it's always this ugly or it was just my environment

new source

Ideas

Yahoo Finance API

• stock price comparison
• crypto price (eg. BTC)

World bank:

• inflation
• GDP per year per country vs eg. inflation - filter by country

test set up on another machine

build a diagram with the process

Idea for now: source from Strava using Airbyte, sink to Postgres, use DBT to transform, and set up a simple viz in Metabase.See https://github.com/desertaxle/airbyte-prefect-recipe for Prefect + Airbyte + dbt.See https://airbyte.io/recipes/build-a-github-activity-dashboard-for-your-project for Airbyte + Metabase.

create notebook code - existing source

• create a flow using an existing source - to local JSON
• create a flow using an existing source - full pipeline

add VSCode set up info to README

• extensions
  - [ ] settings # next iteration

choose Superset / Metabase

create notebook code - intro

• Hello, world flow - local
• Hello, world flow - sbx
• Hello, world flow - dev

platform v2

Example ideas:

• change Postgres to Google Cloud Storage
• add Great Expectations
• add dbt

check if can replace Airbyte with viadot

create general presentation

This will be used to explain concepts during the workshop

• create Prefect execution diagram
• create Airbyte diagram - pending #8
• create Metabase diagram

create notebook instructions - intro

Theory + practice + understanding + exercise + solution.

• Hello, world flow - local
• Hello, world flow - sbx
• Hello, world flow - dev

create notebook instructions - existing source

Existing

• {API_SOURCE} to JSON - local
• {API_SOURCE} to Metabase local
• {API_SOURCE} to Metabase - sbx

clean up envs

Eg. Prefect, GitHub - remove test flows/solutions etc.

add Great Expectations check task