mybi-dbt-core is dbt module to quick start transformations on top of data synced by myBI Connect along with multiple useful macros.
๐ Get experience with myBI Market + dbt demo project
New to dbt packages? Read more about them here.
- Include this package in your
packages.yml
file - Run
dbt deps
to install the package
Reference any data source from sources.yml as select * from source('general', 'dates')
:
Access data tables as simple as select * from ref('stg_yd_ads_facts')
:
- filters on
account_id
applied - joined with date dimension
- comprehensive data testing (
unique
,not_null
,relationships
)
Supported data sources:
general
โ Generalmetrika
โ Yandex.Metrikadirect
โ Yandex.Directads
โ Google Adsmytarget
โ myTargetbitrix24
โ Bitrix24amocrm
โ AmoCRMga
โ Google Analyticscurrency
โ Currency exchange ratesvkontakte
โ Vkontakte
- All the macros and tests from dbt_utils package are available
- source_filter_rows macro to limit rows for specific accounts, dev/test environments
- (WIP) clean_up
Get experience with worked example of this module usage:
Development is done via Docker containers and .csv mock files. Refer to integration_tests folder.
Any PR will be tested with Continuous Integration workflow
Development workflow looks like this:
# launch containers: dbt, clickhouse, postgres
docker-compose build --no-cache
docker-compose up -d
# enter container with dbt installed
docker-compose exec dbt bash
cd integration_tests/
# test connections
dbt --version
dbt debug --target clickhouse
dbt debug --target postgres
# Setup target variables
export DBT_SOURCE_DATABASE=default DBT_SOURCE_SCHEMA=default # for Clickhouse
export DBT_SOURCE_DATABASE=postgres DBT_SOURCE_SCHEMA=public # for Postgres
# introduce any code changes
# install dependencies (modules)
dbt clean
dbt deps
# build and test on dummy data
dbt seed --full-refresh
dbt build --full-refresh
# exit container and clean up
docker-compose down