mini-kep/intro

Why another database for macroeconomic data?

intro's Issues

discussion: dataset rules/end user API

Please let me know what part of writing is not clear yet and ask your questions (English or Russian):
Пожалуйста напишите какая часть описания вызывает вопросы:

https://github.com/mini-kep/intro/blob/master/pipeline/older_versions/datamodel_and_api.md

This will help me streamline discussion based on your questions, which whould be a good way to go. Many thanks!

migration: move parsers to github org

Existing parser list: https://github.com/mini-kep/db/#parser-results

flask_jsondash dashboards (MWE needed)

Need to evaluate https://github.com/christabor/flask_jsondash:

• can in be run with existing application developped at https://github.com/mini-kep/db and deloyed at heroku?
• can ve plot a time series like one here?

TODO: is to take make min working example of flask_jsondash with some data from minikep-db

intro: collect Travis and Codecov status from repos, make a markdown table

• https://github.com/mini-kep/parser-rosstat-kep
• https://github.com/mini-kep/frontend-app

better organisation for specifications

https://github.com/mini-kep/intro/tree/master/specification has spefication drafts.

Concerns are:

• make specifications easier to update
• write less in repo READMEs, example
• eliminate duplication

@zarak, can you please take a look?

reorganise documentation

idea: promote a poll about machine readable data sources in economics

Do you write code that imports macroeconomic time series data?

• all the time
• once in a while
• I don't but people around do a lot
• I stay away from it
• I knew how

Do you feel there is more better data available to work with?

• yes
• no

Do you use Excel as much as 3 years ago?

• even more
• just the same
• I allocate a bit more time to other software
• Excel is history now

Do you write code that imports macroeconomic time series data?

• all the time
• once in a while
• I don't but people around do a lot
• I stay away from it

What public sources do you use?

• FRED
• Quandl
• OECD API
• IMF API
• World bank API
• national statistics agency any
• oh, here is a cool link: __________________________

How do you import data?

• R
• python pandas
• econometrics package
• Excel plugin

What is the nature of work you do macroeconomic data"

• student assignments / excercises
• descriptive statistics / visualisation
• modelling/forecasting

Your affiliation:

• university
• central bank or government agency
• bank or financial sector
• other corporate sector
• non-profit
• other

open for comments: data pipeline demo

This file mimics the data pipline from parsers to end-user API:

1. parsers make data from different sources available
2. database stores the data
3. standard (REST) API returns query results
4. custom end-user API returns json readable by pd.read_json()

Questions and comments are highly appreciated.

additional text about parsers

Definitions:

• 'source' is a file or API backend published on the web by Rosstat or other agency
  • 'clean source' is something we can truct for its structure, usually an API
  • 'messy source' is something that changes once in a while, eg Word files
• 'scrapper' is a program that downloads the data without transforming it (download files, unpack from zip/rar)
• 'parser' is a program that reads raw data and makes 'processed output'
• 'processed output' is canonical result of parsing, importable to production database

In our pipleine:

• Scrapper loads Source to Raw Database
• Parser reads Source from Raw Database layer and produces Processed Output
• Processed Output is imported into Production Database

Cleaner scrappers/parsers, adressing APIs or XML:

• ust: https://github.com/epogrebnyak/ust
• brent: https://github.com/epogrebnyak/data-fx-oil/blob/master/brent.py
• usd_cbr http://www.moex.com/ru/markets/currency/fixing.aspx?code=USDFIX
• usd_moex_fix, no scrapper yet, XML file - http://www.moex.com/ru/markets/currency/fixing.aspx?code=USDFIX
• ecb_usd_

Messy parsers for Rosstat:

• 'kep': converts Word to csv, parses csv to extract data https://github.com/epogrebnyak/mini-kep
• 'sep': no parser yet, Word files in html tree: http://www.gks.ru/bgd/free/B17_00/Main.htm
• '806-regional': parse xls files https://github.com/epogrebnyak/data-rosstat-806-regional
• relaese schedule: http://www.gks.ru/gis/images/graf-oper2017.htm

Gov interest rate curves:

• http://www.moex.com/ru/marketdata/indices/state/g-curve/archive/
• http://www.cbr.ru/hd_base/Default.aspx?Prtid=zcyc_params
• CBR also has 'SOAP' interface, some attemps to use it here: https://github.com/epogrebnyak/cbr-soap-py

USD/euro echange rate at ECB:

• http://www.ecb.europa.eu/stats/policy_and_exchange_rates/euro_reference_exchange_rates/html/eurofxref-graph-usd.en.html
• https://sdw-wsrest.ecb.europa.eu/web/generator/index.html

More source to follow. I think we need a template to describe sources and make sure everyone understands Raw Database layer.

frontend: experimental app URLs

Purgeable/frontend-app#14

db: sample methods for database access

mini-kep/db#2

list of macroeconomic APIs

• FRED
• quandl
• OECD
• World Bank
• EIA

---

oanda

docs: separate documentation

Must separate for easier update:

• README.md (short entry and quick examples)
• CHANGELOG.md (updated often)
• project wiki (rather immutable)

Charts for frontpage

Delete testing guidelines

In favour of https://github.com/mini-kep/intro/tree/master/testing_guidelines

parsers: make controller/information class

mini-kep/parsers#3 working on
https://github.com/mini-kep/parser-template/blob/master/definitions.py

other notes form README.md

Other notes

• 'maas' - forecasting model as a service (experimental)

• This pipeline may also apply to other countries macroeconomic data, or other kinds datasets (eg firm-level data, banking statistics).

• May want to condense all of the above in one repo for tasks in other domains.

migration: app and mini-kep parser

Migrate:

• epogrebnyak/mini-kep -> mini-kep/parser-rosstat-kep
• epogrebnyak/mini-kep-app -> mini-kep/frontpage-app

Change:

• CSV source at https://github.com/epogrebnyak/mini-kep-app/blob/master/config.py
• heroku github source (EP)

Also:

• sample code for download at config.py
• may need to setup Travis at app and parser

update project map