mini-kep / db Goto Github PK
View Code? Open in Web Editor NEWDatabase for parser outputs with flask API to read/write data.
Database for parser outputs with flask API to read/write data.
When looking at the source of csv output I can see the orginal CSV file easily:
view-source:https://minikep-db.herokuapp.com/api/datapoints?name=USDRUR_CB&freq=d&start_date=2017-08-01&end_date=2017-10-01
...but in browser it gets rendered to html. Can we prevent html rendering in browser? What controls for it? some headers?
The document to review is this:
https://github.com/mini-kep/db/blob/master/SPEC.md
I need to spot:
db/db/custom_api/custom_api.py
Line 1 in 21485a7
Can we mount custom API in db project too? See https://github.com/mini-kep/helpers/issues/3 and https://github.com/mini-kep/frontend-app/blob/master/apps/views/time_series.py
Sources of error are:
get_datapoints_response
deleted in code under testLines 189 to 217 in 3cec7d0
This can be a helper function in db
package:
https://github.com/mini-kep/parser-rosstat-kep/blob/master/src/csv2df/util_label.py
We also need a listing of variable units of measurement descriptions, similar to #39, this can be also borrowed from here:
продам гараж
Hello everybody.
This link http://mini-kep.herokuapp.com/ru/series/GDP/a/yoy/12311111111111111111
produces:
Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is >overloaded or there is an error in the application.
because TokenHelper.__as_date method is buggy.
Here's the relevant stacktrace:
127.0.0.1 - - [24/Oct/2017 16:26:24] "GET /ru/series/GDP/a/yoy/12311111111111111111 HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1997, in call
return self.wsgi_app(environ, start_response)
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/varnie/thrash/my-virt-environments/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/varnie/thrash/projects/db/db/custom_api/views.py", line 22, in time_series_api_interface
return custom_api.CustomGET(domain, varname, freq, inner_path).get_csv_response()
File "/home/varnie/thrash/projects/db/db/custom_api/custom_api.py", line 180, in init
ip = InnerPath(inner_path)
File "/home/varnie/thrash/projects/db/db/custom_api/custom_api.py", line 141, in init
self.dates = helper.get_dates_dict()
File "/home/varnie/thrash/projects/db/db/custom_api/custom_api.py", line 73, in get_dates_dict
result['start_date'] = self._as_date(start_year, month=1, day=1)
File "/home/varnie/thrash/projects/db/db/custom_api/custom_api.py", line 107, in _as_date
day=day).strftime('%Y-%m-%d')
OverflowError: Python int too large to convert to C long
Also, not sure whether it is relevant or not, but the following link:
https://minikep-db.herokuapp.com/api/datapoints?name=BRENT&freq=d&start_date=2017-01-01
also produces:
Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is >overloaded or there is an error in the application.
(I've taken this link from the README.MD: http://joxi.ru/J2b57NMiXJWWLm )
Thank you for your time.
This goes from parser 1 to database
====================================
[
{
"date": "2014-01-31",
"freq": "m",
"varname": "CPI_rog",
"value": 100.6
},
....
{
"date": "2015-12-31",
"freq": "m",
"varname": "CPI_rog",
"value": 100.8
},
{
"date": "2015-12-31",
"freq": "m",
"varname": "RUR_EUR_eop",
"value": 79.7
}
]
This goes from parser 2 to database
===================================
[
{
"date": "2014-03-31",
"freq": "q",
"varname": "CPI_rog",
"value": 102.3
}
]
User query
==========
{'end': '2015-12',
'freq': 'm',
'start': '2014-01',
'varnames': ['CPI_rog', 'RUR_EUR_eop']}
App response to query: json with epoch timestamps
=================================================
This format is default input
to user reader function pd.read_json()
{'CPI_rog': {'1391126400000': 100.6,
'1393545600000': 100.7,
...
'1448841600000': 100.8,
'1451520000000': 100.8},
'RUR_EUR_eop': {'1391126400000': 48.1,
'1393545600000': 49.35,
...
'1448841600000': 70.39,
'1451520000000': 79.7}}
User's local dataframe
======================
CPI_rog RUR_EUR_eop
2014-01-31 100.6 48.10
2014-02-28 100.7 49.35
2014-03-31 101.0 49.05
2014-04-30 100.9 49.51
2014-05-31 100.9 47.27
2014-06-30 100.6 45.83
2014-07-31 100.5 47.90
2014-08-31 100.2 48.63
2014-09-30 100.7 49.95
2014-10-31 100.8 54.64
2014-11-30 101.3 61.41
2014-12-31 102.6 68.34
2015-01-31 103.9 78.11
2015-02-28 102.2 68.69
2015-03-31 101.2 63.37
2015-04-30 100.5 56.81
2015-05-31 100.4 58.01
2015-06-30 100.2 61.52
2015-07-31 100.8 64.65
2015-08-31 100.4 75.05
2015-09-30 100.6 74.58
2015-10-31 100.7 70.75
2015-11-30 100.8 70.39
2015-12-31 100.8 79.70
Identical to source data: True
Lines 184 to 197 in 9108da4
_________________ ERROR collecting tests/test_custom_api.py __________________
ImportError while importing test module 'C:\Users\Евгений\Documents\GitHub\mini-kep-db\tests\test_custom_api.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests\test_custom_api.py:2: in <module>
import db.custom_api.custom_api as custom_api
E ModuleNotFoundError: No module named 'db.custom_api.custom_api'
'db' folder conflicts with conventional name 'db' in Flask, not too clean
Based on functionality from #37 can develop following endpoints:
api/dataframe?freq=a
api/dataframe?freq=q
api/dataframe?freq=m
api/dataframe&freq=d
Each endpoint should provide pandas-readable csv with all variables at such frequency
Should alos accept name
, start_date
and end_date
parameters.
api/datapoints/varnames/{freq}
lists all variable names available as that frequency {freq}
(assumed to be in aqmwd
)GDP
-> Gross domestic product
Lines 54 to 60 in 346ae2c
travis call is simply pytest now, not creating .coverage file
POST api/incoming
work on POST api/datapoints
in views.py and associated testnot todo:
Good evening.
http://mini-kep.herokuapp.com/ru/series/HELLO/BUDDY
Internal Server Error
The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
Thank you for your time.
For the purposes of listing variables in groups and easier browsing we need a grouping dictionary that
lists variable nameheads.
Example:
{
'GDP components': ['GDP', 'INVESTMENT'],
'Prices': ['CPI', 'CPI_FOOD', 'CPI_NONFOOD', 'CPI_ALCOHOL'],
'Foreign trade': ['EXPORT_GOODS', 'IMPORT_GOODS'],
'Exchange rates': ['USDRUR_CB']
'Interest rates': []
}
Based on nameheads and variable name splitting method one can derive the list of actual variable names like GDP_yoy
.
http://minikep-db.herokuapp.com/ru/series/CPI_rog/m/
This link should provide all available data for CPI_rog
and m
frequency.
Original class:
Lines 51 to 143 in 6536d1b
D1: all 2016 data in test_datapoints_list_2016.json
D2: all 2017 data in test_datapoints_list_2017.json
D1 should be dev database, D1 and D2 can be used in testing
To be created at around here https://github.com/mini-kep/parsers/blob/master/parsers/runner.py#L183-L189
spec
endpointThis shuts down validation error messages and hurts testing
# Return validation errors as JSON
@api.errorhandler(400)
def handle_validation_error(err):
exc = err.exc
return jsonify({'errors': exc.messages}), 422
#@api.errorhandler(CustomError400)
#def handle_invalid_usage(error):
# """
# Generate a json object of a custom error
# """
# response = jsonify(error.to_dict())
# response.status_code = error.status_code
# return response
http://minikep-db.herokuapp.com/api/datapoints?freq=a&name=GDP_yoy&start_date=2013-12-31:
,GDP_yoy
2013-12-31,101.3
2014-12-31,100.7
2015-12-31,97.2
2016-12-31,99.8
http://minikep-db.herokuapp.com/api/datapoints?freq=a&name=CPI_rog&start_date=2013-12-31:
,CPI_rog
2013-12-31,106.5
2014-12-31,111.4
2015-12-31,112.9
2016-12-31,105.4
Need to implement:
http://minikep-db.herokuapp.com/api/frame?freq=a&names=GDP_yoy,CPI_rog&start_date=2013-12-31:
,GDP_yoy,CPI_rog
2013-12-31,101.3,106.5
2014-12-31,100.7,111.4
2015-12-31,97.2,112.9
2016-12-31,99.8,105.4
This concatenation is easier when time index is the same, while in daily this may not be the case.
Todo:
To discuss:
ideas for server-side implementation
how should json
behave - provide some data structure readable by pandas into dataframe
or provide similar easy listing
This repo has several purposes and branches, mainly:
intro
repo)master
branch), latest commits by @varnieflask-sqlalchemy
), created by @SuperVasyaIn parameter handling we should have:
freq
must be certain letters, notify if notname
not found at this given this frequency, notifyCurrently on all checks return []
, bu that is a simplification.
The parsers should be able to emit variable descriptions in form of list of dictionaries:
[
{'BRENT': dict(ru='Цена нефти Brent', en='Brent oil price')},
# ...
]
Need api/title
endpoint with POST/GET methods to strore/retrieve this information from the database + new model to store this information.
api/title?name=BRENT
should return dict(ru='Цена нефти Brent', en='Brent oil price')
2017-11-08T20:45:23.097873+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=POST path="/api/incoming" host=minikep-db.herokuapp.com request_id=b554c78d-d8b8-4461-af76-f2ee4e1c68dc fwd="5.227.7.58" dyno=web.1 connect=0ms service=30490ms status=503 bytes=0 protocol=https
2017-11-08T20:45:25.292033+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/api/info?freq=d&name=UST_30YEARDISPLAY" host=minikep-db.herokuapp.com request_id=ebbdfee2-0164-43d6-84a6-bb93aa7e3e9a fwd="54.74.71.204" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=http
2017-11-08T20:45:25.285737+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/api/datapoints?freq=d&name=UST_30YEARDISPLAY&format=json" host=minikep-db.herokuapp.com request_id=f908e353-4c58-4afa-af3b-f1df9f4f52e0 fwd="54.74.71.204" dyno=web.1 connect=0ms service=30001ms status=503 bytes=0 protocol=http
Line 116 in 5c91971
хватает указания, что это f-string должно быть
raise CustomError400(message=f'Invalid frequency <{freq}>',Если сейчас посмотрите, сообщение о ошибке немного не то
https://minikep-db.herokuapp.com/api/datapoints?name=ABC&freq=z&format=json
https://github.com/mini-kep/db/blob/master/SPEC.md is not neccessary in repo root as it is hard to maintian two spec files.
The spec is at https://github.com/mini-kep/intro/blob/master/specification/database.md, please put new changes there.
Must delete SPEC.md
Lines 27 to 54 in 3cec7d0
In this pipline, we have the following situation:
api\incoming
to write incoming json to db@SuperVasya: please extend questions/descriptions.
At this issue we want:
@SuperVasya - do we need to extend tests in database, what is the direction for this?
def _find_by(session_factory, condition=None):
# FIXME: why not working with context manager?
# DetachedInstanceError: Instance <Datapoint at 0x9074ef0>
# is not bound to a Session; attribute refresh operation cannot proceed
with Session(session_factory) as session:
query = session.query(Datapoint)
if condition is not None:
return query.filter_by(**condition).all()
else:
return query.all()
[
'rog': dict(ru='% к пред. периоду', en='% change to previous period')
'yoy': dict(ru='% год к году', en='% change to 12 month earlier')
]
Can refactor dictionary below to one above.
UNIT_NAMES = {'bln_rub': 'млрд.руб.',
'bln_usd': 'млрд.долл.',
'gdp_percent': '% ВВП',
'mln_rub': 'млн.руб.',
'rub': 'руб.',
'rog': '% к пред. периоду',
'yoy': '% год к году',
'ytd': 'период с начала года',
'pct': '%',
'bln_tkm': 'млрд. тонно-км'}
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.