hasadna/knesset-data-python

python module that provides api to access Israeli Parliament (Knesset) data

knesset-data python module

A python module that provides api to available Israeli Parliament (Knesset) data

This module provides low-level APIs to support other higher-level modules from the Knesset data project

Installation

• $ pip install knesset-data

Usage Example

• $ python

• from knesset_data.dataservice.committees import Committee

• committees = list(Committee.get_all_active_committees())

• len(committees)

• 19

• print(committees[0].name)

• ועדת הכנסת

Known issues / FAQ / Common problems

• Knesset blocking of some requests - this is a major problem, but it only affects specific cases, so you might not be affected by it. Please add a call to is_reblaze_content to check any content you get from knesset and detect whether it was blocked, or if there is a different problem. See this issue for some more details and possible fixes.
• what logic should knesset-data-python be responsible for?

Contributing

See the contribution guide

Project Administration

Travis publishes to pypi automatically on every published release (or tag)

Publishing a release

• merge some pull requests
• create or update the latest draft release (https://github.com/hasadna/knesset-data-python/releases)
  • update the release notes, save draft
• test / update the release
• when ready, publish the release on GitHub
• travis will automatically publish to pypi

Updating dependant projects

After publishing a release you probably want to update the dependency in dependant projects

In Open Knesset repository -

• edit Open-Knesset/requirements.txt
• knesset-data==1.2.0
• test and open a pull request in Open Knesset

More details regarding travis publishing to pypi

The .travis.yml file contains encrypted variables, you add them using the travis cli client:

$ git config --local travis.slug hasadna/knesset-data-python
$ travis encrypt TRAVIS_PYPI_USER=(YOUR_PYPI_USER) TRAVIS_PYPI_PASS=(YOUR_PYPI_PASSWORD) --add
$ history -c

add the secure env var to .travis.yml

Check out .travis/after_success.sh to see how the publishing to pypi works

knesset-data-python's Issues

should expose the data about presence in the Knesset (נוכחות במשכן)

>>> from knesset_data.html_scrapers.mks_presence import MksPresence
>>> mks_presence = MksPresence.get()
>>> next(mks_presence)
{"mk_id": 909, "mk_name": "עבדאללה אבו מערוף", "present": True, "is_mk": True},
>>> next(mks_presence)
{"mk_id": 1: "mk_name": "יולי יואל אדלשטיין", "present": False, "is_mk": True},
>>> next(mks_presence)
{"mk_id": 41, "mk_name": "אריה מכלוף דרעי", "present": False, "is_mk": False}

• the data is available here: http://www.knesset.gov.il/presence/heb/PresentList.aspx
• there is code in open knesset which does it, but maybe it's better to write it from scratch, I suggest to use pyquery and requests library - it should be easy

should update documentation regarding what logic should knesset-data-python be responsible for

specifically, should make it clear that:

• all methods should by default return a generator that streams the data and raise exceptions in case of any problem
  • optionally, pass skip_exceptions argument which will instead of raising exceptions, yield them as part of the data stream (if possible)
• all tabular data exposed via the API should provide a way to get a json table schema for the data - containing name, data type and description for each field

dataservice should support unified field for id / desc fields from new knesset API

@OriHoch:

the new knesset API have a lot of these fields where you have somethingID and related somethingDesc
worth to create a new KnessetDataServiceField class which will handle it automatically -
so instead of defining those two fields, you would define only 1 field:
("type", KnessetDataServiceIdDescriptionField("SubType", "type of the bill"))
behind the scenes this field can expose a python tuple of (id, description)

should add committee membership data from the new knesset API

expected outcome

this is only a suggestion, feel free to implement it another way

from knesset_data.dataservice.persons import PersonPosition
for person_position in PersonPosition.get_all():
    peson_position.PersonID
    person_position.CommitteeID
    # all other fields..
    
    # mk = Member of Knesset
    # this method should return the standard member id, like:
    # 901 = איציק שמולי
    person_position.get_mk_id()
    
    # given person_position.CommitteeId
    # fetch and return a dataservice.committees.Committee object
    person_position.get_committee()

relevant documentation and details:

• knesset data API documentation - https://docs.google.com/document/d/1FHqabNsYlSNyul1lArmnONLHnYGA4XQNMs3k-dPgUok/edit
• how to add a dataservice object - https://github.com/hasadna/knesset-data-python/tree/master/knesset_data/dataservice#how-to-add-a-new-collection-dataservice
• @davi17g is working on similar issue, have a look at his PR - #8
• related knesset-data issue: hasadna/knesset-data#133

dataservice should support related table field

@OriHoch:

maybe it's worth to create a new class for this type of fields
something like:
("committee", KnessetdataServiceRelatedField("CommitteeID", dataservice_class=Committee))
which will handle it automatically
but need to make sure by default it doesn't fetch the related object - only when user requests it
so I'm not sure what's the best way to implement it..

Should support parts parsing for plenum protocols

We should support the same functionality as the one that exists for committee meeting protocols (see source here).

This will be used by the knesset-data-pipelines project for parsing plenum meeting protocols (see issue here).

Code can and should be based on existing parts parsing from the knesset-data-django project (see source here)

should support both python 2.7 and python 3.6

datapackage-pipelines framework requires python 3.6, see hasadna/knesset-data-datapackage#14

currently I think best way to solve it is to have knesset-data-python support both python versions

Wrap knesset proxies with reblaze detection and throw a dedicated informative exception

Following the discussion here
All outgoing knesset api proxies should be wrapped (probably in base class or something) with the reblaze detection func raise custom ReblazeBlockedException where the error message displayed to user is a link to a doc on github explaining this

vote - time field with value of 00:00 should not be null

@OriHoch commented on Mon Feb 01 2016

this vote's time is 00:00

• http://www.knesset.gov.il/vote/heb/Vote_Res_Map.asp?vote_id_t=23567

but in dataservice, we get vote_time = null

• http://online.knesset.gov.il/WsinternetSps/KnessetDataService/VotesData.svc/View_vote_rslts_hdr_Approved(23567)

looking at another vote on that day, you can see time of 00:01

• http://online.knesset.gov.il/WsinternetSps/KnessetDataService/VotesData.svc/View_vote_rslts_hdr_Approved(23568)

---

@OriHoch commented on Mon Feb 01 2016

the python dataservice library handles this internally at the Vote object, see https://github.com/hasadna/knesset-data/blob/master/python/knesset_data/dataservice/votes.py#L9

---

@OriHoch commented on Tue Feb 09 2016

Record of email correspondence with Knesset:

Sent: Feb 7, 2016 | From: @OriHoch

התחלנו להשתמש בממשק הנתונים החדש, יש עם הממשק בעיות רבות שמקשות לעבוד איתו באופן שוטף, להלן כמה מהן:

• (link to this issue)

Sent: Feb 8, 2016 | From: Deputy Director of the Information Departmnet (סגן מנהל חטיבת המידע)

אנחנו מטפלים ברשימת הבקשות (תקלות) שציינת.

---

@OriHoch commented on Mon Feb 15 2016

Sent: Feb 14, 2016 | From: Deputy Director of the Information Departmnet (סגן מנהל חטיבת המידע)

תקלה אחזור נתוני הצבעות
- תיאור התקלה: Vote – time field with value of 00:00 should not be null
- מצורפת דוגמה מתאריך 09/02/2016
- במידה ועדיין נראה לכם שיש תקלה, בבקשה לצרף דוגמה לדרך שביקשתם את שליפת הנתונים
- קישור:
http://online.knesset.gov.il/WsinternetSps/KnessetDataService/VotesData.svc/View_vote_rslts_hdr_Approved?$top=20&$orderby=knesset_num desc,vote_date desc

Sent: Feb 14, 2016 | From: @OriHoch

כל הפרטים נמצאים בקישורים ששלחתי
נראה שבתשובתכם התעלמתם לחלוטין מכל מה שכתוב בקישורים ששלחתי. אני יכול לחזור ולכתוב פה מה שכתוב שן אבל נראה לי קצת חסר טעם
בקישורים הנ"ל מופיעים כל הפרטים כולל כתובות מדויקות ותיאור של התקלות
אם עדיין לא ברור אשמח להוסיף מידע נוסף

Sent: Feb 14, 2016 | From: @OriHoch

רק שיהיה ברור - כל התקלות שתוארו במייל המקורי עדיין קיימות.

---

@OriHoch commented on Mon Feb 22 2016

Knesset was notified today that we are still waiting for an answer

The following description was sent regarding this bug:

נתוני הצבעות מסויימות מראות שעה שגויה

---

@OriHoch commented on Sun Mar 06 2016

Sent: Mar 6, 2016 | From: @OriHoch To: Knesset Contact

יש הרבה באגים חמורים נוספים שעדיין לא החזרתם לנו תשובה לגביהם:
(link to this bug)

votes html scraper should not fail silently by default

following PR #3 - it looks like in case an unknown member is encountered it will fail silently

in knesset-data-python we would like to be as low-level as possible and make as little assumptions as possible about how the data will be used

in this case, silently ignoring an error might cause unexpected problems down the line

I prefer that by default we will raise an exception, and add a skip_exceptions parameter which will instead of raising an exception yield the exception object as part of the data stream - to allow the end-user decide how to handle the exception

this is done in the dataservice scrapers, see for example this unit test -
https://github.com/hasadna/knesset-data-python/blob/master/knesset_data/dataservice/tests/base/test_exceptions.py#L38