hasadna/knesset-data-datapackage

Provides downloadable Knesset (Israeli parliament) data

Hasadna monorepo

Installation

Install Bazel. That's it!

Build & Test

• Build everything: ./compile.sh build
• Run all tests: ./compile.sh test

About monorepos

A monorepo is a software development approach where all code is stored in a single repository. Some things are easier to do in a monorepo, such as sharing a proto file across front-end and backend, some things are harder, such as per-repo control over collaborators, email notifications, commit history etc.

Some monorepo etiquette:

1. Don't store large files in the repo (>500kb), or many small files. Until we figure out a way to deal with them, you can store them in a different repo.
2. If you use code belonging to another project, talk to the people of that project about it. If you don't, they might accidentally break your code.
3. If you change code common to multiple projects, do it thoughfully.
4. Kindly provide a LICENSE file for your project.

Some good reads about the monorepo approach:

• trunkbaseddevelopment.com/monorepos
• Why Google stores billions of lines of code in a single repository

Platforms

While Bazel supports Linux, Mac and Windows, this repo supports Linux and Mac. If you're on Windows 10, you can use Windows Subsystem for Linux (WSL). It gives you a Linux environment, without the overhead of a virtual machine.

Installing Windows Subsystem for Linux:

Please follow the guide. Note: You may have to do a Windows upgrade. If you need it, it will ask for it at the beginning of the installation.

Contributing

You're welcome to contribute and in doing so, learn these technologies. You can have a look at the issues list, or at the project milestones.

Happy coding!

knesset-data-datapackage's Issues

travis problem: tests pass locally but fail on travis

the bug happens on PR #4
relevant travis build: https://travis-ci.org/hasadna/knesset-data-datapackage
for some reason a specific test fails on travis but passes locally

steps to run locally

git clone --depth=50 https://github.com/hasadna/knesset-data-datapackage.git travis-bug
cd travis-bug
git fetch origin +refs/pull/4/merge
git checkout -qf FETCH_HEAD
virtualenv -p /usr/bin/python2 venv
. venv/bin/activate
.travis/install.sh
bin/run_coverage.sh

expected

• tests should pass both locally and on travis

test_exception_in_resource_while_making_datapackage (knesset_datapackage.tests.test_exceptions.ExceptionsTestCase) ... ok

actual

• tests pass locally but fail on travis

test_exception_in_resource_while_making_datapackage (knesset_datapackage.tests.test_exceptions.ExceptionsTestCase) ... ERROR

http timeout error should not cause entire datapackage to fail

relevant traceback -

knesset_datapackage.root:423	INFO	writing datapackage.json
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.7.12/bin/make_knesset_datapackage", line 11, in <module>
    sys.exit(make_datapackage())
  File "/home/travis/virtualenv/python2.7.12/lib/python2.7/site-packages/knesset_datapackage/cli.py", line 77, in make_datapackage
    mock=args.mock)
  File "/home/travis/virtualenv/python2.7.12/lib/python2.7/site-packages/knesset_datapackage/base.py", line 425, in make
    f.write(self.get_json_descriptor()+"\n")
  File "/home/travis/virtualenv/python2.7.12/lib/python2.7/site-packages/knesset_datapackage/base.py", line 405, in get_json_descriptor
    return json.dumps(self._dict_recursively_remove_nulls(new_descriptor), indent=True)
  File "/opt/python/2.7.12/lib/python2.7/json/__init__.py", line 251, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 332, in _iterencode_list
    for chunk in chunks:
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 442, in _iterencode
    o = _default(o)
  File "/opt/python/2.7.12/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: ReadTimeoutError("SOCKSHTTPConnectionPool(host='knesset.gov.il', port=80): Read timed out. (read timeout=15)",) is not JSON serializable
failed to create datapackage

datapackage should include bills data from knesset-data-python

now that we have dataservice bills from new API - we should expose it in the datapackage

should use the datapackage-pipelines framework to generate datapackages

should modify our current system for generating datapackages to use the datapackage-pipelines framework

this will allow to have more control and visibility over the datapackage creation process

datapackage-pipelines

support fetching from the datapackage

should support fetching data from a datapackage object / resources

• fetch online
• fetch from a previously downloaded datapackage zip / directory

the created datapackage should conform to a version of frictionlessdata specs

reproduction steps

• try to open / use the datapackage from a frictionlessdata compatible tool

expected

• should work

actual

• doesn't work with some or all tools
• need to ensure we conform to the specs or use the datapackage tools to create the datapackage
• need to decide which version of the specs we want to support

datapackage should include committee membership details

datapackage should contain csv of details about which MKs are members of which committees

related issues:

• hasadna/knesset-data#133
• hasadna/knesset-data-python#10

should have retry mechanism + longer http timeout

should have global configurations affecting all resources:

• http timeout (default = 180 seconds)
• retries (default = 10 retries)
• time to wait between retries (default = 60 seconds)

these configurations should be available as parameters from the make_knesset_datapackage CLI tool