mastaal/parlhist

parlhist is a Python application intended to enable more empirical and statistical academic studies of (Dutch) parliamentary minutes and documents

parlhist

parlhist (working title) is a Python application intended to enable more empirical and statistical academic studies of parliamentary minutes and documents. It is developed at the Institute of Public Law of the Leiden Law School (Leiden University, the Netherlands), in order to investigate in a more empiric manner the role of the Dutch Constitution in Dutch parliamentary debates.

The data used by this program is retrieved from the official publications by the Dutch Government. More information can be found on the page of this dataset on data.overheid.nl. The API reference of the SRU API by KOOP can be found here.

Current features

Currently, parlhist can be used to easily crawl all the Dutch parliamentary minutes (Handelingen) and related documents (Kamerstukken). These documents are stored in a database and can be easily queried using the Django database-abstraction API (or by querying the database directly). Additionally, new Django commands can be written to write automated experiments.

Planned ideas

• More advanced natural language processing to get more insight than just counting matches.
• Support for decentralized democratic bodies in the Netherlands (Provinciale Staten, Gemeenteraden, Algemeen besturen van Waterschappen).
• Web interface to make queries
• Store experiment results in database
• Support for parliamentary history from before parliamentary year 2011-2012

Requirements

• A database, preferably PostgreSQL. For small datasets and experiments, SQLite may suffice. But be aware that the software is only tested on PostgreSQL.
• A machine to run parlhist on, preferably a (recent) Linux machine. The software might work on other operating systems.

Usage

Installation

Clone the repository:

$ git clone https://github.com/mastaal/uitspraken.git

Create a Python environment and install all dependencies:

$ cd uitspraken
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -U -r requirements.txt

If you want to edit the code, you might want to also install all the dependencies in development_requirements.txt as well. Then, make the appropriate changes to the DATABASES variable in parlhist/settings.py, so that it is configured to use your database.

Preparing the database

Then, we can initialize the database:

$ ./manage migrate

Then, we can populate the database as follows. Currently it is only possible to crawl one full year of parliamentary history at once:

$ ./manage vergaderdag_crawl_full_vergaderjaar --kamer tk 20212022

Once you have crawled all the parliamentary years you're interested in, you can download the related parliamentary documents using the following commands:

$ ./manage handeling_crawl_uncrawled_behandelde_kamerstukdossiers
$ ./manage handeling_crawl_uncrawled_behandelde_kamerstukken

Depending on how many years of data you have crawled, this may take several hours.

Alternatively, you can run the initialize_database.sh shell script, which initializes the database with all Handelingen of both the Eerste Kamer and Tweede Kamer of the parliamentary years 2011/2012 through 2021/2022, and the related Kamerstukken.

Note on memoization

By default, parlhist stores all responses it gets in a raw format. If you want to re-create your database, you can quickly rebuild everything from these memoized requests. The downside of this, is that the memoized requests could technically be outdated. But as long as you're only working with fully completed parliamentary years, this should not pose a problem.

When no memoized request exists, the crawler will wait some time to prevent overloading the API.

Run your experiments

Now that parlhist is installed and the database populated with data, you can run your experiments. Two main approaches exist to do this. First, you can add a new Django command which runs your experiment (such as parlhistnl/management/commands/experiment_1_grondwet.py we've used). Secondly, you can enter a Django shell using $ ./manage shell and export the data you are looking for to some other format (pandas, json, etc.), and do the data analysis in some other tool (a Jupyter Notebook for example).

Publications

None yet. Stay tuned!

License

Copyright (c) 2023, 2024 Martijn Staal/Universiteit Leiden <m.a.staal [at] law.leidenuniv.nl> / <parlhist [at] martijn-staal.nl>

Available under the European Union Public License v1.2 (EUPL-1.2), or, at your option, any later version.

parlhist's Issues

Automatically match debates in TK and EK if they are about the same dossier / legislative proposal

Automatically match Handeling items relating to the same debate

Add support for Staten-Generaal Digitaal data (1814/1815 - 1994-31-12)

Older parliamentary history is not available via the regular data collection, but is digitalized in the "Staten-Generaal Digitaal" (sometimes abbreviated to sgd) dataset. Support for this involves some extra work, as this dataset is completely based on scans of paper documents.

Web interface for making simple queries

Support crawling complete dossier with more than 1000 results

parlhist/parlhistnl/crawler/kamerdossier.py

Lines 21 to 43 in e59edcd

	def get_kamerstukken_in_kamerstukdossier(dossiernummer: str) -> list[tuple[str, str]]:
	"""Using the SRU api, get all kamerstukken in a kamerstukdossier
	Returns strings in the same formatting as denoted in other metadata, e.g. '35899;7', '35925-VII;31' or '35979;F'
	"""
	# overheid SRU documentation:
	# https://data.overheid.nl/sites/default/files/dataset/d0cca537-44ea-48cf-9880-fa21e1a7058f/resources/Handleiding%2BSRU%2B2.0.pdf
	base_query = f"https://repository.overheid.nl/sru?query=(w.dossiernummer=={dossiernummer})&maximumRecords=1000&startRecord=1"
	try:
	query_response = get_url_or_error(base_query)
	except CrawlerException as exc:
	logger.error("Received exception %s when trying to query the kamerstukken in dossier %s", exc, dossiernummer)
	raise CrawlerException(f"Received exception when trying to query the kamerstukken in dossier {dossiernummer}") from exc
	xml: ET.Element = ET.fromstring(query_response.text)
	number_of_records = int(xml.find("{http://docs.oasis-open.org/ns/search-ws/sruResponse}numberOfRecords").text)
	if number_of_records > 1000:
	logger.critical("More than 1000 results in a kamerdossier!")
	# TODO, implement this case

Add documentation

Add support for Kamerstuk attachments

Add efficient parallelization when indexing new data

Store experiment results in the database

Directly use API for all crawling

Create experiment template

Based on this template, users (who can already program in Python) should be able to easily create new experiments.

• The template should store its inputs and results in an appropriate fashion in the database (also see #8)
• The template should make effective use of parallelization (also see #9)