mastaal/parlhist

parlhist is a Python application intended to enable more empirical and statistical academic studies of (Dutch) parliamentary minutes and documents

parlhist's Issues

Add documentation

Directly use API for all crawling

Add support for Kamerstuk attachments

Add support for Staten-Generaal Digitaal data (1814/1815 - 1994-31-12)

Older parliamentary history is not available via the regular data collection, but is digitalized in the "Staten-Generaal Digitaal" (sometimes abbreviated to sgd) dataset. Support for this involves some extra work, as this dataset is completely based on scans of paper documents.

Support crawling complete dossier with more than 1000 results

parlhist/parlhistnl/crawler/kamerdossier.py

Lines 21 to 43 in e59edcd

	def get_kamerstukken_in_kamerstukdossier(dossiernummer: str) -> list[tuple[str, str]]:
	"""Using the SRU api, get all kamerstukken in a kamerstukdossier
	Returns strings in the same formatting as denoted in other metadata, e.g. '35899;7', '35925-VII;31' or '35979;F'
	"""
	# overheid SRU documentation:
	# https://data.overheid.nl/sites/default/files/dataset/d0cca537-44ea-48cf-9880-fa21e1a7058f/resources/Handleiding%2BSRU%2B2.0.pdf
	base_query = f"https://repository.overheid.nl/sru?query=(w.dossiernummer=={dossiernummer})&maximumRecords=1000&startRecord=1"
	try:
	query_response = get_url_or_error(base_query)
	except CrawlerException as exc:
	logger.error("Received exception %s when trying to query the kamerstukken in dossier %s", exc, dossiernummer)
	raise CrawlerException(f"Received exception when trying to query the kamerstukken in dossier {dossiernummer}") from exc
	xml: ET.Element = ET.fromstring(query_response.text)
	number_of_records = int(xml.find("{http://docs.oasis-open.org/ns/search-ws/sruResponse}numberOfRecords").text)
	if number_of_records > 1000:
	logger.critical("More than 1000 results in a kamerdossier!")
	# TODO, implement this case

Automatically match Handeling items relating to the same debate

Create experiment template

Based on this template, users (who can already program in Python) should be able to easily create new experiments.

• The template should store its inputs and results in an appropriate fashion in the database (also see #8)
• The template should make effective use of parallelization (also see #9)

Web interface for making simple queries

Automatically match debates in TK and EK if they are about the same dossier / legislative proposal

Store experiment results in the database

Add efficient parallelization when indexing new data