Extracts information from:
and expose that information through a simple to use API.
Currently, the following information is being exposed:
- Portuguese Parliament initiatives;
- Portuguese Parliament vote information;
- Information regarding Portuguese legislative elections.
All information provided is free of use by anyone, so that they can be used for information dissemination or investigation.
make setup init
make runlocal
- There is a cronjob in Luís Silva personal computer that extracts the data from Portuguese Parliament to our datalake.
- Source code executed is
cronjob.py
; - This is necessary because requests from Heroku and other cloud providers are rejected;
- Runs every day at 8 pm Portugal time.
- Source code executed is
- The daily updater loads the data from our datalake and extracts the needed information, compute some statistics, and store everything in the datalake to later be consumed by the API.
- This process takes around 1h (when processing XIV legislature);
- Source code executed is
src.daily_updater.parliament.main
. - Runs every day at 3 am GMT.
- Heroku dynos sleep after 30 min of inactivity, meaning the daily updater won't run if the dyno is sleeping. Thus we have an Azure Function making requests to our API to avoid that.
- Source code executed is
AzureFunctions.PortuguesePoliticsDailyUpdate.src.main.py
- Runs every day from 2 am GMT to 5 am GMT at each 20 minutes;
- We can't let this run the full day becuase Heroku has a limit of 1,000 dyno hours per month for our tier.
- Source code executed is
- API runs using Gunicorn and FastAPI
- Source code executed is
src.app.main
- Source code executed is
The Python version used must be one of https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-python, since this is the environment used by Heroku
- If a new data source needs to be downloaded from the Portuguese Parliament it is necessary to update the cronjob task
- Follow the
src.daily_updater
examples to pre-process the data and store it in our datalake - Add the new endpoints to
src.app.main