spudmind/undertheinfluence

Tracking influence in UK politics

undertheinfluence's Issues

Remove illegal characters from source data at parse

MP's Register of Interests export

Display raw record on detail views

UI representations of raw record is stored on node:Funding Relationship

Upgrade to python 3?

At some point, you know.

I’m not sure if there’s much reason to be using python 2.x here.

Add --verbose argument to all tasks

Implement Web Content Accessibility Guidelines

Add an argparser to the script runner

(Rather than commenting stuff out). Seems like this would be a useful incremental improvement.

Influencer Detail Api

Make MITIE a submodule?

I built MITIE from the github source and it works fine. This is possibly an easier install process than downloading the pre-compiled version…? Not sure, really – I’ll investigate!

Scraping – date ranges

[This is not a well-worded issue]

We need a solution for scraping source data within a given date range. For instance, we currently scrape all Electoral Commission data (since 2001) and dump it into one big csv file. It would be preferable to dump yearly csvs, and come up with some solution for the current year.

libmitie.so: cannot open shared object file: No such file or directory

Scrape party funding from source

We currently scrape from a CSV file, but I guess this needs to be manually downloaded?

It would be better to scrape from the EC site. I’ve written the code for this using mechanize – just need to save the file (we should standardise how we’re dumping files.)

ImportError: No module named mitie

Run into this issue too. Fixed by changing utils/mitie to utils/information_extraction.

Not sure why this issue only intermittent, I suspect sys.path issue. The problem the folder under utils was called 'mitie' as well the python mitie interface file (utils/information_extraction/mitielib/mitie.py)

Add query function to mongo wrapper

Entity extraction refactor

The entity resolver needs some work to make it behave consistently across Mp, Lords and donor / registered interest searches

Scrape APPC lobbying register

http://www.appc.org.uk/members/register/
(or the UKPAC site: http://www.publicaffairscouncil.org.uk/en/search-the-register/index.cfm )

Application install on server

Unique IDs for all enities

Data scraping automation

Mps API

Mps API to return registered interests and associated donations

Create Register of Lords Interests parser

Scrape PRCA lobbying register

http://www.prca.org.uk/paregister

Information architecture and design

Backbone interface to API

Search Implementation

Elastic Search + Mongo api data

Error parsing master entities from scratch db

Hitting an error right here:
https://github.com/spudmind/spud/blob/2c5a8ded2/utils/entity_resolver.py#L48

The problem is we’re using master_mps before it has been initialised. On a populated db it’s probably fine, but not on the first run.

Lord detail API

Parsed Electoral Commission data export

Add api_url's to collection apis

Not all parties correctly identified and parsed

Green Party not being tagged a Political Party.

Liberal Democrats and Liberal Democrat Party not recognized as the same party

MP detail API

Drop api mongo collections when running api_gen

Standardize output across detail APIs

Create Register of Lords Interests grapher

Search UI

Political Party API

Lords API

Lords API to return registered interests and associated donations

Paginate collection pages

Entity resolution and search API

Lords Register of Interests export

Politician Detail Api

data_models/core.py: AttributeError: 'NoneType' object has no attribute 'status_code'

Implement general party list

Influencers List API

Basic user interface scaffolding

Server Configuration

Separate out Person ID Mapper component

Resolving a person from an ID is the job of a standalone app, really. We should separate this component out.

See: https://github.com/symroe/MP-ID-Mapping

Scrape meetings log: Ministers (Westminster)

2 sources:

1. Gov.uk; contains data grouped by department, e.g. DWP: https://www.gov.uk/government/collections/dwp-meetings-with-external-organisations
2. data.gov.uk: again by department, e.g. Treasury: http://data.gov.uk/dataset/publication-of-ministers-meetings-hospitality-gifts-and-overseas-travel

Scrape ACOBA

Updated page: http://acoba.independent.gov.uk/former_ministers/former_ministers_appointments_2012_.aspx
https://www.gov.uk/government/collections/appointments-taken-up-by-former-ministers

Annual reports: http://acoba.independent.gov.uk/publications/reports.aspx
http://webarchive.nationalarchives.gov.uk/20140404084345/http://acoba.independent.gov.uk/publications/reports.aspx
(or: https://www.gov.uk/government/latest?departments%5B%5D=advisory-committee-on-business-appointments)

Politicians collection API