mysociety/theyworkforyou

Keeping tabs on the UK's parliaments and assemblies

TheyWorkForYou

Hello everyone,

This is the source code that powers the TheyWorkForYou.com website. It's mainly written in PHP, although there are also some Perl scripts for database loading and indexing. The backend parser code is separate and available on GitHub. You can read more about this on TheyWorkForYou's parser info page

The TheyWorkForYou source code in this distribution is released under a BSD style license. Roughly, this means you are free to copy, use, modify and redistribute the code or binaries made from the code. Commercial or non- commercial use is allowed. However, we disclaim warranty, and expect you not to use our name without our permission. See the file LICENSE.md for exact legal information.

What is TheyWorkForYou anyway?

Everything MPs say in the UK's House of Commons is recorded in a document called Hansard; TheyWorkForYou helps make sense of this vital democratic resource. It also includes things from the House of Lords, the Scottish Parliament, and the Northern Ireland Assembly.

How on earth do I use this code?

See INSTALL.md for installation questions.

If you have questions, the best place to ask is the mySociety TheyWorkForYou email list at https://groups.google.com/a/mysociety.org/forum/#!forum/theyworkforyou

Developing with Docker

You will need a recent version of Docker.

Running docker compose up [ -d ] will start the environment. The first time you run this it may take a few minutes as various images are downloaded and the application image built locally.

Once it's done, you should be able to view the front page at http://localhost:8000. However the default build contains no data, so take a look at INSTALL.md for information about downloading and importing Parlparse data (members, debates, votes, etc).

You can stop the environment by running docker compose down. Adding a -v will remove any Docker volumes that may be in use, including all their data.

DOCKER.md has some more detailed notes on the development environment, together with some useful commands and more detailed Docker-specific setup notes.

To use xdebug in VS Code while using WSL, you'll need to set an environmental variable of the WSL_IP within the subsystem of the IP address of the subsystem.

Developing with codespaces

Start a new codespace on Github by selecting the Code dropdown (top right), and starting a new codespace (or use the GitHub CLI).

You can also use the badge above to use a prebuild with a basic amount of data.

This will setup the Docker container and environment. Once finished, the link to the site should be avaliable in the ports tab of the terminal panel.

To populate with a minimal amount of data, run scripts/quick-populate (about 1 hour).

DEPRECATED: Developing with Vagrant

Please note that we are not currently supporting the Vagrant environment, and may remove it altogether. Please use the Docker environment instead.

You will need the latest versions of VirtualBox and Vagrant, then:

• Run vagrant up.
• Go make a cup of tea. It may take a while whilst Vagrant and Puppet do their thing.
• Point your web browser at http://10.11.12.13 and marvel at modern technology.

See INSTALL.md for instructions on downloading and importing Parlparse data (members, debates, votes, etc).

Compiling Static Assets

If you're working on a page which uses the redesign, you will need to compile static assets after changes:

• vagrant ssh
• cd /vagrant/theyworkforyou/www/docs/style
• bundle exec compass compile for a one-off compilation or bundle exec compass watch to recompile on changes

Testing

TheyWorkForYou includes a test suite, using PHPunit. To run tests, ensure that the environment variables TWFY_TEST_DB_HOST, TWFY_TEST_DB_NAME, TWFY_TEST_DB_USER and TWFY_TEST_DB_PASS are set and contain relevant information for your testing database. The database will be stripped down and rebuilt during testing, so make sure it's not an important copy.

You may find that in some versions of PHPUnit errors are thrown regarding code coverage reports. If this is the case, the version installed by Composer and located at /vendor/bin/phpunit should run correctly.

Build Status

Acknowledgements

Thanks to Browserstack who let us use their web-based cross-browser testing tools for this project.

theyworkforyou's Issues

Show individual recent votes

But how to decide on which?

Show divisions properly on debate pages on TheyWorkForYou, ie showing party breakdowns on each division

Twitter bot that tweets @ mps when their inboxes fill up

Twitter bot that tweets @ mps when their inboxes fill up

We already know who’s inboxes are full, data we require in order to successfully deliver mail. Tweeting @ MPs who’s mail is bouncing would be both useful in terms of telling them to clear their mail, as well as mildly amusing for everyone else.

Separate things MPs say as ministers from things they say as constituency MPs

Members interests for House of Lords on TheyWorkForYou

The house of lords has a register of members' interests similar to that of the commons.

The register is online at: http://www.publications.parliament.uk/pa/ld/ldreg/reg01.htm which is split with several peers on each of the pages:

This is scraped already: http://ukparse.kforge.net/parldata/cmpages/ldregmem/ - so the best place would be to add to the parser there that already does the Commons.

Add video of MP's most recent speech to their MP page

For each law in legislation.gov, show all the MPs who gave speeches on it, or who voted on it

Break recent appearance and recent WRANs into separate sections on the MP pages

Add Select Committees in basic form

Add transcripts of all select committees, add them into alerts. Do not have pages for non-MPs, but give them IDs that can join together multiple appearances.

Photo attributions

If the site allowed a photo to carry an attribution string, we could use a wider variety of photos (e.g. plenty from Wikipedia).

Highlight what people aren't reading and bubble to the top

And reward people for doing so somehow

Wrong predictions identifier tool/category

Allow users to find specific predictions about how a piece of legislation will work, and then connect it to evidence about what actually happened.

MP "trackbacks" - intra-Hansard citations

Suggestion: if a TWFY annotation contains a link to another speech in TWFY, then a link to the annotated speech should automatically appear on the page of the linked speech. Similar to the blogging concept of a trackback.
Example:

¤Lord Razzall citing Vincent Cable
¤Vincent Cable alluding to Winston Churchill
also:

I've noticed a few times MPs asking for written answers, and then making reference to the information in a debate. Not a great example, but:

¤a question from David Davies regarding how much is being spent of videogames in young offender institutions
¤and then in a debate

Annotation filter on TheyWorkForYou bit buggy

Strips "<" for example, bit annoying.

TWFY doesn't cope with two lords with the same name

Lord Aberdare, 4th of that Barony, is present in TheyWorkForYou. He died in 2005, and his son, the 5th Baron Aberdare, was by-elected to the House of Lords recently. /peer/lord_aberdare only gives you the 5th Baron, with no way of viewing the 4th any more.

Search results for written answers should show question

It would be nice if the search results always included part/all of a written question, even if the result was actually found in the answer, so that you have more information as to whether it's a useful result.

commonlib is locked

is there a reason the code in the commonlib is not available?

Video not working

Fix it

People who have multiple pages on TheyWorkForYou

¤http://www.theyworkforyou.com/peer/lord_foulkes_of_cumnock and ¤http://www.theyworkforyou.com/msp/george_foulkes are the same person.

Pages on TheyWorkForYou for the companies represented in evidence to committees

Linking to speech within whole debate page should move video

If you have a link such as ¤http://www.theyworkforyou.com/debates/?id=2009-01-15d.345.0#g350.2 then the Flash should read the anchor and move the video to the start of speech 350.2, rather than starting from 345.0 as it currently does.

Do something about postcodes that straddle boundaries

Even if it's just a manual table that upon being given a postcode we've been told about, explains what's happening. Would be useful for both TheyWorkForYou and WriteToThem.

Spot oversea territory postcodes

As with WriteToThem (fyr/web/index.php has the lookups), we should spot special postcodes such as the Falkland Islands and alert accordingly, rather than just give an error.

Attribution string for photos

If we had a way of attributing photos, we could put Wikipedia photos up of MPs and so on, currently as there's no way of doing that we can't.

Offer email alerts when MPs rebel

Daily pairwise game to compare the interestingness of speeches

Another way of trying to bubble up the best of yesterday's speeches and questions

Glossarise company names that are not in Wikipedia with links to open corporates

Deal with the "multiple alert"s

There are some email alerts that are obviously meant to be more than one alert, with separate search terms separated by commas. We should:

• alter these to be separate alerts,
• email everyone to apologise for our not-clear-enough text/point out searches for their terms,
• set it up so multiple alerts can be specified at once in the alert sign-up.

Add votes and proceedings to TheyWorkForYou (where they reveal statutory instrument titles that are not debated but where the law gets changed anyway)

Heading improvements

The "FOO is in the Chair" statements on Westminster Hall debates are appearing as headings on TheyWorkForYou, which is a bit confusing and can split a debate in two if there's a change of chair halfway through. They should be spotted when the HTML is being parsed and altered to normal stage direction paragraphs. The parser is a separate project, code available here: ¤http://www.knowledgeforge.net/project/ukparse/

Also, the parser has trouble spotting when things are subheadings or full headings. This has been looked at before, but could perhaps be done better nowadays.

Add the Welsh Assembly

We can show debates and send email alerts for Westminster, Scotland and Northern Ireland, but not Wales. It is an elitist discrepancy, and we should import the required data and provide the usual functionality of search, browse, pages per debate and per speech, and email alerts. This does not cover vote analysis or import.

Add European Parliament

Future business

Add the ability to see as much of the forward-looking calendar of debates, votes, committees and events as Parliament has published at any point in time. Provide the ability to browse it, search it, and (most important) set up alerts to warn people of upcoming events that may interest them.

TheyWorkForYou: Petition Your MP

MPs have long complained about not being able to handle spammy constituency mail that consists of thousands of copies of the same thing.

We could build a petitioning system into TheyWorkForYou that would allow people to make petitions aimed at specific MPs. MPs and campaigns could then point people to these, reducing the spammy mail and making it much easier to see what the public is trying to get out of MPs.

Essential features would include:
a) Alerting MPs to petitions and sending them updates as signatures grow
b) Allow MP and petition creator to send limited number of messages to signers
c) Integrate with pre-existing rate-limiting on WriteToThem, to encourage people to turn their bulk mails into petitions.

Get rid of video for "so-and-so: rose---" speeches

There's no point having video for speeches that are just "So-and-so: rose---" because 90% of the time it happens off camera so all you can watch is the current speaker looking in a slightly different direction.

Often, the same applies for speeches that are just "Would so-and-so give way?" but that might be slightly more difficult to code as "so-and-so" could be "the Minister", "the honourable Lady", "my hon. Friend" etc.

Email alert from search results page flow has regressed

It should immediately take you to the same place you get when you now click the Subscribe button. Pretty sure it used to do that.

Add MP's Positions on national issues obtained via DemocracyClub

As part of the Democracy Club process we asked for candidates positions on a range of major issues (i.e http://election.theyworkforyou.com/quiz/S80BF ) . Many of them became MPs, so we have structured data for many of their answers.

We could make MP pages more useful simply by adding these as a list of statements, such as:

"Disagrees with the statement 'Even if Iran develops a nuclear weapon, Britain should not support any military action against Iran'"

We don't have this data for all MPs, but we could allow those who didn't do the survey at the time of the election to do it now.

Later on it might become possible to compare statements against voting behaviour, but that would not be within the scope of this phase of work.

Add future business to TWFY

Parse & incorporate future business into TWFY
Build UI for reading it

Build alerting mechanisms

Thought (not for this ticket) - could we perhaps build an alerting system for 'important' votes of different sorts, so people could subscribe to say 'email me when my MP is going to vote on an important environmental/medical/tax issues'? Would have to do it in conjunction with a media partner to do all the research of course (or have the most amazingly dedicated volunteer).

Visual representation of divisions (votes) in debates, eg: with a graph split by party

Improve internal linking and browsability through crowd-sourcing missing data

There are two main examples of weaknesses in our data that could be improved by crowdsourcing certain gaps in our data:

• Hansard backreferences (e.g. the reference to a previous Hansard on pages like http://www.theyworkforyou.com/wrans/?id=2006-06-12b.75897.h ). Rather than linking somewhat unhelpfully to the column, we could build a tool to enable volunteers to pick the correct result from the column, and then we could stud Hansard with links to the exact right place.
• Things that are deposited in the library of the House - all listed on http://deposits.parliament.uk/ in a big long list - could be linked directly to the document to which they're referring. Use references to MP names in order to speed this up for letters, and see https://secure.mysociety.org/cvstrac/tktview?tn=923,31 for a ruby scraper.

Link lobbying information to TheyWorkForYou somehow

Improve stats generation

For example, not sure it's running on MLAs, or former MPs, for the ones that it would be fine to run (e.g. comments_on_speeches)

Use data.parliament.uk data

e.g. http://data.parliament.uk/resources/Members/api/Members.svc/help
Note (August 2013), there's a new API at http://data.parliament.uk/membersdataplatform/memberquery.aspx which looks to have more.

From this we could get the following:

• Commons membership and biography details
• Lords membership and details, and recently new or left members
• Departments, their minsters and opposition posts
• Committees and their membership (both houses)
• Lords register of interests

A way of fetching/storing/keeping this data up to date (perhaps within the parser where it historically has been), needs to be written. This could also alert us to changes in composition, party changes, and so on.

Glossary titles are sometimes infected by Wikipedia links and other oddities

e.g. ¤http://www.theyworkforyou.com/ni/?id=2009-03-10.5.1 with the visible class="glossary" due to the bad HTML. And http://www.theyworkforyou.com/whall/?id=2009-05-14b.297.0 the PCT link has double linkage.

TWFY PBC attendance numbers should be per person

Currently, Anne Main is listed as attending 0 out of 18 sessions of the Planning Bill PBC, as it had 18 sessions in total. According to the source debates on parliament.uk, she was only a member for the first two sessions, so should more accurately be listed as 0 out of 2.

Better maiden speech links

Currently the code looks for the first contribution in the database per MP that contains any of a set of phrases (e.g. 'maiden speech', 'predecessor'). This should be fine for the current intake, but I assume there are some mistakes in the past. data.parliament.uk does include maiden speech data, in the form of date and column number, which may well be enough to match even better (we have column number data).

Add an admin interface to the title_ignored table

So that it's easier to remove bad/wrong automated Wikipedia links.

Peter Robinson is broken

It gets stuck in an infinite redirect, presumably some MP/MLA confusion as to what URL it should be on.

Remove WriteToThem Spellchecker

And encourage people to upgrade to a browser that has one built in

Changing entering/ leaving reasons for old MP on TWFY

The ref web site ¤http://www.theyworkforyou.com/mp/george_thomson/edinburgh_east indicates he entered parliament on 5 July 1945 - General Election, however a search shows it was a by-election. The site indicates he left parliament in 1947 - unknown. It is now obvious that he left due to being raised to the bench.