propublica/staffers

Interactive and searchable House staffer directory, based on House disbursement data.

House Staff Directory

A staff directory for the House of Representatives, using data culled from the House' quarterly disbursement records.

The House publishes each quarter's data by the end of the following quarter, so the records will generally be 3-6 months out of date.

Setup

Depends on Ruby 1.8.7 - not tested on anything higher. This needs to be updated.

Install dependencies with bundler:

bundle install

Copy config.ru:

cp config.ru.example config.ru

Copy config/config.yml.example:

cp config/config.yml.example config/config.yml

Run with unicorn:

bundle exec unicorn

Loading new staff records

It's an old, blunt, violent process.

1. Get the CSVs generated from the sunlightlabs/disbursements process.

2. Put them into the data/ directory and run:

rake load:all

3. Wait for an hour or more as the data is blown away and reloaded from scratch. It's not very efficient, and the website will show incomplete data for that time.

4. Update the 'as of' date displayed on the home page in views/index.erb to the last day of the quarter processed.

data/ expectations

The data/ directory expects four CSV files:

• staffers.csv: Unique staffer names as they appear in the original disbursement data, and any standardizations or corrections to those titles.
• titles.csv: Unique titles as they appear in the original disbursement data, and any standardizations or corrections to those titles.
• offices.csv: Unique office names as they appear in the original disbursement data, and any standardizations or corrections to those titles. Expanded details for each office (committee ID, building, room, telephone number) are added by hand.
• positions.csv: The "join table" that links staffer names, title names, and office names together as a "position" for each quarter.

Why not the Senate?

The Senate's expenditure data is published every 6 months, can be quite out of date, and is much more difficult to parse than the House's.

Our parser for the House is here. The Senate parser is still experimental, but data can be found here.

License

Currently GPLv3.

staffers's Issues

Mislabeled employee?

Someone appears to have personal knowledge of a mistake:
http://getsatisfaction.com/sunlightfoundation/topics/house_staff_list_possible_bug_office_of_finance_vs_cao

If this is true, we ought to verify in some formal way before making a change.

Format quarters

e.g. "2009 4th Quarter"

Search by title

Standardized title, in a dropdown.
Create a Titles collection.

Staffer page

Show, for each quarter, the title and member they worked for.

Order quarters descending.

Set up Unicorn

For both deploy environments.

Display original titles

Of positions, on lists of staffers. Use some sort of hoverable question mark with a tooltip.

Load in 2010Q3

New data release.

Missing lines, beginning with blank name

The logic to display a line isn't quite correct, and the first result should show a name:
http://staffers.sunlightfoundation.com/staffers?quarter=&firstname=&lastname=&title=&state=AL&party=

JSON API

For every view that has a CSV download, make a JSON endpoint.

Split out titles with "and" into multiple positions

This means:

1. On the process of generating titles.csv, if a title has an "and" (with spaces around it), split it out into two titles and treat them separately, UNLESS it appears already in titles.csv (this would be because of manual intervention to assert that the use of "and" is part of one title).

2. In the process of loading in the data to the staffer directory, if a position's original title has an "and", split it up and load in a position for each constituent title.

And one-time:

3. Delete all titles from titles.csv that currently have an "and" in them, as otherwise they will be interpreted as correct.

Search by last name of member

Must match exactly, case-insensitively.

If only one result, bring right to it.

Slugs for URLs

Use Mongoid::Slug after switching to Mongoid. Base it on each office's name, and assign during data load time.

All objects must end up with the same predictable URL on each data reload.

Allow a user to select multiple states

Turn the select box into a multi-select, and use an $or clause or something on the backend to support it.

Remove firstname_search and lastname_search

Verify they aren't needed, and scrap them.

Office pages

A page that lists all legislators, one for all committees, and one for all regular offices.

Search page

Revamp so it works roughly according to plan, and so that each search is scoped entirely to a quarter.

CSV download links

From every search results page and on every office and staffer results page.

Make data loading less destructive

Wherever we can do an update in place, and not a #delete_all, do so.

Put site inside a Sunlight Foundation frame

Similar to Clearspending.

Office page

Show, for each quarter, the staffers in their office and their titles, with links to each one.

Show each job for a legislator on the search results page

Each job for the given quarter, stacked above each other, without duplicating the staffer name.

Duplicate titles for staff members

Could be a data issue. Regardless, make sure that if the standardized title name comes out to the same, that the position isn't duplicated.

Example:
http://staffers.sunlightfoundation.com/legislator/Y000033

Unicode bug on JCT phone number

Probably a simple data fix.

http://staffers.sunlightfoundation.com/committee/JSTX

also:
http://staffers.sunlightfoundation.com/staffer/andrea-m-del-aguila

Add 2010Q1 data

Bring it up to date.

Switch to Mongoid

Might as well do it now.

Show whether member is currently in office

On both the office list (highlighting perhaps) and on the landing page for that office.

Sort staffers secondarily by first name

If you're gonna sort by last name, sort secondarily by first.

Also, display "Last name, First name"

Set up production environment

sunlightfoundation.com/staffers somehow

Resolve staffers.sunlightfoundation.com

DNS issues.

Rewrite data loader to use new CSV formats

4 CSVs in different formats.

Positions CSV should be indexed by standardized names, not original names

For human readability and sensibility, use standardized names to link everything together.

Load in new dataset from policy team

Latest canonical data.

Set up Bundler

For dependency management.

Change "---" to "All" in dropdowns

More clear.

Fix it so requiring config/environment.rb is all that is necessary

Right now you have to require 'staffers' which also tries to run Sinatra.

Ensure column widths stay the same

Shorter staff titles in a quarter make for variable widths:
http://staffers.sunlightfoundation.com/office/rc-training-program-devel

Let user search by list of unique titles

Through a popup box or something.

Some missing legislators

Example: Barbara Lee, L000551

Allow table to be sortable

Sort on viewable columns, even if the content itself is hidden in some cases. Might need to start duplicating the name in the HTML and marking it as display: none, so that the tablesorter plugin has a way of accessing the sortable attribute in each row.

Rename "Staffer" to "Staff" in title

Copy change.

FAQ page

Copy.

Dropdown should start with "Most recent quarter (____)"

With the most recent quarter in parentheses. The rest can stay just the quarter.

Search by last name of staffer

Needs to match exactly, but case-insensitively.

Basic design

A basic design plan and styling for the search form and display of the daa.

Set up for Capistrano deploys

For convenience.

About page

Copy.

Title search should use subset of common titles

Instead of all titles in the system.

"Report a Problem"

Through GetSatisfaction.

Look into why I can't find any shared employees

I think positions might be getting collapsed or overridden somewhere.

Link to state and district from office page for members

To see other members from that state, or from that district.