newsdev/elex

A wrapper for the AP v2 Elections API.

elex's Issues

Denormalization

Enough to do joins in the event someone would want to do that.

• Denormalize some Election fields into Race.
• Denormalize some Race and Election fields into ReportingUnit and Candidate

`nextrequest` deosn't include the `apiKey` parameter.

• nextrequest in JSON doesn't include the apiKey parameter, for some reason.

Bug: `Candidate` creation uses wrong attribute.

Currently, we set Candidate objects using the candidateid which is actually the least unique of the three identifiers.

Duplicate race objects; one reporting unit per race?

If I'm using &level=reportingUnit&officeID=G, I should only see a single race that contains many reporting units, and those reporting units should contain many candidates.

Instead, the race object is duplicated and there is only a single reporting unit contained in each race.

Needs a logo.

Like this.

Should parse out ballot positions.

This is going to be tricky; they shouldn't be stored as Candidate objects though the AP does. Would prefer to store them as a separate model before inserting them into the DB.

Tricky: Needs to get a description / common name from race.description and race.seatname. Will have to denormalize that down to the ReportingUnit for it to work.

• Create model for BallotPosition.
• Parse BallotPositions out of a ReportingUnit's candidates array.
• Denormalize race.description and race.seatname down to ReportingUnit.
• Denormalize reportingunit.description and reportingunit.seatname to a BallotPosition but ignore for Candidate.

Bug: `Election.get_race_objects()` fails with initialization data

ERROR: test_json_shape (tests.test_race.TestRaceInitialization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_race.py", line 70, in setUp
    self.race_objs = e.get_race_objects(self.raw_races)
  File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 412, in get_race_objects
    payload.append(Race(**r))
  File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 290, in __init__
    self.set_candidates()
  File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 81, in set_candidates
    candidate_dict['statename'] = STATE_ABBR[getattr(self, 'statepostal')]
AttributeError: 'Race' object has no attribute 'statepostal'

----------------------------------------------------------------------

BUG: get_elections failed because get no longer class method

Follow nextrequest link and load partial updates.

nextrequest features an URL with a timestamp that will return partial results as a diff between this state and the previous state.

• Get an URL that can be used to test.
• Load partial models in the DB; with update queries?
• Run aggregations each time a load finishes.
• Make assumptions about timing.

Tests

Anywhere we do math.

• Aggregation: sums.
• Aggregation: percentages.

Parsing layer.

• Correct number of races.
• Correct number of reporting units.
• Correct number of candidate reporting units.

Anywhere we make data.

• initialization_data
• is_ballot_position
• Unique BallotPosition creation.
• Unique Candidate creation.

Winners.

• Detect an AP winner.
• Ask about edge cases.
• Default to manual call if necessary.

turn requirements into a file and read into setup.py

How does a cronned loader implement this conditional requests model?

Need to figure out a practical application of this if we're not using a cache between us and the API endpoint, e.g., just hitting it x times per minute with a cron job.

statePostal attribute should be on race not reportingUnit

It changes after the initialization data? Not sure why this is moved down a level in the tree, so to speak.

Bug: `Election.get_units()` fails to return CandidateReportingUnits for `omitResults=True`

When we're using initialization data, Election.get_units() fails to loop over election.candidates, expecting them to be nested inside of reporting units like they are for results data.

Ballot position parsing is fragile.

Currently using raceType == "Ballot Issue" but this is probably fragile?

Emailed Tracy about getting a list of possible values for both the raceType and raceTypeID fields.

BUG: failing candidate reporting units test on CLI branch

======================================================================
FAIL: test_initialization_data_number_of_races (tests.test_race.TestRaceInitialization)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/deads/.virtualenvs/nyt-ap-elections/src/nyt-ap-elections/tests/test_race.py", line 87, in test_initialization_data_number_of_races
    self.assertEqual(len(self.candidate_reporting_units), 20)
AssertionError: 6 != 20

----------------------------------------------------------------------

should election data objects have a serialize method?

they'll probably need a serialize() and serialize_flat() method (e.g. without hierarchy, to support CSV), unless we assume all serialization should be flat.

write tests for CLI app

helloooo subprocess

Bug: `Candidate.polid` should be None if it is 0.

AP returns 0 for politicians with no AP NPID. We should set that to None.

JSON or XML?

XML request is:

• Smaller
• Downloads faster

JSON:

• Slightly faster to parse

XML: 50 loops; 0.0778681230545s per loop.
JSON: 50 loops; 0.0713178396225s per loop.

BUG (?): init-reporting-units isn't returning anything for '2015-11-03'

Parsing functions aren't modular enough (to test)

Given a JSON dump, we should have a function that can pull:

• candidates
• ballot issue
• reporting units
• candidate results

We don't have the discrete functions to test with.

Should parse out Candidate objects.

And not just what the AP defines as "Candidates" because those are actually more like CandidateResults, e.g., a single candidate's performance in a single reporting area.

• Parse out only true human candidates, e.g., ignore ballot positions.

schematize stuff in docs into json schema

Keys different at different times when requesting the same race.

Sometimes, a request for a race is missing several fields, including test on the races, and reportingUnits. Instead, there's a new key called candidates and that's nothing like what's in the docs.

Bug: Race missing `national` attribute.

ERROR: test_race_get_units_construction (tests.test_race.TestRaceParsing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_race.py", line 59, in test_race_get_units_construction
    self.assertEqual(race.national, True)
AttributeError: 'Race' object has no attribute 'national'

----------------------------------------------------------------------```

make app into simple CLI utility

low priority for now, but might be good MVP candidate

Documentation.

Flow

• Development
• Production: Pre-election day
• Production: On election day
• Production: Once results are coming in
• Production: When we've finished taking results
• Production: When AP stops tabulation

Information Architecture

• What is included?
• What is missing?
• Why did we pick our classes?

Customization

• Recipe book

Refactor 'Election.get_races` into several pieces that are testable.

• Election.get_raw_races()
• Election.get_race_objects()

Create CandidateRace / BallotIssueRace abstractions for aggregating results

Because why do an expensive query on the backside when we can do a slightly expensive aggregation on insert?

merge CLI branch into master

blocked by #47

allow file to be specified on command line instead of date

elex.loader should contain code to load Python objects into a database

• __init__.py should contain the connection info, pulled from the environment.
• Initially supports a Postgres database with Peewee.
• Should drop, create tables.
• Should (somehow) use attributes on the elex.parser.api classes without duplicating code.
• Should use bulk inserts. Consider atomic transactions, pyscopg2 copy_from function.

Bug: Why isn't this a ballot issue?

Codify maps / tables at the bottom of the AP docs.

Might be useful.

Build out aggregates.

There will be more. But these certainly exist:

• Race total votes.
• Reporting unit total votes.
• Reporting unit percentage of total race votes.
• Candidate percentage of total race votes.
• Candidate percentage of total reporting unit votes.

json parser needs single row check

Abstract out the percentage aggregation into a function.

Right now it's all handwritten in the demo __main__.py

Bug: Test for `Race.initialization_data` fails.

Relies on a request to the AP, e.g., omitResults=True in the params.

Tests for AP networking things

• utils.api_request()
• Illegal params
• Non-existent dates
• 5xx?

make all election data objects json-serializable

what exactly is meant by "ballot positions" in the current model?

Make fake JSON with example varied races

Run tests off of test data and not API

Because if we do it in production we will run out of API requests.

There is no way to link candidates back to races in current data

candidates don't have an identifier to join back to races or reporting units; meanwhile, races and reporting units can be joined on raceid

outfit with proper logging

Bug: CandidateReportingUnit missing attribute `incumbent`.

ERROR: test_candidate_reporting_unit_attribute_construction (tests.test_candidate_reporting_unit.TestCandidateReportingUnitParsing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_candidate_reporting_unit.py", line 51, in test_candidate_reporting_unit_attribute_construction
    self.assertEqual(cru.incumbent, True)
AttributeError: 'CandidateReportingUnit' object has no attribute 'incumbent'

re-factor main data objects

Some of the logic would be better off abstracted. e.g.:

races, reporting_units, candidate_reporting_units = election.get_units(race_objs)

It would be nice if the magic happened behind the scenes, and you could just do:

races = election.races

Set environment variables to record to flat files or a JSON store.

User API should be something like export ELEX_RECORD=mongodb or export ELEX_RECORD=flat and then matching exports for ELEX_MONGODB_URL or ELEX_RECORD_DIR.

• Watch environment vars to record JSON.
• Use environment vars to direct types of recording.
• Use environment vars to pass config to the recording method.

talk with AP about polID