newsdev / elex Goto Github PK
View Code? Open in Web Editor NEWA wrapper for the AP v2 Elections API.
License: Other
A wrapper for the AP v2 Elections API.
License: Other
Enough to do joins in the event someone would want to do that.
Election
fields into Race
.Race
and Election
fields into ReportingUnit
and Candidate
nextrequest
in JSON doesn't include the apiKey
parameter, for some reason.Currently, we set Candidate
objects using the candidateid
which is actually the least unique of the three identifiers.
This is going to be tricky; they shouldn't be stored as Candidate objects though the AP does. Would prefer to store them as a separate model before inserting them into the DB.
Tricky: Needs to get a description / common name from race.description and race.seatname. Will have to denormalize that down to the ReportingUnit for it to work.
ERROR: test_json_shape (tests.test_race.TestRaceInitialization)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_race.py", line 70, in setUp
self.race_objs = e.get_race_objects(self.raw_races)
File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 412, in get_race_objects
payload.append(Race(**r))
File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 290, in __init__
self.set_candidates()
File "/Users/jbowers/src/nyt/nyt-ap-elections/elex/parser/api.py", line 81, in set_candidates
candidate_dict['statename'] = STATE_ABBR[getattr(self, 'statepostal')]
AttributeError: 'Race' object has no attribute 'statepostal'
----------------------------------------------------------------------
nextrequest
features an URL with a timestamp that will return partial results as a diff between this state and the previous state.
update
queries?initialization_data
is_ballot_position
It changes after the initialization data? Not sure why this is moved down a level in the tree, so to speak.
When we're using initialization data, Election.get_units()
fails to loop over election.candidates, expecting them to be nested inside of reporting units like they are for results data.
Currently using raceType == "Ballot Issue" but this is probably fragile?
Emailed Tracy about getting a list of possible values for both the raceType and raceTypeID fields.
======================================================================
FAIL: test_initialization_data_number_of_races (tests.test_race.TestRaceInitialization)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/deads/.virtualenvs/nyt-ap-elections/src/nyt-ap-elections/tests/test_race.py", line 87, in test_initialization_data_number_of_races
self.assertEqual(len(self.candidate_reporting_units), 20)
AssertionError: 6 != 20
----------------------------------------------------------------------
they'll probably need a serialize() and serialize_flat() method (e.g. without hierarchy, to support CSV), unless we assume all serialization should be flat.
helloooo subprocess
AP returns 0 for politicians with no AP NPID. We should set that to None
.
XML request is:
JSON:
XML: 50 loops; 0.0778681230545s per loop.
JSON: 50 loops; 0.0713178396225s per loop.
Given a JSON dump, we should have a function that can pull:
We don't have the discrete functions to test with.
And not just what the AP defines as "Candidates" because those are actually more like CandidateResults, e.g., a single candidate's performance in a single reporting area.
Sometimes, a request for a race is missing several fields, including test on the races, and reportingUnits. Instead, there's a new key called candidates and that's nothing like what's in the docs.
ERROR: test_race_get_units_construction (tests.test_race.TestRaceParsing)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_race.py", line 59, in test_race_get_units_construction
self.assertEqual(race.national, True)
AttributeError: 'Race' object has no attribute 'national'
----------------------------------------------------------------------```
low priority for now, but might be good MVP candidate
Election.get_raw_races()
Election.get_race_objects()
Because why do an expensive query on the backside when we can do a slightly expensive aggregation on insert?
blocked by #47
__init__.py
should contain the connection info, pulled from the environment.elex.parser.api
classes without duplicating code.copy_from
function.Might be useful.
There will be more. But these certainly exist:
Right now it's all handwritten in the demo __main__.py
Relies on a request to the AP, e.g., omitResults=True
in the params.
utils.api_request()
Because if we do it in production we will run out of API requests.
candidates don't have an identifier to join back to races or reporting units; meanwhile, races and reporting units can be joined on raceid
ERROR: test_candidate_reporting_unit_attribute_construction (tests.test_candidate_reporting_unit.TestCandidateReportingUnitParsing)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/jbowers/src/nyt/nyt-ap-elections/tests/test_candidate_reporting_unit.py", line 51, in test_candidate_reporting_unit_attribute_construction
self.assertEqual(cru.incumbent, True)
AttributeError: 'CandidateReportingUnit' object has no attribute 'incumbent'
Some of the logic would be better off abstracted. e.g.:
races, reporting_units, candidate_reporting_units = election.get_units(race_objs)
It would be nice if the magic happened behind the scenes, and you could just do:
races = election.races
User API should be something like export ELEX_RECORD=mongodb
or export ELEX_RECORD=flat
and then matching exports for ELEX_MONGODB_URL
or ELEX_RECORD_DIR
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.