mjsir911/va_voter_data_map

Prerequisities

To run, need a python3 installation with virtualenv installed on it and found in $PATH.

Phantomjs also needs to be intalled on the host machine, and found in $PATH

for a debian machine:

# apt install virtualenv
# apt install phantomjs

Installation

make build

Run

make scrape

If you have questions email: [email protected]

va_voter_data_map's Issues

Error Output when tried to install

test -d venv || virtualenv venv --python=python3
/bin/sh: 1: virtualenv: not found
Makefile:16: recipe for target 'venv/bin/activate' failed
make: *** [venv/bin/activate] Error 127

Program Froze Near the end

When I tried to test the scraper it got near the end and froze. I waited about 3 minutes before stopping the program at which point it output this error: https://pastebin.com/hG7FjiGD

FileNotFoundError : Phantom.js

$ make scrape
venv/bin/python src/metadata_scraper.py
Traceback (most recent call last):
File "/home/dsmall/va_voter_data_map/venv/lib/python3.5/site-packages/selenium/webdriver/common/service.py", line 74, in start
stdout=self.log_file, stderr=self.log_file)
File "/usr/lib/python3.5/subprocess.py", line 947, in init
restore_signals, start_new_session)
File "/usr/lib/python3.5/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg)
FileNotFoundError: [Errno 2] No such file or directory: 'include/phantomjs/phantomjs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "src/metadata_scraper.py", line 124, in
driver = webdriver.PhantomJS(DRIVER_PATH)
File "/home/dsmall/va_voter_data_map/venv/lib/python3.5/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 52, in init
self.service.start()
File "/home/dsmall/va_voter_data_map/venv/lib/python3.5/site-packages/selenium/webdriver/common/service.py", line 81, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'phantomjs' executable needs to be in PATH.

Makefile:7: recipe for target 'scrape' failed
make: *** [scrape] Error 1

Failed twice in same spot

Running: make scrape, the script starts downloading but stopped twice in the same spot, failing to make progress:

downloading http://historical.elections.virginia.gov/elections/download/45154/precincts_include:1/
response<http.client.HTTPResponse object at 0x7fc636461a90>
downloading http://historical.elections.virginia.gov/elections/download/45154/precincts_include:0/
response<http.client.HTTPResponse object at 0x7fc6364670f0>

First Page

The scraper skips the first page when going through each year, making it miss out on Presidential Elections and others.