- What is this?
- [Assumptions] (#assumptions)
- [Installation] (#installation)
- [Run the project] (#run)
- [What to expect] (#what-to-expect)
Armslist is a tool for aggregating listings from Armslist.com, a site used as a marketplace for buying and selling guns.
The following things are assumed to be true in this documentation.
- You are running OSX.
- You are using Python 2.7. (Probably the version that came OSX.)
- You have virtualenv and virtualenvwrapper installed and working.
- You have GNU Parallel installed -- a shell tool to execute multiple commands from standard input simultaneously.
For more details on the technology stack used with the app-template, see our development environment blog post.
This code should work fine in most recent versions of Linux, but package installation and system dependencies may vary.
If you don’t already have GNU Parallel, get it like this:
brew install parallel
Then clone the project:
git clone [email protected]:nprapps/armslist.git
cd armslist
Next, create a virtual environment and install the requirements:
mkvirtualenv armslist
pip install –r requirements.txt
Finally, run the script:
./scrape.sh
The scraper will output the listings into a csv format.
Running script will make two main csv files as well as some extra files that may be handy for testing:
cache/index.csv
is a master index of listings with the URL, state and date. This file is used to scrape each page individually.cache/listings.csv
is the bulk of the data. Each row represents a listing and the associated details.cache/state/1/<statename>
includes the log messages in a file calledstderr
and scraped index page data in a file calledstdout
on a per state basis.
The script could take a long time to run. We recommend using a fairly powerful Amazon EC2 instance in the Virginia/us-east-1 region to run the scraper.