systep/skyline-census-api

A Scraping-Based API to US Census Data

census scrapers

This project contains some logic to be able to scrape quick facts from the US census website. The US census contains both CSV versions of census data and a website where some insights have been already derived from digesting that data.

The following project exposes an API to the quick info based on <city, state> input.

The project basically does the following to achieve this API:

• Mapping an address string to FIPS code to be able to search for data
• Scraping census web pages
• Cleaning the scraped data to provide a readable JSON

running the project

To run the project:

cd scrapers && swagger project start

To edit the project's API endpoints:

cd scrapers && swagger project edit

sample request / response:

request:

curl http://localhost:10010/population?fullCityName=phoenix&shortStateName=az

response:

{
	"Population estimates, July 1, 2016,  (V2016)": "NA",
	"Population estimates, July 1, 2015,  (V2015)": "1,563,025",
	"Population estimates base, April 1, 2010,  (V2016)": "NA",
	"Population estimates base, April 1, 2010,  (V2015)": "1,447,624",
	"Population, percent change - April 1, 2010 (estimates base) to July 1, 2016,  (V2016)": "NA",
	"Population, percent change - April 1, 2010 (estimates base) to July 1, 2015,  (V2015)": "8.0%",
	"Population, Census, April 1, 2010": "1,445,632",
	"Persons under 5 years, percent, July 1, 2015,  (V2015)": "X",
	"Persons under 5 years, percent, April 1, 2010": "8.3%",
	"Persons under 18 years, percent, July 1, 2015,  (V2015)": "X",
	"Persons under 18 years, percent, April 1, 2010": "28.2%",
	"Persons 65 years and over, percent,  July 1, 2015,  (V2015)": "X",
	"Persons 65 years and over, percent, April 1, 2010": "8.4%",
	"Female persons, percent,  July 1, 2015,  (V2015)": "X",
	"Female persons, percent, April 1, 2010": "49.8%",
	"White alone, percent, July 1, 2015,  (V2015)": "X",
	"White alone, percent, April 1, 2010": "65.9%",
	"Black or African American alone, percent, July 1, 2015,  (V2015)": "X",
	"Black or African American alone, percent, April 1, 2010": "6.5%",
	"American Indian and Alaska Native alone, percent, July 1, 2015,  (V2015)": "X",
	"American Indian and Alaska Native alone, percent, April 1, 2010": "2.2%",
	"Asian alone, percent, July 1, 2015,  (V2015)": "X",
	"Asian alone, percent, April 1, 2010": "3.2%",
	"Native Hawaiian and Other Pacific Islander alone, percent, July 1, 2015,  (V2015)": "X",
	"Native Hawaiian and Other Pacific Islander alone, percent, April 1, 2010": "0.2%",
	"Two or More Races, percent, July 1, 2015,  (V2015)": "X",
	"Two or More Races, percent, April 1, 2010": "3.6%",
	"Hispanic or Latino, percent, July 1, 2015,  (V2015)": "X",
	"Hispanic or Latino, percent, April 1, 2010": "40.8%",
	"White alone, not Hispanic or Latino, percent, July 1, 2015,  (V2015)": "X",
	"White alone, not Hispanic or Latino, percent, April 1, 2010": "46.5%",
	"Veterans, 2011-2015": "74,708",
	"Foreign born persons, percent, 2011-2015": "20.0%",
	"Housing units,  July 1, 2015,  (V2015)": "X",
	"Housing units, April 1, 2010": "590,149",
	"Owner-occupied housing unit rate, 2011-2015": "53.0%",
	"Median value of owner-occupied housing units, 2011-2015": "$163,400",
	"Median selected monthly owner costs -with a mortgage, 2011-2015": "$1,349",
	"Median selected monthly owner costs -without a mortgage, 2011-2015": "$425",
	"Median gross rent, 2011-2015": "$884",
	"Building permits, 2015": "X",
	"Households, 2011-2015": "525,610",
	"Persons per household, 2011-2015": "2.85",
	"Living in same house 1 year ago, percent of persons age 1 year+, 2011-2015": "80.9%",
	"Language other than English spoken at home, percent of persons age 5 years+, 2011-2015": "37.0%",
	"High school graduate or higher, percent of persons age 25 years+, 2011-2015": "80.7%",
	"Bachelor's degree or higher, percent of persons age 25 years+, 2011-2015": "26.7%",
	"With a disability, under age 65 years, percent, 2011-2015": "7.2%",
	"Persons  without health insurance, under age 65 years, percent": "22.2%",
	"In civilian labor force, total, percent of population age 16 years+, 2011-2015": "65.5%",
	"In civilian labor force, female, percent of population age 16 years+, 2011-2015": "59.3%",
	"Total accommodation and food services sales, 2012 ($1,000)": "3,479,625",
	"Total health care and social assistance receipts/revenue, 2012 ($1,000)": "10,967,040",
	"Total manufacturers shipments, 2012 ($1,000)": "12,978,949",
	"Total merchant wholesaler sales, 2012 ($1,000)   ": "31,193,680",
	"Total retail sales, 2012 ($1,000)": "18,448,380",
	"Total retail sales per capita, 2012": "$12,392",
	"Mean travel time to work (minutes), workers age 16 years+, 2011-2015": "24.8",
	"Median household income (in 2015 dollars), 2011-2015": "$47,326",
	"Per capita income in past 12 months (in 2015 dollars), 2011-2015": "$24,231",
	"Persons in poverty, percent": "23.1%",
	"Total employer establishments, 2014": "X",
	"Total employment, 2014": "X",
	"Total annual payroll, 2014 ($1,000)": "X",
	"Total employment, percent change, 2013-2014": "X",
	"Total nonemployer establishments, 2014": "X",
	"All firms, 2012": "124,033",
	"Men-owned firms, 2012": "63,479",
	"Women-owned firms, 2012": "44,294",
	"Minority-owned firms, 2012": "39,596",
	"Nonminority-owned firms, 2012": "78,757",
	"Veteran-owned firms, 2012": "10,784",
	"Nonveteran-owned firms, 2012": "106,575",
	"Population per square mile, 2010": "2,797.8",
	"Land area in square miles, 2010": "516.70",
	"Metropolitan or Micropolitan Statistical Area": "None",
	"FIPS Code": "0455000"
}