saisyam/reviews-scraper

Scrape Reviews from sites like Google, Trip advisor etc

Reviews scraper

This repo contains scrapers to scrape reviews from Google, TripAdvisor etc. You can find the domain specific reviews under the following URLs:

1. Google
2. Trip Advisor - TBD

I am using Google Chrome browser to load the page and do the scrolling. So, you need to have Google Chrome web driver in your path in order to make this scraper work. Make sure you download the version same as your Chrome browser.

I am using Splinter and Beautifulsoup to extract and parsing the HTML respectively.

Webdriver security issue on MacOS

Mac will throw a security error while running chromedriver. To make the chromedriver binary file trusted run the following command:

$ xattr -d com.apple.quarantine <path to chromedriver>

reviews-scraper's Issues

Add proxy support to the scraper

Add proxy support to Google scraper. Read a list of proxies from a file and use a different proxy for each URL.

Build a scraper for tripadvisor

Build a web scraper to extract reviews for the restaurants

Add sentiment for the review text

Add sentiment for the review text using vader sentiment analysis

Remove reviews data files from git repo

Remove the reviews data files from this repo

Update README

Update README file to explain the code better