baronrustamov/seleniumcrawler

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

seleniumcrawler

This is a Webcrawler based on Scrapy and Selenium frameworks

This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directions Dublin-Liverpool / Liverpool-Dublin with departure tomorrow and returning in 3 days.

HOW TO EXECUTE:

Is required an environment with the following requirements installed: -Python 2.7 -Scrapy 0.18 -Selenium web-drivers

To execute the crawler the following command should be executed from the project path

scrapy crawl crawlermate_selenium -a category=[dublin or liverpool] -o [filename] -t json

for an example, to generate tickets for dublin to liverpool on items.json file you should execute scrapy crawl crawlermate_selenium -a category=dublin -o items.json -t json

References:

http://docs.seleniumhq.org/ http://http://scrapy.org/