eglet27/scraper-and-searcher

Crawls specific list of websites, scrapes useful data using scrapy, stores and searches elastic search and displays on a simple GUI using flask and Jinja.

Scraper-and-Searcher

Crawls specific list of websites, scrapes useful data using scrapy, stores and searches elastic search and displays on a simple GUI using flask and Jinja.

Steps:

1. Install Scrapy in python.
2. CRAWLING : Run the spider quotes_spider.py using the command "scrapy crawl quotes" on command prompt form the tutorial folder.
3. SCRAPING : Run the spider scraping_data using command "scrapy crawl data_scraper2".
4. Install flask,elastic search for python using pip.
5. Install elastic search on your computer from https://www.elastic.co/downloads/elasticsearch and run elastic search in command prompt.
6. Run main.py present in user_interface folder.
7. Open the link provided in the output.
8. Start Searching!