naqushab/searchenginescrapy

Scrape data from Google.com, Bing.com, Baidu.com, Ask.com, Yahoo.com, Yandex.com

SearchEngineScrapy - Scrape data from Google.com, Bing.com, Baidu.com, Ask.com, Yahoo.com, Yandex.com,

Intro

SearchEngineScrapy is a web crawler and scraper for scraping data off various search engines such as Google.com, Bing.com, Yahoo.com, Ask.com, Baidu.com, Yandex.com It is based on Python Scrapy project and is developed using Python 2.7

Setup

    virtualenv --python="2" env
    env/bin/activate
    git clone https://github.com/naqushab/SearchEngineScrapy.git
    cd SearchEngineScrapy
    pip install -r requirements.txt

Usage

Prefix : -a

Params

searchQuery="" [Required Parameter]

searchEngine="" [Options: Google/Bing/Ask/Yandex/Baidu/Yahoo] [Optional Parameter] [Default: Bing]

pages= [Number of pages to crawl] [Optional Parameter : Default- 3]

Prefix : -o

Params

[Output the resulta to a file] [Optional Parameter] [Supported:json/jsonl/csv/xml]

Examples

scrapy crawl SearchEngineScrapy -a searchQuery="I'm Batman"
scrapy crawl SearchEngineScrapy -a searchQuery="I'm Batman" -o filename.json 
scrapy crawl SearchEngineScrapy -a searchQuery="I'm Batman" -a searchEngine="Google" -o filename.xml 
scrapy crawl SearchEngineScrapy -a searchQuery="I'm Batman" -a searchEngine="Google" -a pages=5 -o filename.csv

TODO

• Add support for DDG
• Ability to provide parameter of what to save
• Ability to export to various formats (currently limited to JSON, JSONLINES, CSV, XML)
• Contributing section

searchenginescrapy's Issues

how to solve the problem of " Unknown command: crawl "

i came across this problem:
Scrapy 1.6.0 - no active project

Unknown command: crawl

Use "scrapy" to see available commands

this only work in windows?

i am try install the dependencies and receive this erros in ubuntu:
Could not find a version that satisfies the requirement pywin32>=223 (from pypiwin32->-r requirements.txt (line 2)) (from versions: )
No matching distribution found for pywin32>=223 (from pypiwin32->-r requirements.txt (line 2))
so i am download the complete project to compile
https://github.com/mhammond/pywin32
and recive this
$ python setup.py build
Traceback (most recent call last):
File "setup.py", line 95, in
import _winreg
ImportError: No module named _winreg

so I think that this project only work in windows, please confirm this if I forgot something

I think it missing config.cfg file

C:\ub16_prj\SearchEngineScrapy>scrapy crawl SearchEngineScrapy -a searchQuery="I'm Batman"
Scrapy 1.5.1 - no active project

Unknown command: crawl

Use "scrapy" to see available commands

C:\ub16_prj\SearchEngineScrapy>