tpeng/weibosearch

A distributed Sina Weibo Search spider base on Scrapy and Redis.

WeiboSearch

A distributed Sina Weibo Search spider base on Scrapy and Redis.

tpeng [email protected]

Installation

$ sudo apt-get install mysql-server
$ sudo apt-get install redis-server
$ sudo apt-get install python-mysqldb
$ sudo pip install -r requirements.txt

Usage

1. put your keywords in items.txt
2. scrapy crawl weibosearch -a username=your_weibo_account -a password=your_weibo_password
3. add another spider with scrapy crawl weibosearch -a username=another_weibo_account -a password=another_weibo_password

or

1. scrapy crawl weibosearch -a username=your_weibo_account -a password=your_weibo_password -a savedb=False to avoid save to mysql db. (easier for experiment)

weibosearch's Issues

请教：weibo如何识别程序已经登录？

请教一个跟项目本身相关度不大的问题：具体到weibosearch这个项目，weibo.com是如何识别程序已经登录的？我在按照你的代码写了login程序后，启动scrapy，发现在访问一个微博页面时被redirect到http://passport.weibo.com/visitor/visitor?a=enter&url=http%3A%2F%2Fweibo.com%2F2887339314%2Finfo&_rand=1421933752.2223

请问我这里哪里出错了呢？