michaelevil/douban_crawl

a python scrapy project for crawling douban user data.一个抓取豆瓣用户数据的python scrapy项目

douban_crawl

a python scrapy project for crawling douban user data.一个抓取豆瓣用户数据的python scrapy项目

functions

• crawl user relationship tree
• crawl user basic data(according relationship tree，the same below)
• crawl user interests(want to read, reading, read)
• crawl item(books, movies, musics) data
• crawl user commits and reviews

features

• using a map database to show data integrity

usage

to be continue...

douban_crawl's Issues

不能正确的运行

执行doubandetail出现Traceback (most recent call last):
File "E:\Ana\lib\site-packages\scrapy\utils\defer.py", line 102, in iter_errback
yield next(it)
File "E:\Ana\lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 30, in process_spider_output
for x in result:
File "C:\Users\cg\douban_crawl\douban\middlewares.py", line 37, in process_spider_output
for i in result:
File "E:\Ana\lib\site-packages\scrapy\spidermiddlewares\referer.py", line 339, in
return (_set_referer(r) for r in result or ())
File "E:\Ana\lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 37, in
return (r for r in result or () if _filter(r))
File "E:\Ana\lib\site-packages\scrapy\spidermiddlewares\depth.py", line 58, in
return (r for r in result or () if _filter(r))
File "C:\Users\cg\douban_crawl\douban\spiders\doubandetail.py", line 83, in queue_requests
mc.execute("select user_id from user_map where user_book_do = 0")
这种异常。
执行doubanuser出现2018-08-16 18:13:29 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://accounts.douban.com/login?alias=&redir=https%3A%2F%2Fwww.douban.com&source=None&error=1001> from <POST https://www.douban.com/login>
2018-08-16 18:13:30 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://accounts.douban.com/login?alias=&redir=https%3A%2F%2Fwww.douban.com&source=None&error=1001> (referer: https://www.douban.com)
2018-08-16 18:13:30 [doubanuser] DEBUG: login:failed!
2018-08-16 18:13:30 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.douban.com/login> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)登录失败异常