Giter Site home page Giter Site logo

Comments (8)

ResolveWang avatar ResolveWang commented on May 22, 2024

可能是 if server_data['showpin']: 这个语句的问题

替换成下面代码试试呢?
if server_data.get('showpin', None)

from weibospider.

Joker-zc avatar Joker-zc commented on May 22, 2024

好了,谢谢大佬,不过执行搜索后又有报错了。。
[2018-04-18 16:20:27,986: INFO/MainProcess] Received task: tasks.search.search_keyword[27f9ef57-4963-4641-883b-f27f866f0f47]
2018-04-18 16:20:27 - crawler - INFO - We are searching keyword "快手"
[2018-04-18 16:20:27,989: INFO/ForkPoolWorker-1] We are searching keyword "快手"
2018-04-18 16:20:27 - crawler - INFO - the crawling url is http://s.weibo.com/weibo/%E5%BF%AB%E6%89%8B&scope=ori&suball=1&page=1
[2018-04-18 16:20:27,990: INFO/ForkPoolWorker-1] the crawling url is http://s.weibo.com/weibo/%E5%BF%AB%E6%89%8B&scope=ori&suball=1&page=1
2018-04-18 16:20:27 - crawler - ERROR - failed to crawl http://s.weibo.com/weibo/%E5%BF%AB%E6%89%8B&scope=ori&suball=1&page=1,here are details:'NoneType' object is not subscriptable, stack is File "/homen_gu/Desktop/weibospider-master/decorators/decorators.py", line 17, in time_limit
return func(*args, **kargs)

[2018-04-18 16:20:27,996: ERROR/ForkPoolWorker-1] failed to crawl http://s.weibo.com/weibo/%E5%BF%AB%E6%89%8B&scope=ori&suball=1&page=1,here are details:'NoneType' object is not subscriptable, stack is File "/homen_gu/Desktop/weibospider-master/decorators/decorators.py", line 17, in time_limit
return func(*args, **kargs)

2018-04-18 16:20:27 - crawler - WARNING - No search result for keyword 快手, the source page is
[2018-04-18 16:20:27,998: WARNING/ForkPoolWorker-1] No search result for keyword 快手, the source page is
[2018-04-18 16:20:27,998: INFO/ForkPoolWorker-1] Task tasks.search.search_keyword[27f9ef57-4963-4641-883b-f27f866f0f47] succeeded in 0.009885783000072479s: None

from weibospider.

ResolveWang avatar ResolveWang commented on May 22, 2024

检查一下你的redis中是否有cookies,然后手动测试一下,确认你的账号是否可以用于搜索

from weibospider.

Joker-zc avatar Joker-zc commented on May 22, 2024

redis里没有cooikes,
(WeiboSpider)lin_gu@ww:~/Desktop/weibospider-master$ ./redis-3.2.9/src/redis-cli
127.0.0.1:6379> auth weibospider
OK
127.0.0.1:6379> keys *
(empty list or set)
127.0.0.1:6379>

帐号高级搜索是可以用的。
配置如下
redis:
host: 127.0.0.1
port: 6379
password: 'weibospider'
cookies: 1 # store and fetch cookies
# store fetched urls and results,so you can decide whether retry to crawl the urls or not
urls: 2
broker: 5 # broker for celery
backend: 6 # backed for celery
id_name: 8 # user id and names,for repost info analysis. Could be safely deleted after repost tasks
# expire_time (hours) for redis db2, if they are useless to you, you can set the value smaller
expire_time: 48
# redis sentinel for ha. if you neet it, just add sentinel host and port below the sentinel args,like this:
###############################
#sentinel: #
# - host: 2.2.2.2 #
# port: 26379 #
# - host: 3.3.3.3 #
# port: 26379 #
# #
###############################
sentinel: ''
master: '' # redis sentinel master name, if you don't need it, just set master: ''
socket_timeout: 5 # sockt timeout for redis sentinel, if you don't need it, just set master: ''

from weibospider.

ResolveWang avatar ResolveWang commented on May 22, 2024

你确定你用的1.7.2?貌似1.7.2默认不是采用

###############################
#sentinel: #
# - host: 2.2.2.2 #
# port: 26379 #
# - host: 3.3.3.3 #
# port: 26379 #
# #
###############################

这个注释风格的。

login.pyget_session函数的return session之前打印一下当前cookies呢?

Cookies.store_cookies(name, session.cookies.get_dict())
print(session.cookies.get_dict())   # 加这句话,然后观察登录的时候是否有cookies打印出来
return session

from weibospider.

Joker-zc avatar Joker-zc commented on May 22, 2024

我按照这个lssues改了之后可以登录并打印出cookies,但是redis里没有cookies
注释风格应该是拷贝的问题。。。我在另一台电脑上用QQ拷过来的。。

from weibospider.

ResolveWang avatar ResolveWang commented on May 22, 2024

1.7.2的配置文件不长这个样。要不重新在releases中下载稳定版的代码跑跑?

或者你懂Python的话,调试一下redis_db.py吧,看看是不是哪里有问题

from weibospider.

Joker-zc avatar Joker-zc commented on May 22, 2024

嗯,我重新下一个跑跑吧,谢谢大佬啦~

from weibospider.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.