chenjiandongx / async-proxy-pool Goto Github PK

View Code? Open in Web Editor NEW

372.0 372.0 135.0 60 KB

🔅 Python3 异步爬虫代理池

License: MIT License

Python 100.00%

proxy-pool python3

async-proxy-pool's People

Contributors

Stargazers

Watchers

Forkers

sunhailin-leo beanwei bestart rollingkeyboard zzz233 zhangzhaolei duyamin ddingx hrchao disk-kk fashtimedotcom ahshiweiping helianghit songxuecheng csy512889371 shiliuxing cimor zylinkus heliusa shellsec mrcomer windyzjw powerbijay zixuedanxin tdlist giddily machao326 amapoftheworld dahaoniuniu ilizhu jiangming1 followfate noobpythoner znl1087 daodaoliang qw243211462 emaiqi lianghappy coooold wf1314 gavinchiu luoyihua giegloop kingcrab1990 redyjq hlh981029 cced3000 xazaj lixucan hblong cl545740896 anhilo mr-zsy antonyzhao 276077345 qq431169079 tenark judaniubi sdpku flagang lgvcc fieri krpronie ccf19881030 henrylee123 code-cold fdkek freeide liudongdonggoforit sunxiuwen2018 hedou limuitech pgcheukpang lxgfairytail kaidong123 cnxue lstarby logic-luo haoeric casually foxgeek36 iotmentor liaokai kingking888 liangmengmeng930104 moriweiji samson886 xhui28 upen24931251 asushugo inspireling murphykobe xiaogea01 ajdecade jxzft2015 bbkali wz-git guitaoliu jishuzhain chann1n9

async-proxy-pool's Issues

internal server error

起了sanic, 訪問http://localhost:3289/pop失敗

{"description":"Internal Server Error","status":500,"message":"The server encountered an internal error and cannot complete your request."}

看代码不是同时并发多个代理网站

从总的来看，是访问一个代理网站，再访问另一个网站

｀｀｀
for func in all_funcs:
for proxy in func():
redis_conn.add_proxy(proxy)
logger.info("Crawler √ {}".format(proxy）
｀｀｀

额，python requests 代理只用https

目标主机不可达

本机访问没任何问题，但是外部（虚拟机外部）访问不了。
其他应用没问题，所以应该不是网络问题。

用test_proxy测试，成功率不是很高

r = session.get('http://127.0.0.1:3289/count')
r.json()
{'count': '4200'}
抓到了特别多的https,是不是这个影响的呢？

测试代理： http://127.0.0.1:3289/get/10
测试网站： https://zhihu.com
测试次数： 1000
成功次数： 246
失败次数： 754
成功率： 0.246

测试代理： http://127.0.0.1:3289/get/200
测试网站： https://zhihu.com
测试次数： 1000
成功次数： 237
失败次数： 763
成功率： 0.237

Here's another source for more proxies

Site: http://pzzqz.com/
API Doc: https://pzzqz.com/settings/profile

运行server.py报错

[INFO] Goin' Fast @ http://localhost:3289
[ERROR] Unable to start server

OSError: [Errno 99] error while attempting to bind on address ('::1', 3289, 0, 0): cannot assign requested address

请求商务推广合作

作者您好，我们也是一家专业做IP代理的服务商，极速HTTP，我们注册认证会送10000IP(可以帮助您的学者适当薅羊毛试用：) 。想跟您谈谈是否能够达成商业推广上的合作。如果您，有意愿的话，可以联系我，微信：13982004324 谢谢（如果没有意愿的话，抱歉，打扰了）

error

执行python client.py，报错：AttributeError: 'str' object has no attribute 'items'，我的环境是windows10+python3.6

用https的网站来测试，requests会用本机来调用，似乎并不准确

用test_proxy测试时候发现，用一个https的网址来测试，本机的ip被ban了，好像用requests的proxies的时候，只设置http代理的话，https的访问并不会走代理。
我自己加上了一句话才好用
proxy['https'] = proxy['http']

ok 改好了

Originally posted by @chenjiandongx in #2 (comment)

您好，我再次遇到了这个问题，然后手动修改了 requirements 文件中 redis 的版本号，重新安装了一遍 redis 库才好，请修复之

报错了,py3.7.1+centos

[2018-11-27 21:53:37 +0800] [5347] [INFO] Goin' Fast @ http://localhost:3289
[2018-11-27 21:53:37 +0800] [5347] [ERROR] Unable to start server
Traceback (most recent call last):
  File "uvloop/loop.pyx", line 1106, in uvloop.loop.Loop._create_server
  File "uvloop/handles/tcp.pyx", line 88, in uvloop.loop.TCPServer.bind
  File "uvloop/handles/streamserver.pyx", line 89, in uvloop.loop.UVStreamServer                                                                                                                                                             ._fatal_error
  File "uvloop/handles/tcp.pyx", line 86, in uvloop.loop.TCPServer.bind
  File "uvloop/handles/tcp.pyx", line 26, in uvloop.loop.__tcp_bind
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sanic/server.py", line 612, in se                                                                                                                                                             rve
    http_server = loop.run_until_complete(server_coroutine)
  File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1667, in create_server
  File "uvloop/loop.pyx", line 1111, in uvloop.loop.Loop._create_server
OSError: [Errno 99] error while attempting to bind on address ('::1', 3289, 0, 0                                                                                                                                                             ): cannot assign requested address
[2018-11-27 21:53:37 +0800] [5347] [INFO] Server Stopped
[root@yj007 async-proxy-pool]# python3 server.py
[2018-11-27 21:53:47 +0800] [5355] [INFO] Goin' Fast @ http://localhost:3289
[2018-11-27 21:53:47 +0800] [5355] [ERROR] Unable to start server
Traceback (most recent call last):
  File "uvloop/loop.pyx", line 1106, in uvloop.loop.Loop._create_server
  File "uvloop/handles/tcp.pyx", line 88, in uvloop.loop.TCPServer.bind
  File "uvloop/handles/streamserver.pyx", line 89, in uvloop.loop.UVStreamServer                                                                                                                                                             ._fatal_error
  File "uvloop/handles/tcp.pyx", line 86, in uvloop.loop.TCPServer.bind
  File "uvloop/handles/tcp.pyx", line 26, in uvloop.loop.__tcp_bind
OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/sanic/server.py", line 612, in se                                                                                                                                                             rve
    http_server = loop.run_until_complete(server_coroutine)
  File "uvloop/loop.pyx", line 1446, in uvloop.loop.Loop.run_until_complete
  File "uvloop/loop.pyx", line 1667, in create_server
  File "uvloop/loop.pyx", line 1111, in uvloop.loop.Loop._create_server
OSError: [Errno 99] error while attempting to bind on address ('::1', 3289, 0, 0                                                                                                                                                             ): cannot assign requested address
[2018-11-27 21:53:47 +0800] [5355] [INFO] Server Stopped

$ python3 client.py
2018-11-21 09:57:07,169 - Crawler working...
Traceback (most recent call last):
File "client.py", line 7, in
run_schedule()
File "/mnt/d/repos/async-proxy-pool/async_proxy_pool/scheduler.py", line 20, in run_schedule
schedule.every(CRAWLER_RUN_CYCLE).minutes.do(crawler.run).run()
File "/home/venyo/.local/lib/python3.5/site-packages/schedule/init.py", line 411, in run
ret = self.job_func()
File "/mnt/d/repos/async-proxy-pool/async_proxy_pool/crawler.py", line 35, in run
redis_conn.add_proxy(proxy)
File "/mnt/d/repos/async-proxy-pool/async_proxy_pool/database.py", line 47, in add_proxy
self.redis.zadd(REDIS_KEY, proxy, score)
File "/home/venyo/.local/lib/python3.5/site-packages/redis/client.py", line 2263, in zadd
for pair in iteritems(mapping):
File "/home/venyo/.local/lib/python3.5/site-packages/redis/_compat.py", line 123, in iteritems
return iter(x.items())
AttributeError: 'str' object has no attribute 'items'

没有明白最后readme最后是什么意思?不支持抓取https的ip?

换句话说,只能校验http的ip?

项目clone后运行client.py报错

Traceback (most recent call last):
File "D:/Program Files/PycharmProjects/untitled/async-proxy-pool/client.py", line 8, in
run_schedule()
File "D:\Program Files\PycharmProjects\untitled\async-proxy-pool\async_proxy_pool\scheduler.py", line 20, in run_schedule
schedule.every(CRAWLER_RUN_CYCLE).minutes.do(crawler.run).run()
File "D:\Program Files\PycharmProjects\untitled\venv\lib\site-packages\schedule_init_.py", line 466, in run
ret = self.job_func()
File "D:\Program Files\PycharmProjects\untitled\async-proxy-pool\async_proxy_pool\crawler.py", line 34, in run
redis_conn.add_proxy(proxy)
File "D:\Program Files\PycharmProjects\untitled\async-proxy-pool\async_proxy_pool\database.py", line 45, in add_proxy
self.redis.zadd(REDIS_KEY, proxy, score)
File "D:\Program Files\PycharmProjects\untitled\venv\lib\site-packages\redis\client.py", line 2320, in zadd
for pair in iteritems(mapping):
File "D:\Program Files\PycharmProjects\untitled\venv\lib\site-packages\redis_compat.py", line 109, in iteritems
return iter(x.items())
AttributeError: 'str' object has no attribute 'items'