kristobaljunta/scrapy-proxymesh-py3

Proxymesh downloader middleware for Scrapy

scrapy-proxymesh-py3

Proxymesh downloader middleware for Scrapy.

This is a fork of scrapy-proxymesh that supports Python 3.

This package replaces the original, as it ships under the same name (scproxymesh).

Installation

From PyPI

pip install scrapy-proxymesh-py3

From GitHub

pip install -e git+https://github.com/KristobalJunta/scrapy-proxymesh@master#egg=scproxymesh

Usage

• Register free acount from https://www.proxymesh.com/
• Add the following to settings.py:

DOWNLOADER_MIDDLEWARES = {
    'scproxymesh.SimpleProxymeshMiddleware': 100,
}

# if you use user & password auth
PROXYMESH_URL = 'http://username:[email protected]:31280'

# or, if you use ip-based auth:
PROXYMESH_URL = 'http://us-il.proxymesh.com:31280'

# Proxymesh request timeout
PROXYMESH_TIMEOUT = 60

scrapy-proxymesh-py3's Issues

Добавить оформление пакета на PyPi

• README
• ссылка на репозиторий
• теги

Only takes in 1 proxymesh address

Is it possible to load multiple of the addresses? I.E - proxymesh us-wa AND world?

I frequently get 503 errors and want to add more than the 10 to the rotation.