scrapy-nimble
is a Scrapy Downloader Middleware that helps to integrate
Scrapy with Nimble Web API.
You can install scrapy-nimble
as a regular Python package from
PyPI using:
pip install scrapy-nimble
-
If you don't have it yet, open an account with Nimble.
-
Provide your credentials and enable the middleware through Scrapy settings.
# settings.py NIMBLE_ENABLED = True NIMBLE_USERNAME = "username" NIMBLE_PASSWORD = "password"
-
Add the downloader middleware to your
DOWNLOADER_MIDDLEWARES
Scrapy setting.# settings.py DOWNLOADER_MIDDLEWARES = { "scrapy_nimble.middlewares.NimbleWebApiMiddleware": 570, }
If you have
scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware
enabled (it is enabled by default in DOWNLOADER_MIDDLEWARES_BASE setting with default order equal to 590), configurescrapy-nimble
middleware before it.
Once the downloader middleware is properly configured, every request goes through the Nimble's Web API. There is no need to change anything in your spider's code.
scrapy-nimble
uses Nimble Web API with Real-time URL
requests. In addition to the default GET request for a specific URL, this API provides some extra options
that allow you to execute geolocated requests, render dynamic content, among others.
Right now the following request options can be used. Check the documentation for usage and the valid values that can be provided. If the option is not given, the default value from Web API will be used.
method
country
locale
headers
cookies
render
render_options
Add the options you want to be used inside the meta
key of your request, appending nimble_
to the
option name such as:
# Inside your spider
yield scrapy.Request(
"https://nimbleway.com",
meta={
"nimble_country": "DE",
"nimble_locale": "uk",
"nimble_render": True,
}
)
We suggest the use of pyenv to manage your Python version and create an isolated environment where you can safely develop. After installing it, you can prepare the environment using the following commands:
$ pyenv virtualenv 3.11.6 myvenv
$ pyenv activate myvenv
$ python -m pip install -e .
To keep a standard in code formatting and do some linter checks, we use pre-commit hooks.
Install pre-commit
package and install the project hooks using:
$ pre-commit install
Now you are ready to start development.