jxlil/scrapy-impersonate

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.

scrapy-impersonate's Issues

Operation timed out after 30002 milliseconds

In some cases it's possible to get a timeout error:

  File "/usr/local/lib/python3.9/site-packages/scrapy_impersonate/handler.py", line 45, in _download_request
    response = await self.client.request(**RequestParser(request).as_dict())  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/curl_cffi/requests/session.py", line 554, in request
    raise RequestsError(e)
curl_cffi.requests.errors.RequestsError: Failed to perform, ErrCode: 28, Reason:  \
'Operation timed out after 30002 milliseconds with 1096454 bytes received'

It should be possible to define the timeout for curl_cffi from scrapy.Request

TypeError: AsyncSession.request() got an unexpected keyword argument 'thread'

File "/home/neiellcare/.cache/pypoetry/virtualenvs/rgdl-spiders--SESLHN9-py3.11/lib/python3.11/site-packages/scrapy_impersonate/handler.py", line 44, in _download_request
    response = await self.client.request(**RequestParser(request).as_dict())  # type: ignore
TypeError: AsyncSession.request() got an unexpected keyword argument 'thread'

Support of fresh browser versions

Hello, please let us know if you have plans to release support new brosers versions chrome 120+ in you future updates?