Comments (5)
Sorry, I've been a little busy, I'm going to get back to this today.
from scrapy-impersonate.
Now, RequestParser
does not handle proxy authentication, so you can use custom authentication instead:
yield Request(
url=f"http://example.com",
headers={
"Proxy-Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyX2lkIiwiZXhwIjowMDAwMDAwMDAwfQ.8ZLkrPvu9-hKLnq8fghbZYdv-a-KMJfHtSndfFmT3PQ",
},
meta={
"impersonate": "chrome110",
"proxy": "http://0.0.0.0:80000",
},
)
from scrapy-impersonate.
Hi @seldcat, thanks for your comment.
Could you provide an example of how you are making the request, so I can test?
from scrapy-impersonate.
If we consider a typical proxy format:
<scheme>://<user>:<pass>@<host>:<port>
In my case, the pass
field is a dictionary with keys like token
, country
, etc. When using such proxies, the current implementation fails because it doesnโt encode the dictionary properly.
To resolve this, you need to encode the dictionary as I described earlier.
from scrapy-impersonate.
It's worth noting that Scrapy decodes the username and password when collecting the header (see this line in the Scrapy source code). However, scrapy-impersonate
does not encode it back. Therefore, if the username or password contains special characters, it will cause a crash.
Also, everything works fine for me if I disable the scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware
(the password and login are decoded here and then not properly encoded, sending the improperly encoded password to the next stage), so perhaps the problem is on my side.
from scrapy-impersonate.
Related Issues (9)
- Operation timed out after 30002 milliseconds
- TypeError: AsyncSession.request() got an unexpected keyword argument 'thread' HOT 5
- Support of fresh browser versions HOT 6
- Requests generated through Media Pipeline is not using scrapy-impersonate download handler HOT 5
- Scrapy spider 3x slower after implementing this HOT 4
- User-agent not set HOT 1
- Proxy Authentication failure HOT 9
- ERROR: Loading "scrapy_impersonate.ImpersonateDownloadHandler" for scheme "http" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scrapy-impersonate.