Support proxy authentication with dictionary

[seldcat]

Support proxy authentication with dictionary

[jxlil]

Sorry, I've been a little busy, I'm going to get back to this today.

[jxlil]

Now, RequestParser does not handle proxy authentication, so you can use custom authentication instead:

yield Request(
    url=f"http://example.com",
    headers={
        "Proxy-Authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ1c2VyX2lkIiwiZXhwIjowMDAwMDAwMDAwfQ.8ZLkrPvu9-hKLnq8fghbZYdv-a-KMJfHtSndfFmT3PQ",
    },
    meta={
        "impersonate": "chrome110",
        "proxy": "http://0.0.0.0:80000",
    },
)

[jxlil]

Hi @seldcat, thanks for your comment.

Could you provide an example of how you are making the request, so I can test?

[seldcat]

If we consider a typical proxy format:

<scheme>://<user>:<pass>@<host>:<port>

In my case, the pass field is a dictionary with keys like token, country, etc. When using such proxies, the current implementation fails because it doesn’t encode the dictionary properly.

To resolve this, you need to encode the dictionary as I described earlier.

[seldcat]

It's worth noting that Scrapy decodes the username and password when collecting the header (see this line in the Scrapy source code). However, scrapy-impersonate does not encode it back. Therefore, if the username or password contains special characters, it will cause a crash.

Also, everything works fine for me if I disable the scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware (the password and login are decoded here and then not properly encoded, sending the improperly encoded password to the next stage), so perhaps the problem is on my side.