http proxy support?

[code4craft]

http proxy support?

[yuany]

the core of crawling is httpClient, so I think it support proxy, but it need covert it in implement.

[code4craft]

You can override HttpClientDownloader.download() (copy most codes of the method) to implements HttpProxy in 0.3.0 and below.
example

HttpClientDownloader is too complex, I will refactor it in next version. HttpProxy will also be supported.

[l1z2g9]

Thanks.

In the begining, I consider webmagic is base on jsoup, but now I get that it is built on HttpClient.

I googled some code snippet about jsoup, it is supported as this style of http proxy.

System.setProperty("http.proxyHost", "proxyserver");
System.setProperty("http.proxyPort", "8080");

Document doc = Jsoup.connect("http://www.g.cn").get();
System.out.println("content " + doc.html());

Hope these convenient configuration would be supported in webmagic in future.

[code4craft]

will be supoorted in 0.4.0, usage:

site.setHttpProxy(new HttpHost("your-host",port)