Comments (4)
the core of crawling is httpClient
, so I think it support proxy, but it need covert it in implement.
from webmagic.
You can override HttpClientDownloader.download()
(copy most codes of the method) to implements HttpProxy in 0.3.0 and below.
example
HttpClientDownloader is too complex, I will refactor it in next version. HttpProxy will also be supported.
from webmagic.
Thanks.
In the begining, I consider webmagic is base on jsoup, but now I get that it is built on HttpClient.
I googled some code snippet about jsoup, it is supported as this style of http proxy.
System.setProperty("http.proxyHost", "proxyserver");
System.setProperty("http.proxyPort", "8080");
Document doc = Jsoup.connect("http://www.g.cn").get();
System.out.println("content " + doc.html());
Hope these convenient configuration would be supported in webmagic in future.
from webmagic.
will be supoorted in 0.4.0, usage:
site.setHttpProxy(new HttpHost("your-host",port)
from webmagic.
Related Issues (20)
- There's a code injection vulnerability of `us.codecraft.webmagic.downloader.PhantomJSDownloader`
- java.lang.OutOfMemoryError: Java heap space HOT 1
- HttpClientDownloader download error HOT 1
- WebMagic Extension version 0.9.0 has CVE-2023-2976 vulnerability
- 请教怎样实现一个基于数据库的Scheduler? HOT 3
- 请教怎样控制爬虫延时或者暂停? HOT 3
- 如何保持cookie不过期?
- 文档中的图片都变破图了,请修复,谢谢! HOT 1
- 启动时,自定义参数放在Request后面接收不到 HOT 2
- 如何在PageProcessor.process中,将page.getHtml()的内容传入至Xpath2Selector HOT 1
- 多久支持playwright
- 希望作者支持一下动态重试? HOT 3
- 使用RedisScheduler会报找不到rpush方法,使用默认的Schduler正常。 HOT 1
- 循环点击下一页,并设置循环结束条件
- 希望作者可以将doCycleRetry改成protect访问级别 HOT 2
- doCycleRetry not working when using immutable maps in extras
- 最新版本中测试用例无法运行:us.codecraft.webmagic.samples.HuabanProcessor HOT 1
- 爬取百度百科需要耗时100多秒,请问怎么解决
- 访问https form方式传参的post接口报错javax.net.ssl.SSLException: Connection reset
- 请求本地服务 响应报:400 Bad Request
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webmagic.