Comments (3)
你检查过你的I/O输出情况吗?
批量输出的大小、并发的多少都会影响内存的;
输出与抓取是异步的,假如输出始终慢于抓取速度,那么内存就会一直增加;
另外,你的规则逻辑、带宽也很重,问题是多方面的;
你从我提到这几方面,做一下分析。
from pholcus.
另外,我不知道你的使用的版本是不是含义gc回收监控的,这个一定程度也可以抑制内存增长。
因为go的对闲置内存是不会马上归还系统的,而Pholcus又是在高速运行,产生大量垃圾,因此gc的效果就不那么明显了,加了手动gc,可以一定程度弥补这一缺陷。
from pholcus.
非常感谢回答!你说的几点对我来说是非常重要的信息~
I/O目前用的SSD,走的mysql,10000条信息存一次,用的系统默认,代码方面。是按照官方的写法做的,字段其实很少。GC方面我要研究下,还没接触过,总之,感谢!
from pholcus.
Related Issues (20)
- config中的版本号未修改
- 使用代理无法成功
- 历史记录名称存在bug,windows下有可能无法写入历史记录文件
- scraping a site that require using javascript to scroll down
- 请问这个框架支持json请求吗 HOT 2
- go get github.com/henrylee2cn/pholcus 的时候报错了 HOT 5
- 安装问题 HOT 1
- 不知道什么原因,单机版没有成功抓取到任何信息
- runtime error: slice bounds out of range HOT 1
- fatal: repository 'https://github.com/henrylee2cn/pholcus_lib/' not found HOT 1
- 请问 /web/pholcus-web.go 中 appInit() 中app.SetLog(Lsc).SetAppConf() 中的Lsc 是什么呢? HOT 2
- 执行 go get -u -v github.com/henrylee2cn/pholcus 失败 HOT 7
- windwos 编译出错 HOT 5
- 使用Context的AddQueue方法并发添加元素时,直接挂了请问是为什么呢?
- 完善主从分布式爬虫开发时的业务编排示例文档
- Any roadmap for english docs?
- Kafka:kafka: invalid configuration
- 分布式中你是如何实现客户端调用Rule函数呢?
- 建个楼,能把下面Python的爬虫用golang pholcus重写
- 运行example的demo 出错, teleport 的包好像改名了, 以前的里面的方法也都没有了 HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pholcus.