Comments (17)
Hi @swb-ops could you indicate the version you are using?
Is there slowness on the Kafka side?
Does the memory leak happens when -kafka=false
?
from goflow.
Hi @lspgn, thanks for the answer.
I use GoFlow v3.4.2.
Yes, the memory leak happens when -kafka=false.
# systemctl status goflow
● goflow.service - GoFlow
Loaded: loaded (/lib/systemd/system/goflow.service; disabled; vendor preset: enabled)
Active: active (running) since Tue 2020-07-21 20:41:30 MSK; 14min ago
Main PID: 30059 (goflow)
Tasks: 14 (limit: 4915)
CGroup: /system.slice/goflow.service
└─30059 /usr/bin/goflow -sflow.addr 192.168.226.22 -sflow.port 9953 -kafka=false
When I started the service:
# free -m
total used free shared buff/cache available
Mem: 32167 1441 26275 32 4450 30136
After 15 minutes of work:
# free -m
total used free shared buff/cache available
Mem: 32167 4158 22761 32 5247 27409
from goflow.
How many flows per second are you processing?
from goflow.
We are currently processing about 3000 flows per second.
from goflow.
I tried using versions 3.4.2, 3.4.0 and 3.1.0, sending netflow and sflow to the collector, migrating goflow to another VM. Memory leaks occurred in all cases.
from goflow.
Hi @lspgn,
Maybe you have any other ideas what could be the reason for the memory leak ?
from goflow.
Are you only sending sFlow to this port? Could you make a packet capture of the UDP traffic?
from goflow.
dump_sflow.zip
Yes, we are sending only sFlow to this port. We are currently processing sflow from several devices, it is about 300 flows per second and 32 GB of RAM enough for about 12 hours.
Packet capture in an attachment.
from goflow.
I checked the file @swb-ops: I will try to replay it later this week.
Wireshark alert on Frame 7 (Gryphon protocol) but it is unlikely to be the issue. Could be the IPv4 data section along with the raw packet header.
Out of curiosity what is the sFlow agent?
from goflow.
Hello,
We are getting the sflow from Juniper and Huawei.
from goflow.
Sorry for the delay, following up again: what's the version of the Juniper you're running? Trying to correlate with another issue.
from goflow.
Also tried to reproduce by replaying your file in a loop, but I am not seeing an increase of RAM usage :( .
If the issue is still ongoing, would you be able to use custom version of GoFlow with pprof and collect performance data from there?
from goflow.
Hi, thanks for the answer.
We use 18.1R3-S6.1 and 18.4R2-S3 versions of the Juniper.
Yes, we are ready to try custom version of GoFlow.
from goflow.
Hello @lspgn,,
We can hope that you will provide us custom version of GoFlow with pprof ?
from goflow.
Hello,
Yes, I will craft a custom version with pprof but I have not had time yet. My apologies.
Thank you
from goflow.
Ok, we will wait.
Thank you.
from goflow.
Hello @lspgn
We are using Nginx as a load balancer and it changed src ports. I configured Nginx transparently and now goflow works very well.
Thank you.
from goflow.
Related Issues (20)
- How to lookup the received netflow packets HOT 2
- Can this project be used as a library? HOT 2
- Renormalise the sampled data HOT 2
- gloflow2 doesn't multiply sampling rate with bytes/packets
- Nat Event On Output(Question) HOT 2
- Protobuf messages are corrupted or don't match the provided schema HOT 4
- Port Mirror Question HOT 2
- vendor-specific field parsing HOT 2
- Support for IPFIX dataLinkFrameSection HOT 1
- SIGSEGV when port is already in use
- GoFlow segfaults with bad input HOT 3
- How can we configure goflow to enable collector for Netflow V9 format?
- Same Time* for sflow even if they are different
- [Kafka SSL] Custom truststore and keystore.
- IPFIX Template decoding does not handle Enterprise-Specific Information Elements
- Compiling FlowMessagePb proto
- Cardinality explosion in flow_traffic metrics HOT 1
- Is this project alive ? HOT 3
- Sampler Address HOT 3
- Openvswitch ovs-vsctl connection HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from goflow.