Performance impact question

[cloudflare]

Performance impact question

[lspgn]

The first thing I can think of affecting NetFlow decoding performance would be the shared template cache.
Protobuf encoding (+memory allocations) may also be the reason. Would need to compile a specific version which bypasses this. Also would need to pprof.

[mirsblog]

Ok. I will consider that and start another thread when I have more to share.

[lspgn]

Do you have per-process monitoring? Something like prometheus-node-exporter and process-node-exporter/cadvisor?
For just goflow, this should be able to handle thousands of flows per second.

If you want to dive on the effect of adding this function, I would suggest using pprof.

[mirsblog]

@lspgn FWIW, I ran some tests on a VM with 4vCPU 2.3GHz and 32GB RAM and compared it with nfacct. The maximum rate at which I could decode IPFIX and publish to kafka, without dropped packets, was at 15000 packets/second. Anything more than that caused a significant packet loss. nfacct in comparison could easily scale up to 60000 packets/second.

I ran the tests with -kafka=false and it seemed to not have any effect. I also increased the number of workers but I did not find any marked difference in packet drops with 1 or 100 workers.

Test setup:
Host type: VM
CPU: 4 vCPU Intel Xeon E312xx
Memory: 32GB

I used IPFIX PCAP with tcpreplay to send packets from one host to another.
$ sudo tcpreplay -i ens3 -K --loop=50000 -p 15000 ipfix.pcap

I monitored packet drops @ /proc/net/udp6.

[lspgn]

Hi @jcdaniel14
What metrics are you looking at that make you say your server is overloaded?
Most of the processing of GoFlow is done when decoding Filtering on the interface should be negligible.

[jcdaniel14]

Hi, I have a bare metal server, 128GB RAM 24 cores AMD Opteron 6174, CPU processing seems to be around 50% utilization with spikes up to 70%.
I also have in this server kafka and ELK Stack so it is difficult to tell if I'm putting pressure by adding these lines of code inside goflow methods.

[lspgn]

@mirsblog interesting, thanks for the insights.
Does it use all the processors?

[mirsblog]

@mirsblog interesting, thanks for the insights.
Does it use all the processors?

I assume so given runtime.GOMAXPROCS(runtime.NumCPU()) is set in goflow.go. Would that be a correct assumption?

[lspgn]

Yes it should use all processors, was just curious if the load distribution would be the same when looking at htop. Did you compile GoFlow or did you get a specific binary?

[mirsblog]

Yes it should use all processors, was just curious if the load distribution would be the same when looking at htop. Did you compile GoFlow or did you get a specific binary?

GoFlow: v3.4.2
GoLang: 1.14
Built Alpine image using Dockerfile found in the v3.4.2. Ran using the instructions from README.

Edit: Tested just now and checked in htop to confirm CPU load distribution is even with workers=4

[jcdaniel14]

Do you have per-process monitoring? Something like prometheus-node-exporter and process-node-exporter/cadvisor?
For just goflow, this should be able to handle thousands of flows per second.

If you want to dive on the effect of adding this function, I would suggest using pprof.

Don't really have per-process monitoring but will dive into it, the server is processing 25k flows/sec atm according to logstash and hasn't been affected noticeable by the changes I made. I was just worried that it handles the processing of flows/kafka/elasticsearch/logstash at the same time and I could put some stress by adapting the code the way I did, thanks for clarifying and giving some good practices advice.