Unusual high memory and cpu consumption after hours which causes system instablility: expected behaviour due to presence of dynamically allocated IPv6 counters

[DenisKlimek]

Unusual high memory and cpu consumption after hours which causes system instablility: expected behaviour due to presence of dynamically allocated IPv6 counters

[pavel-odintsov]

Hello! Thank you for report! How many unique hosts do you have in your network? FastNetMon prints it in log file. I can see that overall amount of memory is very small and I can recommend adding at least 8G of memory. If machine started using swap disk then it may be expected to see behaviour you have observed as increase of memory latency will lead to speed calculation delays. If you have swap on hypervisor then it may be another reason of such behaviour. Generally FastNetMon does not allocate large amount of memory during operations, it does allocate it when it starts and after that it may be only very small amount of memory allocations . If you operate large network then I can recommend looking on our commercial edition FastNetMon Advanced which features exceptional scalability capabilities https://fastnetmon.com/compare-community-and-advanced/ <https://fastnetmon.com/compare-community-and-advanced/?amp> Sincerely yours, Pavel Odintsov

…

On Mon, 5 Feb 2024 at 08:18, Denis Klimek ***@***.***> wrote: I'am evaluation the latest FastNetMon Version 1.2.6 internally and setup a fresh Ubuntu 24.04 (Daily Build). This VM is also observed by Observium and collecting CPU, Memory etc stats. After some hours I noticed that the CPU usage is increasingly, together with memory usage and finally the detection times / calculation times of packets are also increasing from 0.2s up to 3-4s after three days. The configuration of FastNetMon is not very special, we are sending NetFlow V9 packets from Cisco IOS-XR devices to it, only incoming traffic. No connection tracking enabled. 1:3000 Sampling. Flow Calcuation itself is disabled. No other flows or traffic captures (pcap/mirror) are configured. Attached I uploaded two screenshots of our monitoring suite that shows the increasements. The drops are caused by rebooting the whole server and/or restarting the fastnetmon daemon via systemctl :-) Firefox_Screenshot_2024-02-05T08-09-12.975Z.png (view on web) <https://github.com/pavel-odintsov/fastnetmon/assets/28755170/82c6bbb0-69b5-4fa4-8ee5-e6a7e0d7b495> Firefox_Screenshot_2024-02-05T08-08-43.952Z.png (view on web) <https://github.com/pavel-odintsov/fastnetmon/assets/28755170/f7f8bcab-519e-40d6-8cc6-a69f55a6c5c8> — Reply to this email directly, view it on GitHub <#1003>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU56ZXAXTTI2WAGX3FDWLTYSCIVRAVCNFSM6AAAAABCZV7CVSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYTOOJTGI2TIMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[DenisKlimek]

Good Morning Pavel,

thanks for your quick reply! :) We have about 250k unique hosts behind our network - I changed the amount of memory from 4GB to 8GB based on your advice and keep monitoring it.

Currently we are only in evaluation phase to look and compare against mutliple DDoS detection and mitigation methods. The advanced version might be also interesting, but currently we continue testing community version with basic functionality.

Thank you

[pavel-odintsov]

Hello! 250k may be on brink of limit for community edition capabilities. Advanced one uses different way more advanced algorithms and can scale easily to such number of hosts. Sincerely yours, Pavel Odintsov

…

On Mon, 5 Feb 2024 at 10:24, Denis Klimek ***@***.***> wrote: Good Morning Pavel, thanks for your quick reply! :) We have about 250k unique hosts behind our network - I changed the amount of memory from 4GB to 8GB based on your advice and keep monitoring it. Currently we are only in evaluation phase to look and compare against mutliple DDoS detection and mitigation methods. The advanced version might be also interesting, but currently we continue testing community version with basic functionality. Thank you — Reply to this email directly, view it on GitHub <#1003 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAU56ZX5BKHBSWYAUV4CAQ3YSCXONAVCNFSM6AAAAABCZV7CVSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRWGY2TONRXHA> . You are receiving this because you commented.Message ID: ***@***.***>

[DenisKlimek]

Hello Pavel,

we did some more testing and disabled the transfer of IPv6 flow data to FastNetMon. After receiving IPv4 flows only the cpu and memory consumption stays on a constant level for two days already. See attached screenshots. Looks like that problem is related to IPv6 only. See attached screenshots.

Regards,
Denis

[pavel-odintsov]

Hello!

That's absolutely expected behaviour. If you have many IPv6 hosts in your network then FastNetMon will need to allocate memory to track each IPv6 address in your network. So it will lead to gradual increase of memory usage: https://fastnetmon.com/docs/memory_consumption/ That's exactly what you observe.

IPv4 counters have different design and FastNetMon Community allocates them on load and then it cannot grow. This approach is impossible for IPv6 and that's why we allocate counters on demand.

If you do want to reduce memory usage then Advanced edition is here for your disposal: https://fastnetmon.com/docs-fnm-advanced/flexible-counters-for-traffic-calculation/