Out of memory issue

[nmonvisualizer]

Out of memory issue

[hpresnall]

It may be that you just need more memory for a large number of files or files with a lot of data. But 10GB does seem like it should be enough unless you have more than 1GB of NMON files.

I have made one small change in the latest commit if you want to try that. It may only help when processing files with a lot of process / TOP data on IBM JVMs though. In my case, it saved about 200MB of memory when processing 3 files totaling 125MB. Heap used before the fix was about 800MB.

Past that, I will need to dig deeper which may take a while to get to with the holidays coming up.

[michaello1]

I can simulate high memory consumption (around 8,9 GB) with just 12 nmon files representing 6 days (3 MB per nmon file, 36 MB total). Those files are from AIX and I did complex analysis without problems in the past (I can process Q1/2017 and Q2/2017 without problems. Q3 and Q4 only when I find first bad day and divide interval to 2 parts). Probably some data causes this problem. Let me know if I can provide you more relevant information (like Eclipse MAT analysis or so).

I will try your latest patch a bit later.

[hpresnall]

I've made a few more changes but I don't think it's solving the main issue. Please try the latest commit though to see if anything is different.

Another thing you can try is to turn off logging. Go to Help -> View Log (CTRL-L). In the dialog that appears, change the log level drop down to WARNING or OFF.

I think it would help if I had your nmon files. Something is definitely wrong if 36MB of data is ballooning to 8.9 GB of heap.

Also, can you confirm that the Q1 and Q2 data are fine now, with your current JVM and NMONVisualizer version? I want to make sure it's the data and not something about your current setup.

[michaello1]

Your changes did not help. And it can not compile with Java 9 (I locally fixed it with adding final keyword to the scroller variable):
[javac] /Users/user/Data/tools/NMONVisualiser/build/nmonvisualizer/src/com/ibm/nmon/gui/util/LogViewerDialog.java:95: error: local variable scroller is accessed from within inner class; needs to be declared final [javac] scroller.setViewportView(log); [javac] ^ [javac] 1 error [javac] 4 warnings

Today I was performing analysis of nmon files from IHS servers. I loaded 529 nmon files (1.12 GB in total, average 2.1 MB per file) and with 9.15 GB consumed memory I was able to work with NMONAnalyzer without problem. Then I tried to load nmon files from second IHS server and it stopped after 450th file (OOM). When I try it with smaller interval I can simulate memory pressure (again just 16 files, 33.9 MB in total).

And yes Q1 and Q2 data are fine with Java 9 and compiled NMONVisualizer from source code.

Please write me an email (michael at upcmail dot sk) and I will send you a link to files.

[hpresnall]

Sent you an email last week requesting files.

[michaello1]

Thanks for the remainder. Check your email now.

[hpresnall]

I am pretty sure this due to logging and a possible issue with how JTextArea works. Can you turn off logging before you process any data and then rerun with all your files? Go to Help -> View Log (CTRL-L). In the dialog that appears, change the log level drop down to OFF.

If you confirm that works, I will also add a patch that removes the logging that's happening with your files. Looking at that code again it's not needed anyway.

[michaello1]

It helped. I was able to load all 529 files from the second box. There is still some difference though. Parsing 529 files from the first box consumed around 9,3 GB memory. Parsing 529 files from the second box consumed around 10,8 GB memory. At the end it works so this is not a big issue.

Sorry that I missed your recommendation about logging before :/

[hpresnall]

Great!

I made another commit that should work without turning off logging. Can you confirm that has the same effect? If it works, I think we can close.

[michaello1]

Yes, it works perfectly. I have tested it with many inputs and always with success. Thank you!