apple/gcgc

GCGC : Garbage Collection Graph Collector

GCGC uses a Jupyter notebook interface to analyze GC log files.

There are 17 generated plots, which analyze latency, concurrent and STW events, heap information, allocation rates, frequencies of events, and event summaries, comparing any number of log files and external data sources. The tool uses Jupyter notebook data visualization allows for easy customization of provided plots.

The analysis is built into a provided notebook, and generates plots and tables from collected GC information. The collected data for each log is parsed into a python pandas 'event log'. Then, using the event logs as a persistent database, the event information can be sorted, filtered, and grouped in both pre-set and customizable ways to display relevant trends and outliers.

Currently supports collectors in JDK11 & JDK 16.

Requirements

• Python3
• The following Python3 packages
  • numpy
  • pandas
  • matplotlib
  • Jupyter notebook

Installation explained here: docs/setup.md

How to run analysis

Follow the instructions in docs/how-to-run.md

---

Known edge cases:

Note: The following edge cases are known and not handled automatically:

1. Shenandoah has two phases per garbage collection cycle reporting Heap allocation, will lead to two plotted heap occupancy metrics for each GC phase.
2. ZGC in JDK16 Puts information in safepoints, does not automatically print these in log analysis as it currently stands. These safepoints have comparable metrics to pause times, but ZGC does not report them in the same fashion, so these must be manually enabled on plots.
3. ZGC bytes reclaimed calculation (This may extend to Shenandoah) may be negative, if the rate of allocation exceeds the rate of gc collection. Information is correctly provided in logs, not properly analyzed here. Feature is being fixed in a later version, see issue #61
4. Trying to plot a graph or plot with a returned matlpotlib.axes variable declared in another cell does not show up inline in Jupyter notebooks.

---

Generating a log file

Creating a log file is quite easy, just add these logging flags to your appllication.

-Xlog:gc*:./filename.log

More detailed logging can be added, but the tool should be able to handle the gc log reported using the above Java runtime flags.

gcgc's Issues

RFE: OpenJ9 GC log analysis

Hi,

OpenJ9 has a different gc log format and a different memory layout.
https://www.eclipse.org/openj9/docs/vgclog/

It would be really helpful if this format could be added to this tool.

Thanks!

Incorrect number of seconds in a day

Hi there, I used a code search and found an incorrect number of seconds in a day (84600 instead of 86400) listed in one of your files. I recommend fixing it to avoid incorrect timings that could potentially lead to unexpected behavior.

GCGC/src/filter_and_group.py

Line 192 in fa043ad

# Then, scale the times into seconds (x84600). Scaling explained here: https://matplotlib.org/stable/api/dates_api.html

Generational ZGC Support

Generational ZGC adds 'Minor' and 'Major' to the label causing the (?:Garbage Collection) group not to match:

[2022-11-22T04:18:09.618+0000][1583.644s][5085][info ][gc               ] GC(163) Minor Garbage Collection (High Usage) 42238M(95%)->16152M(36%) 0.641s

It's also not aware of generations, so the heap filled metrics get confused when a major collection happens. Log attached for convenience:

GenZGC.log.zip

Notebook README links don't work

Currently the links to the notebook README.md file point to what I assume is an internal Apple Github instance so these aren't resolvable on the public internet.

STW phases anaysis

Hi, first of all, good work. Unfortunately your tool analyze the stop-the-world phases generated by GC. It's ok if it is your goal, but you have to remember that there are other JVM internals that needs STW. JIT for example. If you want to provide complex information you need to analyze Xlog:safepoint log. If you don't want to analyze it then it would be nice to give a clarification at the begging of the Section A.

Any planned issues?

Are there any planned issues that will be released? Or am I welcome to contribute new issues now? Thanks!
I remember there being a stylistic logarithmic heat map issue that needed resolution.