[BUG/ISSUE] Slow memory leak in GCHP 12.5.0

[geoschem]

[BUG/ISSUE] Slow memory leak in GCHP 12.5.0

[lizziel]

No, a fix will not go into 12.5.0 which is currently being benchmarked. The fix will go in as soon as it is available, hopefully 12.5.1. GEOS users are in the same situation as us and GMAO is working on diagnosing the problem and developing at least a work-around.

Which version to use prior to 12.5.0 depends on what you will be doing with the model. Which bugs in 12.4.0 are you concerned about? 12.3.2 may be a better option depending on your concerns.

[msulprizio]

There are no critical bug fixes going into 12.5.0. The primary fixes since 12.4.0 are to run with updated offline biogenic VOC emissions and volcano emissions to address issues discovered in 12.4.0 benchmarking.

The GCSC decided for us to release 12.4.0 and 12.5.0 at the same time to avoid version fatigue since 12.5.0 was supposed to be a quick version to benchmark. Benchmarking is in progress now, and that version will hopefully be released next week.

[JiaweiZhuang]

The memory leak is more pronounced for long simulations and can result in a simulation crashing

Typically how long can the model run before crashing? Higher-resolution grid probably has more memory leak?

[lizziel]

I have seen inconsistencies in the memory leak rate, which leads to inconsistencies in how long the model can run before crashing given say, one node to run on. Typically, however, a full chemistry run will not complete a 1-month run given a single node with 128G. See the GCHP performance spreadsheet I previously shared with you for start and end memory for all 7-day timing tests for more information.

[JiaweiZhuang]

See the GCHP performance spreadsheet

OK I saw that memory usage increases by ~30% after 7 days of standard run. That's fine for timing & benchmark purpose (for my project), but probably not good for actual research use... Doing a bunch of small 7-day runs seems quite inconvenient for most users

Here's my question: Do you expect the fix to go into 12.5.0? If not, which version of GCHP would you recommend before this problem is fixed? 12.3.2? (I saw in the GC release page that 12.4.0 has some bugs to fix).

[JiaweiZhuang]

Which bugs in 12.4.0 are you concerned about?

I just notice in http://wiki.seas.harvard.edu/geos-chem/index.php/GEOS-Chem_12#12.4.0 that:

We ask GEOS-Chem users to wait until GEOS-Chem 12.5.0 is released before updating. GEOS-Chem 12.5.0 will contain some more bug fixes and updates.`

Any critical bugs?

[lizziel]

The full list of 12.5.0 updates are here. These are GEOS-Chem core updates, not specific to GCHP.

[lizziel]

The memory leak is fixed in MAPL 2.0 which will be included in the upcoming 13.0.0 release. The memory leak will persist for remaining 12 series release.

[lizziel]

I am closing this issue since it is fixed in dev versions of GCHPctm 13.0.0.