[BUG/ISSUE] Early termination at different points depending on diagnostics configuration

[geoschem]

[BUG/ISSUE] Early termination at different points depending on diagnostics configuration

[lizziel]

To summarize current status, after switching from MPICH 3.3 to OpenMPI 2.1:

1. All diagnostic collections off: timing info incomplete due to early termination (see geoschem/GCHP#6)
2. Any diagnostic collections on: diagnostic write hang at end of run (this issue)

I suggest seeing if this issue goes away after upgrading to OpenMPI 3 since switching from OpenMPI 2 to OpenMPI 3 on the Odyssey cluster fixed this same issue after a switch to a new operating system (CentOS6 to CentOS7).

[JiaweiZhuang]

With OpenMPI 3.1.3:

• With no diagnostics, it does not crash at 00:10 (fixes no. 1 error).
• With any diagnostics, it is able to write output files (fixes no. 2 error):
• The timing info is still incomplete (no. 3 error still exits geoschem/GCHP#6 (comment) )

[JiaweiZhuang]

Because OpenMPI 3 seems the most robust configuration right now, I suggest focusing on fixing its timing issue, and put aside MPICH3 and OpenMPI2 for now.

I made another AMI ami-01074a30392daa0f9 with OpenMPI 3.1.3

cd ~/tutorial/gchp_standard
mpirun -np 6 -oversubscribe ./geos

-oversubscribe is needed for OpenMPI 3 runtime when the # of physical cores is less the # of MPI processes.

[lizziel]

Thanks Jiawei, I am closing this issue since switching to OpenMPI 3 fixed the file write hang issue. The issue for incomplete timing info at the end of the run is still open and will be tracked separately in geoschem/GCHP#6.

[JiaweiZhuang]

I think we should put a warning on wiki regarding issues with MPICH3 and OpenMPI2. On a lot of shared systems users cannot install whatever MPI they want, unlike the cloud.

[lizziel]

Yes, I completely agree. This will be done this week.

…

-- Lizzie Lundgren Scientific Programmer GEOS-Chem Support Team [email protected]<mailto:[email protected]> http://wiki.geos-chem.org/GEOS-Chem_Support_Team Please direct all GEOS-Chem support issues to the entire GEOS-Chem Support Team at [email protected]<mailto:[email protected]>. This will allow us to serve you better. From: Jiawei Zhuang <[email protected]> Reply-To: geoschem/gchp <[email protected]> Date: Thursday, December 13, 2018 at 12:07 PM To: geoschem/gchp <[email protected]> Cc: "Lundgren, Elizabeth W" <[email protected]>, State change <[email protected]> Subject: Re: [geoschem/gchp] Early termination at different points depending on diagnostics configuration (#10) I think we should put a warning on wiki regarding issues with MPICH3 and OpenMPI2. On a lot of shared systems users cannot install whatever MPI they want, unlike the cloud. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_geoschem_gchp_issues_10-23issuecomment-2D447045826&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=ufHnU7-e1-NW4fL96VAHJJr174h3vxgazhrtxKGJQgQ&s=6f6u1oFzPqSf5cHG9HpJ-G4vEvyyvSDIQMDrU2bMudA&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAnyq09cMu5zvuDdK38ZwWlzKIuWHNuAks5u4okugaJpZM4ZQdu1&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=xyVOGV-4mAPz62S8RZON4khwZesGKcGg2_BHL4y5NjQ&m=ufHnU7-e1-NW4fL96VAHJJr174h3vxgazhrtxKGJQgQ&s=iItnQopIpWq1nE8tf4Lgj-HQIBlxmQGvgfteKIe-xZw&e=>.