js1019/normalmodes

Normal Modes at Planetary Scales

normalmodes's Issues

assert output errors for demos

Bringing to github our email communication. I asked:

we are interested in adding a validation phase that checks the output data

@js1019's answer:

I am not sure how to perform a validation. Are you suggesting a benchmark of some standard results from other platforms? The current output shows the relative errors. If the errors were small, ~10^-11, the users could be confident of their results

---

based on the above, I was wondering if we could add a script that automates these checks on the output log. We can work on a PR for adding this script, if you think this would be useful.

Fatal error when running NormalModes under Open MPI

In attempting to run NormalModes under Open MPI, I notice that there is a Fortran --> C translation of an MPI_Comm that is not happening correctly between NormalModes and ParMetis (http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview).

I filed a bug with ParMetis (see http://glaros.dtc.umn.edu/flyspray/task/174), but ParMetis appears to be abondonware (** see below), so I am filing the bug here for you, NormalModes maintainers. 😄

I represent a vendor partner mentoring one of the student teams for SC'19; they are attempting to run NormalModes with Open MPI. I also happen to be one of the core developers of Open MPI.

The problem is here:

1. geometry_mod::pnm_apply_parmetis() is a Fortran subroutine that invokes ParMETIS_V3_PartKway().
2. The last parameter to ParMETIS_V3_PartKway() is an MPI communicator handle.
3. geometry_mod::pnm_apply_parmetis() passes in 0, which is actually the correct Fortran integer value for MPI_COMM_WORLD.
4. ParMETIS_V3_PartKway() is a C function. The last parameter is of type (MPI_Comm*). THIS IS WRONG

MPI defines MPI handles as INTEGERs in the Fortran mpi module (which is what geometry_mod uses). The Fortran compiler/runtime will therefor pass that parameter as a pointer to an integer. MPI defines that this type must be received on the C side as (MPI_Fint*).

Specifically, ParMETIS does two things wrong:

1. It defines the MPI handle in the Fortran entry point function as type (MPI_Comm*). It needs to be (MPI_Fint*).
2. It does not convert the Fortran handle to a C handle before using it with C MPI API functions. It needs to use MPI_Comm_f2c() on the (MPI_Fint) to convert it to an (MPI_Comm).

Unfortunately, ParMETIS does not do this: it just dereferences the (incorrect) (MPI_Comm*) argument and tries to use it with C MPI API calls.

In MPICH-flavored MPI implementations, this just happens to work fine because MPICH MPI handles are integers in both C and Fortran.

Open MPI's C handles, however, are pointers (which has been valid since MPI-1.0 in 1994). As such, when NormalModes effectively passes MPI_COMM_WORLD (which is defined to be integer value 0) to the ParMETIS function, ParMETIS dereferences it and gets a 0 back. When it passes 0 to the Open MPI C MPI API functions, they (correctly) interpret that as NULL and throw an MPI exception.

---

NOTE: The last release from ParMetis was in 2013. With a quick look through their bug tracker, it looks like the last time a maintainer assigned a bug was also back in 2013. I don't know for sure, but it looks like ParMetis is abondonware. 😢

If ParMETIS is abondonware and won't be fixed, I think the only real option you have is to write C wrappers around the ParMETIS calls -- i.e., you invoke the MPI handle f2c functions and then invoke the real underlying ParMETIS C API calls.