MPI Fortran library cannot be found when linking in downstream projects when STRUMPACK is enabled

[white238]

MPI Fortran library cannot be found when linking in downstream projects when STRUMPACK is enabled

[white238]

Narrowing this down and it is a problem/missing feature in CMake:

spack/spack#24685

We are attempting to allow Spack to build the MPI itself, hopefully this goes well.

Thanks for helping track this down and there is nothing MFEM could do to work around this.

[v-dobrev]

This is normal since MFEM's make config does not really try to automate the config process for dependencies. It provides sensible defaults for the variables <PACKAGE>_OPT (compile flags) and <PACKAGE>_LIB (link flags). If these defaults do not work, the user is expected to define them on the make config ... command line or in their config/user.mk file (which should be copied from config/defaults.mk and then modified where needed).

In the case of STRUMPACK, STRUMPACK_LIB uses another variable to configure the MPI+fortran dependence -- MPI_FORTRAN_LIB. In your case, it sounds like it will be sufficient to just modify accordingly this variable and leave STRUMPACK_{OPT,LIB} as they are.

For example, on my Mac, I modify MPI_FORTRAN_LIB by adding the following to my user.mk:

# Additional Fortran library:
MPI_FORTRAN_LIB += -L$(BREW_DIR)/Cellar/gcc/13.1.0/lib/gcc/13 -lgfortran

where BREW_DIR is the path to my Homebrew directory.

In your case, you probably just need to add the path to where libmpi_mpifh.* is, before -lmpi_mpifh. Normally, this path will already be specified on the linking command line, so it is not explicitly needed. However, in your case, the MPI link path probably appears after -lmpi_mpifh. So maybe a simpler solution is to just move the MPI link path before the MFEM flags.

[v-dobrev]

I just realized that you may be using config.mk file from the CMake build system where things should be a bit more automated. Are you using CMake to build MFEM?

[white238]

This is building MFEM through Spack on OSX, so it would be your Makefile build system. Should a fix go into your Spack package?

Maybe something like:

// Get MPI_DIR
// Get MPI_LIB
MPI_FORTRAN_LIB += -L$(MPI_DIR) -l$(MPI_LIB)

I'll figure out the specifics later when caffiene kicks in.

[v-dobrev]

In Spack this should be handled -- in the contents of MFEM_EXT_LIBS, I see the flags: -Wl,-rpath,/opt/local/lib -L/opt/local/lib -lmpi_mpifh. The path /opt/local/lib is obtained from mpi.prefix.lib, see here: https://github.com/spack/spack/blob/c38ef72b0668b02a566ce7f5cee114f776ccc9b6/var/spack/repos/builtin/packages/mfem/package.py#L751-L752. I'm not sure what's wrong -- isn't libmpi_mpifh in that directory? Or is this an MPI implementation that is not handled? We have logic for the following MPI implementations: https://github.com/spack/spack/blob/c38ef72b0668b02a566ce7f5cee114f776ccc9b6/var/spack/repos/builtin/packages/mfem/package.py#L749-L754.

[white238]

mpi_mpifh was in the correct directory but no -L<mpi directory> was being added to the link line.

[v-dobrev]

Shouldn't -L<mpi directory> -lmpi (or something like that) be added to the link line where you're linking an MPI executable without the MPI compiler wrapper?

If we want to add these flags to MFEM_EXT_LIBS, (by setting MPI_{OPT,LIB}) we may need a lot of extra logic to figure out what flags are actually added by the MPI wrapper -- it is not always just -I<mpi dir>/include and -L<mpi dir>/lib -lmpi. It is much more portable to just rely on the MPI wrapper, I think.

Maybe specifically in Spack this is easier to do? Do you know if we can get the needed MPI flags from Spack or the MPI-provider packages in Spack?

[white238]

Shouldn't -L<mpi directory> -lmpi (or something like that) be added to the link line where you're linking an MPI executable without the MPI compiler wrapper?

Kind of, it generally adds an absolute path to the MPI libraries (we don't add the Fortran MPI library because we don't explicitly require it) but that isn't the problem. MPI_EXT_LIBS adds the Fortran MPI library to the link line w/o the directory to find it.

If we want to add these flags to MFEM_EXT_LIBS, (by setting MPI_{OPT,LIB}) we may need a lot of extra logic to figure out what flags are actually added by the MPI wrapper -- it is not always just -I<mpi dir>/include and -L<mpi dir>/lib -lmpi. It is much more portable to just rely on the MPI wrapper, I think.

The includes are and the absolute C++ libraries are added by our project but no -L's.

Maybe specifically in Spack this is easier to do? Do you know if we can get the needed MPI flags from Spack or the MPI-provider packages in Spack?

Staring at the multiple levels, I think what is happening is the Spack package adds the proper information to STRUMPACK_LIB so that MFEM itself builds, but that information is not getting passed to the config/defaults.mk|whatever generates the config.mk. I could be wrong but I think this line is injecting the $(MPI_FORTRAN_LIB) but not the -L:

mfem/config/defaults.mk

Line 331 in ac88580

STRUMPACK_LIB = -L$(STRUMPACK_DIR)/lib -lstrumpack $(MPI_FORTRAN_LIB)\

Does the Makefile build system not override that command line STRUMPACK_LIB before it generates the outputted config.mk? I don't fully understand how your Makefile generates that file w/o more effort.

[white238]

When @btalamini gets back I can get the command line from Spack to see if it will show more information of what is going on. I can't reproduce this myself.

[v-dobrev]

Just to clarify: the MFEM's spack packages does add the correct path for -lmpi_mpifh or, more precisely, what I think is the correct path derived from the MPI package: the MPI prefix + /lib. This is done on this line:

                        sp_lib += [ld_flags_from_dirs([mpi.prefix.lib], ["mpi_mpifh"])]

(direct link to the above code)

In this context, the variable mpi is defined as:

                    mpi = strumpack["mpi"]

(direct link to the above code)

Judging by the value of MFEM_EXT_LIBS posted above, the path mpi.prefix.lib resolved to /opt/local/lib. Is this not the correct path? Maybe the libraries are in lib64 sub-directory and not in lib? If this is the case, I can fix the code to actually look at the file system to determine the right path.