fortran .mod files installed in libdir instead of includedir

[minrk]

fortran .mod files installed in libdir instead of includedir

[ggouaillardet]

I do not necessarily agree with your interpretation.
If I understand correctly, you expect $BUILD_PREFIX/bin/mpifort to be a cross compiler out of the box and that is simply not the case.
You can double check that with $BUILD_PREFIX/bin/mpifort --showme ..., you will likely see -L$BUILD_PREFIX/lib which is correct but not what you expect (you expect -L$PREFIX/lib)
Bottom line, when cross compiling, use gfortran or your Fortran compiler directly and manually set include and lib paths.

[minrk]

Sorry, I should have clarified that LDFLAGS, FCFLAGS, etc. are all set to find the directories in the proper prefix, so -I$PREFIX/include and -L$PREFIX/lib are specified, not $BUILD_PREFIX. I'm not expecting a cross compiler out of the box, but I would expect include files to be in the include directory. -L$PREFIX/lib does not find .mod files, -I$PREFIX/lib does, which is the unusual bit and requires special handling that other mpi implementations and fortran modules don't. All the other fortran packages I can find install their .mod files in include, not lib, plus fortran language docs suggest this location as well.

[minrk]

A more succinct illustration, excluding the compiler wrappers and cross-compilation, just gfortran:

program test_use_mpi

use mpi
implicit none

integer :: ierr, numprocs, proc_num

call mpi_init(ierr)
call mpi_comm_size(MPI_COMM_WORLD, numprocs, ierr)
call mpi_comm_rank(MPI_COMM_WORLD, proc_num, ierr)

print *, 'Hello from Process number', proc_num, &
         ' of ', numprocs, ' processes'

call mpi_finalize(ierr)

end program test_use_mpi

compilations fails with standard args, which I think is reasonably expected to work:

> gfortran -I$PREFIX/include -L$PREFIX/lib -lmpi_mpifh test.f90
    3 | use mpi
      |     1
Fatal Error: Cannot open module file ‘mpi.mod’ for reading at (1): No such file or directory

but succeeds when mixing include arguments and lib directories:

gfortran -I$PREFIX/include -I$PREFIX/lib -L$PREFIX/lib -lmpi_mpifh test.f90

[ggouaillardet]

There are two main MPI implementations: MPICH and Open MPI, the other libraries are generally derivatives.

If you want to cross-compile Scalapack with the MPI wrappers, make sure they are designed for cross-compilation.
Otherwise, fix the detection to correctly handle both MPICH and Open MPI.

Even if the .mod files are moved into $PREFIX/include, that won't happen before Open MPI 6, so better fix/enhance Scalapack or use the right wrappers.

[minrk]

Yes, it is easy enough to work around by adding -I$PREFIX/lib to FCFLAGS, which we've done. Cross compilation isn't really relevant to the issue as I illustrated above, though cross compilation is working fine, and the issue isn't related to scalapack, that's just where it was encountered recently (CMake's internal FindMPI is what fails). I just wanted to report the issue of surprising installation paths and compiler arguments, which appear to be unique to openmpi among fortran libraries. Certainly feel free to stick with it and close if the deviation from standards is deliberate or the cost of change is too high. Since the workaround is simple, there is little urgency.

In conda-forge, I think we'll fix installation of the modules to the include directory, as is done in debian's libopenmpi-dev, for example.

[ggouaillardet]

I am a bit surprised cmake failed here and it could be worth investigating.
Is it because it is asked to use a busted cross compiler and it assumes it works without testing it thoroughly?

If you can point me to some logs, I will have a look at it.

[minrk]

I shared the output from CMakeConfigureOutput.yaml in the initial report, but here is the full output.

Is it because it is asked to use a busted cross compiler and it assumes it works without testing it thoroughly?

Not at all. It's detecting that the compiler cannot find mpi.mod, doesn't figure out where mpi.mod is because it's not in a standard include location, and stops.

FindMPI searches include paths for mpi.mod (like gfortran itself), so setting -DMPI_FORTRAN_ADDITIONAL_INCLUDE_DIRS=$PREFIX/lib would likely also work.

As you've mentioned, we don't expect cross-compilation to work without help, so we fully expect to have to specify include and lib directories, which we do. FCFLAGS="-I$PREFIX/lib ${FCFLAGS}" is what we need, where $FCFLAGS is what works for ~every other package. I only wanted to raise that openmpi installs include files to the library directory, unlike other fortran packages, and unlike the expectations of CMake, gfortran, etc.

[dalcinl]

@ggouaillardet Our root complaint is the following: Open MPI is installing Fortran .mod files in the libdir location rather than the recommended and almost universally agreed includedir.

Perhaps Open MPI could learn a new configure option to install Fortran modules in includedir intead?

[ggouaillardet]

Thanks for the pointer.
I do not fully understand everything (especially how cmake is instructed to use which compiler) but from what I can see, it uses the MPI wrappers $BUILD_PREFIX/bin/mpicc and $BUILD_PREFIX/bin/mpifort. If I understand correctly, cmake will then mpifort --showme:compile to get the required flags, and I suspect your wrapper returns -I$BUILD_PREFIX/lib instead of -I$PREFIX/lib (since it is not supposed to be used out of the box). My interpretation is that cmake does not know how to handle busted Open MPI wrappers, and then fails.

I do appreciate having the Open MPI Fortran modules in the include directory would kind of fix this issue (if mpifort --showme:compile is indeed executed, some directories from $BUILD_PREFIX will get pulled and as long as $PREFIX takes precedence, or the files in $BUILD_PREFIX are "enough compatible", it could be fine), but at this stage, I do believe the right fix is to fix the MPI wrappers when cross-compiling.

[minrk]

My interpretation is that cmake does not know how to handle busted Open MPI wrappers, and then fails.

I'd say that cmake correctly identifies the compiler as not working and halts without bending over backwards to fix it, but sure. This seems like entirely sensible behavior and I have no complaints.

at this stage, I do believe the right fix is to fix the MPI wrappers when cross-compiling.

it's unclear if you mean for you or for us, but if you mean for us, I agree and this was resolved some time ago (adding -I$PREFIX/lib). I don't think there's any action to take related to cross-compiling, though we have also fixed openmpi's module install path in our packaging.

When cross compiling, I fully expect to need to find compiler and link arguments myself, and am fully okay with that requirement as a packager. It was in this process of finding what I needed -I and -L args to be that I encountered the surprising -I$PREFIX/lib. I really am satisfied with the current cross-compilation requirements and regret bringing it up, since it's not actually related to the issue of putting standard files in a surprising location, it just happens to be how I found it, presumably a small mistake that's bothered nobody for years because the compiler wrappers embed -I$PREFIX/lib.

[ggouaillardet]

Since you are switching wrappers from compiler to cross-compiler, it is up to you to fix the wrappers.

You can do that by setting the right paths in $BUILD_PREFIX/share/openmpi/*-wrapper-data.txt, in the case of Fortran, that would be $BUILD_PREFIX/share/openmpi/mpifort-wrapper-data.txt

[minrk]

Yup, and we've done that. All cross compilation's working fine. The only issue here is the peculiar installation path of include files in the lib directory.

[ggouaillardet]

I am not sure I understand correctly. Are you saying the build still fail (e.g. cannot find mpi.mod) after you updated mpifort-wrapper-data.txt?

[minrk]

No builds fail. As mentioned in the original report, adding -I$PREFIX/lib to OMPI_FCFLAGS (or FFLAGS if not using wrappers) is sufficient to get working cross-compilation. This issue is not about cross compilation or anything being broken.

[ggouaillardet]

Your initial report was

I don't understand why only the cross-compilation case is affected. adding -I$PREFIX/lib to OMPI_FCFLAGS results in successful compilation.

My point is -I$PREFIX/lib should be pulled by cmake from $BUILD_PREFIX/bin/mpif90 --showme:compile, so you should not have to set OMPI_FCFLAGS at all for it to work. I suspect -I$BUILD_PREFIX/lib gets pulled instead, and this is likely because $BUILD_PREFIX/share/openmpi/mpifort-wrapper-data.txt has not been updated for cross-compilation.

Anyway, it seems we cannot understand each other, so I will let other folks comment on this.
If you feel any need to tag me, I will be happy to give this an other try.

[minrk]

Sorry for the miscommunication. Yes, I should not have mentioned my lack of understanding of which cases were affected because it was not relevant to the issue at hand, and is why I updated the description to remove it since it could be read as a question, which it wasn't meant to be, and tried to clarify the relationship between the issue (paths) and how we happened to encounter it (cross compilation). I think I understand why now, but it remains not relevant to the actual issue, which affects all openmpi installs, not just cross compilation.

[ggouaillardet]

FWIW, here is a simple CMakeLists.txt that checks use mpi works

In my environment, gfortran is used as the Fortran compiler, Open MPI mpif90 is in the $PATH so -I/.../lib gets pulled and I do not need to set OMPI_FCFLAGS nor FFLAGS.

cmake_minimum_required(VERSION 3.0)
project(foo LANGUAGES Fortran)

find_package(MPI COMPONENTS Fortran REQUIRED)

if (NOT ${MPI_Fortran_HAVE_F90_MODULE})
	message(FATAL_ERROR "use mpi could not be found")
else()
	message(STATUS "use mpi was found")
endif()

[minrk]

Thanks. I'm trying really hard to be clear that I'm not asking for help building against mpi, nor am I trying to claim that anything doesn't work as intended. I really just wanted to report that the include files are in the wrong folder, not that mpifort can't find them. openmpi correctly tracks where it puts module files and records that in the compiler wrapper data. It just happens to be a weird folder, and it looks like a typo.

[dalcinl]

Well, I don't believe it is a typo, but a deliberate choice of Open MPI. While I agree libdir is not the usual place to store for .mod files, I also have to acknowledge that .mod files may not be truly platform-independent as C .h header files can be. Therefore, I can understand the argument about libdir being an appropriate location for module files.