Compilation with make and dependency files

[bsc-pm]

Compilation with make and dependency files

[rofirrim]

Hi @Z10Frank,

could it be the absence of the whitespace? Looks like GCC may accept this without the whitespace.

Perhaps you can modify your build system to include it?

Kind regards,

[Z10Frank]

Thank you!
That's strange, compiling with gcc without Mercurium did not need the whitespace between the flag and the filename.
Anyway, adding the whitespace in the makefile the error message disappears and the *.d files for the dependencies are generated.
That's great progress, but the *.o files are still not generated, so something is still missing.

Maybe the compilation log can help?
compile_smilei_Mercurium.txt

[rofirrim]

Hi @Z10Frank,

That's strange, compiling with gcc without Mercurium did not need the whitespace between the flag and the filename.

Yeah, probably GCC undertands this but Mercurium doesn't.

Maybe the compilation log can help? compile_smilei_Mercurium.txt

I don't see any obvious failure, do the commands printed in the log have actually executed?

Kind regards,

[Z10Frank]

Sorry, my bad, it was a dry print for make. Also, some environment variables were missing for the Python part, which is now causing some errors like:

Compiling src/Diagnostic/DiagnosticRadiationSpectrum.cpp
/gpfs/users/massimof/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:30: error: syntax error, unexpected __attribute__, expecting ',' or '}'
Compilation failed for file 'src/Collisions/CollisionalFusionDD.cpp'

Normally it does not do that with gcc, maybe Mercurium has an interaction with the Python part?

[rofirrim]

Compiling src/Diagnostic/DiagnosticRadiationSpectrum.cpp
/gpfs/users/massimof/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:30: error: syntax error, unexpected __attribute__, expecting ',' or '}'
Compilation failed for file 'src/Collisions/CollisionalFusionDD.cpp'

That is probably Mercurium choking with C++ headers. The support of C++ in Mercurium has always been a bit difficult.

Do you need to use Mercurium (you only need it if you use Fortran or some of the unsupported features described in [1]) or you can use llvm-ompss instead?

Find it in https://github.com/bsc-pm/llvm/releases
Documentation: https://pm.bsc.es/ftp/ompss-2/doc/user-guide/llvm/index.html

Sorry for the detour with this.

[1] https://pm.bsc.es/ftp/ompss-2/doc/user-guide/llvm/index.html#unsupported-features

[Z10Frank]

Hello, thank you, sorry for my late answer, I managed to compile llvm-ompss 2 from the release page:

tar -xvf llvm-github-release-2021.11.tar
cd llvm-github-release-2021.11
cd llvm
mkdir install
cd install

module load llvm/9.0.1/gcc-9.2.0 
module load anaconda2/2019.10/gcc-9.2.0 
module load openmpi/4.0.2/gcc-9.2.0
module load cmake/3.16.2/intel-19.0.3.199 

export INSTALLATION_PREFIX_NANOS6=/gpfs/users/massimof/nanos6/install
export INSTALLATION_PREFIX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install

cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS=clang -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6 

make 
make install

Now, compiling the code with llvm (adding the flag --ompss-2 in the makefile and using SMILEICXX ?= mpic++):

module load openmpi/4.0.2/gcc-9.2.0 
module load hdf5/1.10.7/gcc-9.2.0-openmpi

export HDF5_ROOT_DIR=/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/hdf5-1.10.7-j6enx3ruksp5mjo5ed6lhywrzozihamq

export I_MPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++ 
export OMPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++

export LD_LIBRARY_PATH=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib:$LD_LIBRARY_PATH

make -j 10

an error occurs:

/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

In general I can't find the library libc++.so.1 in the installation folder of llvm, so even adding the lib subfolder to the LD_LIBRARY_PATH does not find it. Was there an error in my build of llvm from the last release present in https://github.com/bsc-pm/llvm/releases?

[rofirrim]

Hi @Z10Frank

In general I can't find the library libc++.so.1 in the installation folder of llvm, so even adding the lib subfolder to the LD_LIBRARY_PATH does not find it. Was there an error in my build of llvm from the last release present in https://github.com/bsc-pm/llvm/releases?

This is strange. I am not sure why are you seeing this.

What is happening is that the clang++ binary that you have compiled is using the LLVM C++ library (libc++.so). When compiling with g++ the GNU C++ library is used (libstdcxx.so). So it looks like you have compiled LLVM with clang, which is OK (AFAICT g++ can only use libstdcxx.so).

Perhaps the clang you're using to compile llvm-ompss has been configured to use libc++ by default (I think this is a possibility) but somehow the library is not found in your system.

However, I would expect you to be able to find those libraries in general when running a program compiled with the clang you have used to compile llvm-ompss. Otherwise a very simple C++ "hello world" should fail for you as well.

So to be fair I don't have a good answer to your problem. Can you look into your environment and verify that the compiler you used to compile llvm-ompss can be used to compile a C++ "hello world" that you can actually run?

Kind regards,

[Z10Frank]

Thank you, indeed the Hello_World.cpp test with my environment creates the error:

/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++: error while loading shared libraries: libtinfow.so.6: cannot open shared object file: No such file or directory

Should I try to build llvm-ompss with gcc instead?

[rofirrim]

Hi @Z10Frank,

This is confusing. We're using one compiler to compile the llvm-ompss compiler. Forget about the latter (llvm-ompss compiler), you care about the first one.

You can check about the compiler used to compile llvm-ompss, opening the CMakeCache.txt that you will find in the directory where you ran cmake and there should be a CMAKE_CXX_COMPILER option. This is the compiler you used to compile.

Can you try to use that compiler to compile a simple hello world?

// hello.cc
#include <iostream>
int main() { std::cout << "Hello C++\n"; }

$ <the-compiler-you-found-in-cmakecache> -o hello hello.cc

Now try to run hello

$ ./hello

Does it work for you?

Kind regards,

[Z10Frank]

Thank you, in the cluster I am working on there is an installed version of LLVM (not llvm-ompss), which works perfectly when compiling Smilei, with this environment

module load anaconda2/2019.10/gcc-9.2.0
module load hdf5/1.10.7/gcc-9.2.0-openmpi
module load llvm/9.0.1/gcc-9.2.0
module load openmpi/4.0.2/gcc-9.2.0
export LANG=en_US.utf8
export LC_ALL=en_US.utf8

export I_MPI_CXX=clang++
export HDF5_ROOT_DIR=/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/hdf5-1.10.7-j6enx3ruksp5mjo5ed6lhywrzozihamq

In the associated directory, /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib (which is automatically appended to the LD_LIBRARY_PATH when loading the module) there is for example the file libc++abi.so.1 , which is absent from my build of llvm-ompss. A simple test for hello world in c++ and even a more complex one like compiling Smilei works perfectly with this installation present in the machine, so I suspect that something is misplaced in how I built llvm-ompss.

[rofirrim]

Thank you, in the cluster I am working on there is an installed version of LLVM (not llvm-ompss), which works perfectly when compiling Smilei, with this environment

module load anaconda2/2019.10/gcc-9.2.0
module load hdf5/1.10.7/gcc-9.2.0-openmpi
module load llvm/9.0.1/gcc-9.2.0
module load openmpi/4.0.2/gcc-9.2.0
export LANG=en_US.utf8
export LC_ALL=en_US.utf8

In the associated directory, /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib (which is automatically appended to the LD_LIBRARY_PATH when loading the module) there is for example the file libc++abi.so.1 , which is absent from my build of llvm-ompss.

Ok but this is not the problem you're having. The problem is that you cannot execute the compiler that you just have compiled. And you can't because the libc++.so library is not found. And you should observe a similar problem if you compile a hello world with the compiler of your environment (not the compiler that you are compiler).

Can you verify that your clang compiler is operating as expected for a hello world such as the one I showed you above?

If it does work, ldd hello on the binary will tell you where libc++.so is found. And then run ldd on the clang++ of llvm-ompss that is failing for you and let's try to investigate from here.

Kind regards,

[Z10Frank]

Ok, thank you, so with the clang++ compiler of the machine that works (ldd hello, where hello is compiled with clang++ -o hello Hello_world.cpp ):

linux-vdso.so.1 =>  (0x00007ffdd6d72000)
	libc++.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++.so.1 (0x00002b999b1bb000)
	libc++abi.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++abi.so.1 (0x00002b999b298000)
	libm.so.6 => /lib64/libm.so.6 (0x00002b999b3bc000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b999b6be000)
	libc.so.6 => /lib64/libc.so.6 (0x00002b999b8d4000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b999bca2000)
	librt.so.1 => /lib64/librt.so.1 (0x00002b999bebe000)
	/lib64/ld-linux-x86-64.so.2 (0x00002b999b198000)

I don't know if it is important, but the llvm on the machine was compiled with gcc-9.2.0.

Now, using ldd on the clang++ created from my build of llvm-ompss (ldd /gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++) I obtain:

linux-vdso.so.1 =>  (0x00007ffce5901000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002abd81306000)
	librt.so.1 => /lib64/librt.so.1 (0x00002abd81522000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002abd8172a000)
	libm.so.6 => /lib64/libm.so.6 (0x00002abd8192e000)
	libz.so.1 => /lib64/libz.so.1 (0x00002abd81c30000)
	libtinfow.so.6 => not found
	libc++.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++.so.1 (0x00002abd81116000)
	libc++abi.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++abi.so.1 (0x00002abd811f3000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002abd81e46000)
	libc.so.6 => /lib64/libc.so.6 (0x00002abd8205c000)
	/lib64/ld-linux-x86-64.so.2 (0x00002abd810e2000)

[rofirrim]

Ok thanks!

libtinfow.so.6 => not found

this seems the culprit. Looks like you will need ncurses? Perhaps you need to load a module in your environment.

Kind regards,

[Z10Frank]

Thank you, I have installed ncurses, now the command ldd /gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++ seems to show everything in place:

linux-vdso.so.1 =>  (0x00007ffd5212f000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b47fff95000)
	librt.so.1 => /lib64/librt.so.1 (0x00002b48001b1000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00002b48003b9000)
	libz.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/zlib-1.2.9-jthi6nubs2nrhi47wv7yexyau5v27cwu/lib/libz.so.1 (0x00002b48005bd000)
	libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00002b48007d4000)
	libstdc++.so.6 => /gpfs/softs/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-11.2.0-mpv3i3uebzvnvz7wxn6ywysd5hftycj3/lib64/libstdc++.so.6 (0x00002b48009fe000)
	libm.so.6 => /lib64/libm.so.6 (0x00002b4800e05000)
	libgcc_s.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-11.2.0-mpv3i3uebzvnvz7wxn6ywysd5hftycj3/lib64/libgcc_s.so.1 (0x00002b4801107000)
	libc.so.6 => /lib64/libc.so.6 (0x00002b4801320000)
	/lib64/ld-linux-x86-64.so.2 (0x00002b47ffd71000)

However when building Smilei this error occurs: fatal error: 'omp.h' file not found (I know that OpenMP will not be used when OmpSs-2 is activated, but the headers are still included). To avoid putting preprocessor directives before the include of 'omp.h', I wanted to build also OpenMP with the llvm-ompSs together with clang++, so restarting the process:

cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6

The make command afterwards goes well, but the make install command gives this error:

CMake Error at projects/openmp/runtime/src/cmake_install.cmake:47 (file):
  file INSTALL cannot find
  "/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib/libomp.so":
  No such file or directory.
Call Stack (most recent call first):
  projects/openmp/runtime/cmake_install.cmake:42 (include)
  projects/openmp/cmake_install.cmake:42 (include)
  projects/cmake_install.cmake:42 (include)
  cmake_install.cmake:57 (include)


make: *** [install] Error 1

[rofirrim]

Hi @Z10Frank

I'm at loss here. I've never seen this error before and I'd expect your cmake line to work fine when building openmp.

CMake Error at projects/openmp/runtime/src/cmake_install.cmake:47 (file):
  file INSTALL cannot find
  "/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib/libomp.so":
  No such file or directory.
Call Stack (most recent call first):
  projects/openmp/runtime/cmake_install.cmake:42 (include)
  projects/openmp/cmake_install.cmake:42 (include)
  projects/cmake_install.cmake:42 (include)
  cmake_install.cmake:57 (include)


make: *** [install] Error 1

Kind regards,

[Z10Frank]

Hello, thank you, is there any intermediate step missing?
I mean, after downloading the llvm-ompss version, in theory can I directly compile clang and OpenMP like this?

tar -xvf llvm-github-release-2021.11.tar
cd llvm-github-release-2021.11
cd llvm
mkdir install
cd install

module load gcc/11.2.0/gcc-4.8.5 
module load anaconda2/2019.10/gcc-9.2.0 
module load cmake/3.16.2/intel-19.0.3.199 
module load openmpi/4.0.2/gcc-9.2.0
module load zlib/1.2.9/gcc-9.2.0

export INSTALLATION_PREFIX_NANOS6=/gpfs/users/massimof/nanos6/install
export INSTALLATION_PREFIX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install

cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6

make -j 10
make install

Specifically, is there something to build with LLVM before, to enable the subsequent LLVM compilation of clang and OpenMP? It is my first experience with LLVM, so any crucial step you may consider obvious is probably missing from my attempts.

[rofirrim]

Hi @Z10Frank,

I tried to reproduce your problem without success.

I used the following steps in case you want to try.

$ wget https://github.com/bsc-pm/llvm/archive/refs/tags/github-release-2021.11.tar.gz
$ tar xf github-release-2021.11.tar.gz 
$ cd llvm-github-release-2021.11/
$ mkdir build
$ cd build/
$ make ../llvm -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp"  -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$NANOS6_INSTALL
$ make -j$(nproc)
$ make install

My environment is an Ubuntu 18.04 LTS and cmake is 3.19.3.

[Z10Frank]

Thank you! Now I hope we are converging, the build of clang and OpenMP seems to arrive almost to the end. My error was to create a destination install directory that was also the place where CMake was called to build clang and OpenMP.

Creating a build directory, different from the $INSTALL_PREFIX directory like in your commands, made all smooth except for the make install step:

Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:41 (file):
  file cannot create directory: /lib/cmake/llvm.  Maybe need administrative
  privileges.

Since I have no administrative privileges in this cluster, is it possible to specify a different path for CMake to create the llvm directory in this last step?

[rofirrim]

Since I have no administrative privileges in this cluster, is it possible to specify a different path for CMake to create the llvm directory in this last step?

I didn't need administrative privileges earlier. cmake should honour CMAKE_INSTALL_PREFIX here.

Maybe something is still off in your environment?

[Z10Frank]

Thank you! Fixing the environment the LLVM-OmpSs compiler with OpenMP is built successfully, now I'll try to use it to compile Smilei.