mdx from cmake pipeline results in segfault with conda compilers

[isce-framework]

mdx from cmake pipeline results in segfault with conda compilers

[piyushrpt]

This is on a linux machine. When building on OS X with macports, everything works as expected.

[lijun99]

Conda compilers (gcc_linux-64, gxx_linux-64,gfortran_linux-64) seem to be lack of OpenMP support, which caused me problems. I had to use other standard sets of gcc/g++/gfortran compilers. I use libraries from conda, including motif, x11, and use -DCMAKE_PREFIX_PATH=$CONDA_PREFIX to enforce it. This seems to work for me on Linux, even with CUDA (needs cmake=3.17).

[piyushrpt]

On linux, openmp should be available with the compilers. This is my cmake line which seems consistent with your experience

cmake -DCMAKE_INSTALL_PREFIX=$installdir -DCMAKE_PREFIX_PATH=$CONDA_PREFIX $srcdir

compilation has not been a problem. I think it has to do with some optimizations which scons suppresses.

[piyushrpt]

Yes there is one minor issue with sysroot and libpthread but the can be overcome by just extending this part of the cmake line

-DCMAKE_PREFIX_PATH=$CONDA_PREFIX:$CONDA_BUILD_SYSROOT

[piyushrpt]

The solution it turns out is to remove any "-fopenmp" in the compile lines for the fortran files only for mdx. Any easy cmake solution for this.
Will need to check that compiler is gnu and if fopenmp is in the command line and then take it out,

[lijun99]

I found my mdx on linux didn't work either. The cause to my problem, as I figured out, is that -lXm needs to appear in front of -lXt. Somehow, in cmake generated link.txt, libXm.so shows always behind libXt.so.

@piyushrpt's PR #156 seems to have addressed the issue. But adding a dependency to enforce the lib link order might be better than simply adding "Xm Xt".

[piyushrpt]

I agree @lijun99 . I think the right way to do this is with the targets. The PR needs to be modified possibly by Ryan - who understands the target setup better.

[piyushrpt]

@lijun99 - turns out target_link_libraries preserves order - https://cmake.org/pipermail/cmake/2008-November/025612.html
So, just listing them in correct sequence should do.

[lijun99]

Not for me. Maybe a bug in CMake? Can you check your cmake build: components/contrib/mdx/CMakeFiles/mdx.dir/link.txt file?

[rtburns-jpl]

What cmake versions are you using? If < 3.14 the x11 targets are created manually so I might need to add link dependencies for libxt/libxm. Also the motif target is created manually so it might need transitive link dependencies for x11 libs used by motif.

[lijun99]

I am using 3.17. I guess it's a bug in Cmake: it preserves the link order if I use directly ${MOTIF_LIBRARIES} instead of the Motif::Motif.

[piyushrpt]

You are right. I made some changes to still use targets and get file names through them.
I'm ok with either solution - to use motif directly or fix the interface.

The problem is that X11::X11 and X11::Xt are labelled as UNKNOWN_LIBRARY and Motif::motif is an INTERFACE_LIBRARY and mixing them seems to not make it happy. Making them all the same types, strings or unknown_library will fix it.

[rtburns-jpl]

Does https://github.com/rtburns-jpl/isce2/tree/motif-deps fix this for you? I haven't been able to repro locally but this might be a more correct approach

[piyushrpt]

Doesnt work. I tried adding dependencies and X11::X11 to that line as wel. That didnt work either.

[rtburns-jpl]

I think I need more info to repro this. I'm able to build fine using my motif-deps branch and x11 libs from conda-forge. Can't seem to trigger any segfaults.

> ldd components/contrib/mdx/mdx                                          (isce2-dev-mdx) 
	linux-vdso.so.1 (0x00007ffec6f6b000)
	libXm.so.4 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libXm.so.4 (0x00007f19047e0000)
	libXt.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libXt.so.6 (0x00007f1904774000)
	libICE.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libICE.so.6 (0x00007f1904756000)
	libSM.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libSM.so.6 (0x00007f190454b000)
	libX11.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libX11.so.6 (0x00007f1904408000)
	libgfortran.so.4 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libgfortran.so.4 (0x00007f19042da000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f1904160000)
	libgcc_s.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libgcc_s.so.1 (0x00007f190414c000)
	libquadmath.so.0 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/libquadmath.so.0 (0x00007f1904112000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f1903f4b000)
	libXmu.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXmu.so.6 (0x00007f1903f2f000)
	libuuid.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libuuid.so.1 (0x00007f1903d2b000)
	libXext.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXext.so.6 (0x00007f1903d14000)
	libXft.so.2 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXft.so.2 (0x00007f1903afd000)
	libfontconfig.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libfontconfig.so.1 (0x00007f19038b9000)
	libxml2.so.2 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libxml2.so.2 (0x00007f1903751000)
	libicui18n.so.58 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libicui18n.so.58 (0x00007f19034d2000)
	libicuuc.so.58 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libicuuc.so.58 (0x00007f190331f000)
	libicudata.so.58 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libicudata.so.58 (0x00007f1901a1b000)
	liblzma.so.5 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./liblzma.so.5 (0x00007f19019f2000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f19019e7000)
	libiconv.so.2 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libiconv.so.2 (0x00007f1901701000)
	libfreetype.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libfreetype.so.6 (0x00007f190164d000)
	libXrender.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXrender.so.1 (0x00007f1901640000)
	libxcb.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libxcb.so.1 (0x00007f1901612000)
	libXau.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXau.so.6 (0x00007f190160d000)
	libXdmcp.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libXdmcp.so.6 (0x00007f1901605000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f19015ff000)
	libjpeg.so.9 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libjpeg.so.9 (0x00007f19013c3000)
	libpng16.so.16 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libpng16.so.16 (0x00007f190138a000)
	libz.so.1 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/./libz.so.1 (0x00007f1901368000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f1901346000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f1906148000)
	libstdc++.so.6 => /home/user/pkg/conda/envs/isce2-dev-mdx/lib/././libstdc++.so.6 (0x00007f19011d2000)

> conda list | grep xorg                                                  (isce2-dev-mdx) 
xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
xorg-libice               1.0.10               h516909a_0    conda-forge
xorg-libsm                1.2.2                h470a237_5    conda-forge
xorg-libx11               1.6.9                h516909a_0    conda-forge
xorg-libxau               1.0.9                h14c3975_0    conda-forge
xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
xorg-libxext              1.3.4                h516909a_0    conda-forge
xorg-libxft               2.3.2                h7c24a56_3    conda-forge
xorg-libxmu               1.1.3                h516909a_0    conda-forge
xorg-libxrender           0.9.10            h516909a_1002    conda-forge
xorg-libxt                1.1.5             h516909a_1003    conda-forge
xorg-printproto           1.0.5             h14c3975_1001    conda-forge
xorg-renderproto          0.11.1            h14c3975_1002    conda-forge
xorg-xextproto            7.3.0             h14c3975_1002    conda-forge
xorg-xproto               7.0.31            h14c3975_1007    conda-forge

This is an arch linux box but afaict the entire toolchain and libs are from conda_cos6 and should work anywhere.

[lijun99]

OK, I repeated the segfault with conda gnu compiler (gcc_linux-64 7.5.0) on Ubuntu. The generated flags.cmake for mdx appears as

# compile C with /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-cc                                                                           
# compile Fortran with /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-gfortran                                                               
C_FLAGS = -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/\
anaconda3/include

C_DEFINES = -DF77EXTERNS_LOWERCASE_TRAILINGBAR -DGFORTRAN -DIO64 -DNEEDS_F77_TRANSLATION -DSUN

C_INCLUDES =

Fortran_FLAGS = -fopenmp -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem\
 /opt/anaconda3/include   -ffixed-line-length-none -fno-range-check -fno-second-underscore

Fortran_DEFINES = -DF77EXTERNS_LOWERCASE_TRAILINGBAR -DGFORTRAN -DIO64 -DNEEDS_F77_TRANSLATION -DSUN

Fortran_INCLUDES = -I/opt/anaconda3/include  

Removing -fopenmp solves the problem, as @piyushrpt suggested. If I use the system gcc/gfortran compiler, there is no -fopenmp flag and mdx runs ok.

The problem seems to be with conda gcc compiler.

[rtburns-jpl]

Interesting, thank you. If I manually add the -fopenmp flag I'm getting segfaults now. But I still don't see why this is showing up for you - this flag should only be added via the OpenMP::OpenMP_C/CXX/Fortran targets which are not used by mdx. Is there a variable in your environment that is causing these to be picked up by cmake?

[lijun99]

$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-gfortran -v
Reading specs from /opt/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/specs
Reading specs from /opt/anaconda3/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/specs
COLLECT_GCC=/opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-gfortran.bin
COLLECT_LTO_WRAPPER=/opt/anaconda3/bin/../libexec/gcc/x86_64-conda_cos6-linux-gnu/7.5.0/lto-wrapper
Target: x86_64-conda_cos6-linux-gnu
Configured with: /home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/configure --build=x86_64-build_pc-linux-gnu --host=x86_64-build_pc-linux-gnu --target=x86_64-conda_cos6-linux-gnu --prefix=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/gcc_built --with-sysroot=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/gcc_built/x86_64-conda_cos6-linux-gnu/sysroot --enable-languages=c,c++,fortran,objc,obj-c++ --with-pkgversion='crosstool-NG 1.24.0.123_1667d2b' --enable-__cxa_atexit --disable-libmudflap --enable-libgomp --disable-libssp --enable-libquadmath --enable-libquadmath-support --enable-libsanitizer --enable-libmpx --with-gmp=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/.build/x86_64-conda_cos6-linux-gnu/buildtools --with-mpfr=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/.build/x86_64-conda_cos6-linux-gnu/buildtools --with-mpc=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/.build/x86_64-conda_cos6-linux-gnu/buildtools --with-isl=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/.build/x86_64-conda_cos6-linux-gnu/buildtools --enable-lto --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --enable-threads=posix --enable-target-optspace --enable-plugin --enable-gold --disable-nls --disable-multilib --with-local-prefix=/home/conda/feedstock_root/build_artifacts/ctng-compilers_1589464284303/work/gcc_built/x86_64-conda_cos6-linux-gnu/sysroot --enable-long-long --enable-default-pie
Thread model: posix
gcc version 7.5.0 (crosstool-NG 1.24.0.123_1667d2b) 

I guess when gfortran is configured with --enable-libgomp, CMake picks it up and adds -fopenmp flag. Maybe a solution is to override it as @piyushrpt did.

[piyushrpt]

conda sets "-fopenmp" in FFLAGS by default on linux I believe.

[piyushrpt]

The cmake for the PR should now be working. I'm wondering, if its simpler to setup interface library as X11ISCE::X11 X11ISCE::Xt in TargetX11.cmake all the time and move the logic there. Then using the interface library should work in mdx's CMakeList.txt.. The problem is that the new cmake exports X11 as an UNKNOWN_LIBRARY and sets INTERFACE_LINK_LIBRARIES there which leads to. I'm also convinced that the link order required by mdx is not what is recommended.

[rtburns-jpl]

Could you explain a little more what happens when you link in the wrong order and how I can repro that? I still have a feeling that I set up the Motif lib improperly and just need to fix that somehow. If not, I think your X11ISCE:: idea would be best so that we're not fudging with the internal x11 targets.

[lijun99]

When the -lXm -lXt are not linked in order, mdx returns Bad Windows (..) error. You may reproduce it by moving libXm.so behind libXt.so in link.txt. Another example is https://www.softintegration.com/chhtml/toolkit/demos/X11/motif/, hello.c. If you link it by -lXt -lXm -lX11, it compiles but the program doesn't work.

[rtburns-jpl]

Oh sorry, I understand that and have tested it locally. What I meant was that I added Xt as a dependency to motif in my motif-deps branch which fixed it for me, but @piyushrpt said it's still not working. I thought adding Xt as a dependency would enforce its linking order and am wondering why that's not the case

[piyushrpt]

X11::Xt lists X11::X11 as a INTEFACE_LINK_LIBRARY in FindX11 from cmake. The library order resolving algorithm goes for a spin because of that. I tried introducing a bunch of dependencies but that did not resolve the issue.

[piyushrpt]

I simplified the interface in TargetX11.cmake and this should work with both the old and new versions of cmake.

[rtburns-jpl]

• The issue isn't X11 vs Xt linking order, it's Xt vs Xm right? Xt depends on X11 so I think that's just my improper setup for cmake 3.12/13 that should be fixed. And w.r.t. Xt vs Xm, Motif seems to depend on Xt so I think I should have already added Xt to its link libraries?
• I'd like TargetX11 to mimic the cmake≥3.14 FindX11 functionality as closely as needed. If we need to create a pseudo-X11isce target for mdx that should be contained in contrib/mdx/CMakeLists.txt so we don't accidentally use it somewhere else, and so I can still use the FindX11-style libs elsewhere
• I'm not convinced that this isn't a cmake issue, since cmake documentation claims that "Repeated calls [to target_link_libraries] for the same <target> append items in the order called" https://cmake.org/cmake/help/v3.14/command/target_link_libraries.html#overview I don't want to complicate our recipes to fix a bug in cmake :S

[OliviaDdy]

how to install Motif?