CUDA link issue with incremental build

[esseivaju]

CUDA link issue with incremental build

[pcanal]

I.e. for each compilation of a RDC object file, it generates a 'unique' identifier and thus requires the running of the nvlink that depends on that .cu.o file.

The symbols:

__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_314c4826_29355
__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_ce41cbe7_29634

[drbenmorgan]

Hi @esseivaju, I'm not able to reproduce this with the default preset build using the current tip of develop (b427fde) and a local spack env with gcc 11.3 as:

1. Build from scratch cmake --preset=default ., run tests, everything fine
2. Add a new empty line in src/celeritas/global/ActionLauncher.device.hh
3. Run ninja again and the incremental build completes without error.

Which commit are you building, and which CUDA version (I'm on 11.7.1)?

[sethrj]

Could it be #847 introducing

celeritas/CMakeLists.txt

Line 177 in b427fde

celeritas_set_default(CMAKE_LINK_DEPENDS_NO_SHARED ON)

? If you run with -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF is the problem resolved?

[esseivaju]

I am building b427fde with gcc11.3 / cuda 12.1. Setting -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF does seem to fix the issue.

[pcanal]

I also see similar problem in an updated build. Let me investigate.

[pcanal]

: && /srv/software/gnu10/10.2.0/bin/g++ -Wall -Wextra -pedantic -g  test/CMakeFiles/testdetail_TestMacros.dir/TestMacros.test.cc.o -o test/testdetail_TestMacros  -Wl,-rpath,/nashome/p/pcanal/geant/gcc10/builds/scalar-cuda12/celeritas/lib64:/nashome/p/pcanal/geant/gcc10/install/opt/lib64  lib64/libtestcel_harness.so  lib64/libcorecel_final.so  /nashome/p/pcanal/geant/gcc10/install/opt/lib64/libgtest.so.1.10.0  -pthread  lib64/libcorecel.so && :
lib64/libcorecel.so: undefined reference to `__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_314c4826_29355'
lib64/libcorecel_final.so: undefined reference to `__fatbinwrap_4e0d1928_9_Filler_cu_438f1dcd_109866'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
pcanal@wc:celeritas $ touch ~/geant/sources/celeritas/src/corecel/data/detail/Filler.cu
pcanal@wc:celeritas $ ninja -j 1
[4/365] Linking CXX executable test/testdetail_TestMacros
FAILED: test/testdetail_TestMacros
: && /srv/software/gnu10/10.2.0/bin/g++ -Wall -Wextra -pedantic -g  test/CMakeFiles/testdetail_TestMacros.dir/TestMacros.test.cc.o -o test/testdetail_TestMacros  -Wl,-rpath,/nashome/p/pcanal/geant/gcc10/builds/scalar-cuda12/celeritas/lib64:/nashome/p/pcanal/geant/gcc10/install/opt/lib64  lib64/libtestcel_harness.so  lib64/libcorecel_final.so  /nashome/p/pcanal/geant/gcc10/install/opt/lib64/libgtest.so.1.10.0  -pthread  lib64/libcorecel.so && :
lib64/libcorecel.so: undefined reference to `__cudaRegisterLinkedBinary_4e0d1928_9_Filler_cu_ce41cbe7_29634'
lib64/libcorecel_final.so: undefined reference to `__fatbinwrap_4e0d1928_9_Filler_cu_438f1dcd_109866'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

[pcanal]

And indeed reverting #847 (completely for now) solves the problem (and of course doing ninja clean is a (lame) work-around)

[sethrj]

OK, maybe we need to just set CMAKE_LINK_DEPENDS_NO_SHARED conditionally on RDC (aka vecgeom + cuda). I'm surprised because VecGeom has this option set as well.

[pcanal]

I am building b427fde with gcc11.3 / cuda 12.1. Setting -D CMAKE_LINK_DEPENDS_NO_SHARED=OFF does seem to fix the issue.

That's odd. It works for me (with and without the space after the D).

[pcanal]

I'm surprised because VecGeom has this option set as well.

I see it in the source (and even I see that line executed) but I don't see it in the CMakeCache.txt (i.e. is it somehow not activated?)

[sethrj]

It's not being set as a cache variable, just as a directory scope one (so it will be "seen" by the targets but not saved from one configuration to the next).

[pcanal]

Ok. It is still 'uneffective' (it leads to the proper rebuilds ... I am checking why)

[pcanal]

And this is because there is a (hard) dependency from the result of nvlink and the static libraries. In Celeritas the dependency on the static library is only soft/order while the shared library dependency is 'hard' when it works and soft when it fails. This might be because we introduce the static library dependency as:

          target_link_options(${_target_final}
            PRIVATE
            $<DEVICE_LINK:$<TARGET_FILE:${_libstatic}>>
          )