build failing

[nvlabs]

build failing

[Tom94]

That's really useful information, thank you!

I have a very similar setup here with the main difference being that I only have a single RTX 3090. This leads me to believe that the automatic GPU architecture detection/treatment in the CMake project file is problematic when multiple GPUs with different architectures are present.

To verify, could you try the following two steps?

1. Could you try building with make VERBOSE=1 and paste the full log here?
2. Please try replacing the block

include(FindCUDA/select_compute_arch)
cuda_select_nvcc_arch_flags(ARCH_FLAGS)
list(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})

set(CMAKE_CUDA_ARCHITECTURES off)

with the single line

set(CMAKE_CUDA_ARCHITECTURES 86)

in CMakeLists.txt and see if that fixes the problem on your end.

Thank you!

[Tom94]

Hi there, more information about your system would be helpful:

1. Which OS are you running? And on which version?
2. Which GPUs are installed in the system?
3. Which version of CUDA and which version of the NVIDIA driver is installed?

[Tom94]

Another one that might be helpful, actually: which version of CMake are you running? (This is useful, because CMake has recently changed a lot about their CUDA support.)

[rathken]

Ah yes sorry... That would have been smart of me.

Ubuntu 20.04
RTX 3090
RTX 4000
CUDA 11.5 + 495.44
cmake version 3.21.0-rc3

I note that if I role back to commit cdb419d it builds and runs as expected.

[rathken]

Replacing the block of code did fix the issue..... Thanks for this great bit of code!

[Tom94]

Hi @rathken . Could you try compiling again with the latest version of this repository? I've tried to fix the multi-GPU issue and would like to make sure. Thanks!

[rathken]

Will do today.

[rathken]

It took a while to build, but it built without any issues....

[Tom94]

Amazing, thank you very much for testing!

[jdiazram]

Hi everyone.
I have a similar problem,

gmake[2]: *** [dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/build.make:216: dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/src/fully_fused_mlp.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:288: dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

but a don't find the block of code that your comment in CMakelist.txt:

include(FindCUDA/select_compute_arch)
cuda_select_nvcc_arch_flags(ARCH_FLAGS)
list(APPEND CUDA_NVCC_FLAGS ${ARCH_FLAGS})

set(CMAKE_CUDA_ARCHITECTURES off)

Only I find the:
set(CMAKE_CUDA_ARCHITECTURES ${TCNN_CUDA_ARCHITECTURES})

and replace for:
set(CMAKE_CUDA_ARCHITECTURES 86)

And the new error is similar to the first one:

gmake[2]: *** [dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/build.make:216: dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/src/fully_fused_mlp.cu.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:288: dependencies/tiny-cuda-nn/CMakeFiles/tiny-cuda-nn.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

I have:
Ubuntu 22.04 LTS
RTX 3080 ti mobile
NVIDIA-SMI 515.48.07
Driver Version: 515.48.07
CUDA Version: 11.7
CMake version 3.22.1

Thx for your advice