Install fails on AWS DLAMI v. 29.0

[lmnt-com]

Install fails on AWS DLAMI v. 29.0

[amurashov]

same error with:

Deep Learning AMI (Ubuntu 18.04) Version 29.0

no idea why:

import os
print(os.environ['LD_LIBRARY_PATH'])

/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/extras/CUPTI/lib64:/usr/local/cuda-10.1/lib:/usr/local/cuda-10.1/efa/lib:/opt/amazon/efa/lib:/opt/amazon/efa/lib64:/usr/lib64/openmpi/lib/:/usr/local/lib:/usr/lib:/usr/local/mpi/lib:/lib/:/usr/lib64/openmpi/lib/:/usr/local/lib:/usr/lib:/usr/local/mpi/lib:/lib/::/home/ubuntu/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow

---

I think it compiles with wrong CUDA - 10.0 while TF clearly uses 10.1 in this setup, and 10.0 is NOT on the PATH of default TF installation. I think HASTE takes 10.0 as it is the default CUDA dir in /usr/local/cuda -> which points to cuda-10.0, not cuda-10.1.

Is it possible to pull CUDA version from TF during the make haste_tf run like you do with include directories may be?

[amurashov]

Actually adding /usr/local/cuda-10.0/lib64 to the LD_LIBRARY_PATH solved it.

[amurashov]

Or - laternatively - changing /usr/local/cuda to point to /usr/local/cuda-10.1 instead of /usr/local/cuda-10.0 at haste compile time.

[sharvil]

/usr/local/cuda is the standard path for the "current" installation of CUDA Toolkit. If multiple versions of CUDA Toolkit are present, /usr/local/cuda should point to the one you'd like to use. If Amazon's DLAMI builds TensorFlow against 10.1 but points /usr/local/cuda to 10.0, that's a misconfiguration on their part.

Regardless, I've made it possible to override the CUDA Toolkit path to build against without having to modify the Makefile. You can either export the $CUDA_HOME environment variable or specify it on the command-line while building Haste:

CUDA_HOME=/usr/local/cuda-10.1 make

Thanks for pushing through these issues and reporting them, @amurashov. I appreciate it.

[sharvil]

There are two issues with detecting which version of CUDA TensorFlow is built against. First, there's no great way to do that as far as I can tell. The second is that Haste is multi-framework and PyTorch/TensorFlow could be built against different CUDA versions.

I think making it easier for the user to choose the CUDA version to build against is the best I can do here. Closing this out since there's no further action I can take on my part. Please re-open if you have suggestions on what else I could do.