CUDA 7.5 fails with pip install and docker (Ubuntu 14.04)

[soumith]

CUDA 7.5 fails with pip install and docker (Ubuntu 14.04)

[zdx3578]

For this error
ImportError: libcudart.so.7.5: cannot open shared object file: No such file or directory

I have executed this command to solve it
sudo ldconfig /usr/local/cuda/lib64

[lukesimo]

@ebrevdo yes

printenv LD_LIBRARY_PATH
/usr/local/cuda-7.5/lib64

[ebrevdo]

Out of curiosity, have you set your LD_LIBRARY_PATH to your cuda installation's lib64 directory?

[soumith]

@nivwusquorum haha that's a terrible workaround, as it'll start issues with other libraries. Thanks a lot though. I'm installing CUDA 7.0

[lukesimo]

@nivwusquorum no.

I'm running into the same issue. Looks like I'll be downgrading then.

[mdda]

On the subject of CUDA library versions ... CUDA 7.0 works for me (as expected), but it really insists on cuDNN 6.5 (which Nvidia now has as 'legacy').

Exact same library locations, etc, but downgrading from cuDNN 7.0 to 6.5 worked.

[graphific]

yes its within the tensorflow code, so just some simple python hacking wont solve it :)
(_pywrap_tensorflow.so when you pip install the binary):

_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

[emergix]

i assume i have same problem:
I have the 7.5 installed with tensorflow and when I try (like in the tutorial about gpu)
with tf.device('/gpu:0'):
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
print(c)
sess.run(c)

it breaks !
(in torch7, I have no pbs with gpus)

[jimaldon]

Can we assume tensorflow to be forward compatible with cuda 7.5?

[andorremus]

I've got the same problem.

So are you saying that downgrading from cuda 7.5 to 7 should do the trick?

[andorremus]

If it helps, I've installed cuda toolkit 7.0 and changed the bash profile reference to the new one and it works.

[emergix]

yes you do not need to reinstall the cuda 7.0 driver, just provide the path to libcudart.so.7.0 at the end of the LD_LIBRARY_PATH variable. After discussing of it with someone of the team, they told me a strange story. It appears thar the old CUDA drivers are very much in demand by the people using AWS of amazon !
can someone confirm ?

[ebrevdo]

Long story short: tensorflow currently requires cuda 7.0. If you install version 7.0 in a separate directory from 7.5, and point tensorflow at it via the configure script (or LD_LIBRARY_PATH), it will work. Leaving this open to track future upgrades to the 7.5 SDK.

[FabHan]

I'm curious why it is hard to upgrade to CUDA 7.5 and CuDNN v3? Anyone helps me understand?

[pannous]

@emergix so you got it to work just by providing libcudart.so.7.0 without reinstalling / downgrading to old cuda?

[andorremus]

Yes. It doesn't require you to uninstall the previous one. You can just install them separately and reference v7.0 in your bashrc file

[pannous]

Thanks! Workaround confirmed here.

[fivejjs]

symbolic link libcuda_.7.5 to libcuda_.7.0
It works.

[esube]

This issue and co. are open for more than two months now. Is there any progress to support 7.5 cuda and 7.0 cudnn other than the workarounds? This could be a turn off for some people who already have been working with 7.5 cuda for a while.

[kmhofmann]

I completely agree. Ideally (assuming reasonable API/ABI stability on NVIDIA's side), TensorFlow should not be dependent on specific older versions of CUDA and cuDNN. (I'd rather understand it if the latest version was required to make use of certain features.)
This is the one issue that puts me off using TensorFlow with GPU support.

[cmcneil]

+1 Please support CUDA 7.5

[ville-k]

If you want to try out CUDA 7.5 under Linux, you could try building my pull request branch:
#664
It adds support for CUDA on OSX and uses CUDA 7.5 when building under OSX. I haven't tried 7.5 under Linux, but it seems to work ok with OSX.
The only change you'd need to make is to edit the configure file and set CUDA_VERSION='7.5' when Linux is detected. Lines 48-49 would look like:

if [ "$OSNAME" == "Linux" ]; then
  CUDA_VERSION='7.5'

[wangg12]

@ebrevdo @andorremus Could you share the method of installing different cuda(7.0 and 7.5) separately?

@emergix @fivejjs Does symbolic linking libcuda.7.5 to libcuda.7.0 or just providing libcuda.7.0 have side effects?

[andorremus]

@wangg12 A simple google search "install cuda 7.0/7.5 for 'OS here'" should surface all of the information you need

[wangg12]

@andorremus I can find a lot of instructions for installing a single version cuda, but I can't find any information about installing 7.0 and 7.5 simultaneously. How do you make it?

[andorremus]

Well you install them separately as they are both standalone and the you point to the one you want in the bashrc file afterwards

[wangg12]

@andorremus Thanks.

[wangg12]

@andorremus Sorry to bother you. One more question, do I need to install driver when I install the second cuda version or I just need to install the libraries?

[andorremus]

I think if you've already got the drivers it should be only the libraries
but I'm not really sure on that.

On Tuesday, 19 January 2016, Wang Gu [email protected] wrote:

@andorremus https://github.com/andorremus Sorry to bother you. One more
question, do I need to install driver when I install the second cuda
version or I just need to install the libraries?

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

[esube]

Just need the toolkit and cudnn. You only need one driver (the latest for
your system, that is)
On Jan 19, 2016 3:45 AM, "Wang Gu" [email protected] wrote:

@andorremus https://github.com/andorremus Sorry to bother you. One more
question, do I need to install driver when I install the second cuda
version or I just need to install the libraries?

—
Reply to this email directly or view it on GitHub
#20 (comment)
.

[wangg12]

Thank you @andorremus @esube. I've installed cuda 7.5 and then install 7.0's libraries. It seems to work for me.

[bhack]

/cc @ducha-aiki

[ducha-aiki]

@wangg12
could you please clarify, at the end, CUDA 7.0 is needed for only installing, or operate as well? Same for cudnn. My concerns are having two drivers (your and other experiences say it is OK) and performance. CuDNN 2 and 3/4 differs really a lot. Sorry for stupid question :)

@bhack thanks for tagging!

[ihciah]

Linking libcudart.so.7.0 to 7.5 solved the problem.
And maybe it should also link libcufft and some other files.
Please support cuda 7.5 and cudnn v3

[joelclim]

Installing the download for Ubuntu 14.10 works for me. After installing 7.5 as prescribed on NVIDIA's website. I downloaded the Local Package Installer and used the Synaptic Package Manager to install. My installation looks like this:

lrwxrwxrwx 1 root root 8 Feb 2 15:27 cuda -> cuda-7.0
drwxr-xr-x 13 root root 4096 Feb 2 15:27 cuda-7.0
drwxr-xr-x 13 root root 4096 Feb 2 15:09 cuda-7.5

Make sure the PATH and LD_LIBRARY_PATH variables reflect the right location of the cuda toolkit you use. Using the path the cuda link works very well.

[vrv]

@zheng-xq recently added a feature so that:

TF_UNOFFICIAL_SETTING=1 ./configure

allows you to configure which version of cuda/cudnn/etc to use without requiring manual symlinks.

[aboulch]

Hello,

I replaced cuda 7.0 -> 7.5 and cudnn 6.5 -> 7.0 directly in the code.
Seems to work on ubuntu 14.04.
https://github.com/aboulch/tensorflow

I did :

perl -pi -e 's/7.0/7.5/g' configure
perl -pi -e 's/6.5/7.0/g' configure
perl -pi -e 's/7.0/7.5/g' tensorflow/stream_executor/dso_loader.cc
perl -pi -e 's/6.5/7.0/g' tensorflow/stream_executor/dso_loader.cc
perl -pi -e 's/7.0/7.5/g' tensorflow/core/platform/default/build_config/BUILD
perl -pi -e 's/6.5/7.0/g' tensorflow/core/platform/default/build_config/BUILD
perl -pi -e 's/7.0/7.5/g' third_party/gpus/cuda/BUILD
perl -pi -e 's/6.5/7.0/g' third_party/gpus/cuda/BUILD
perl -pi -e 's/7.0/7.5/g' third_party/gpus/cuda/cuda_config.sh
perl -pi -e 's/6.5/7.0/g' third_party/gpus/cuda/cuda_config.sh

[esafak]

I pip installed tf 0.7 with CUDA 7.5 and cuDNN4 on Ubuntu 14.04, and tf still sought CUDA 7.0 (libcudart.so.7.0). I had to symlink it to the 7.5 version to get it to work.

[timsainb]

@esafak how dod you do that?

[EderSantana]

@esafak do you mind explaining what you did? Perhaps in a gist?

[vrv]

The pip packages for 0.7.0 should be unversioned -- they should look for libcudart.so, do you have logs to suggest this isn't happening?

[timsainb]

(tensorflow)tsainbur@txori:~$ pip freeze | grep tensorflow
tensorflow==0.7.0

Installed using the virtualenv method, where I had to rename the whl to:
pip install --upgrade tensorflow-0.7.0-cp27-none-linux_x86_64.whl
to avoid
tensorflow-0.7.0-py2-none-linux_x86_64.whl is not a supported wheel on this platform.

Now I get
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

[vrv]

Weird, we were supposed to have built the wheels unversioned. https://github.com/tensorflow/tensorflow/search?utf8=%E2%9C%93&q=libcudart shows no missing hardcoding of 7.0 anywhere...

@martinwicke, @jendap, any ideas?

[esafak]

@timsainb I changed to the appropriate directory and executed sudo ln -s libcudart.so.7.5 libcudart.so.7.0. But you might not have to do this if the next paragraph applies.

@EderSantana @vrv I found the problem: the new installation of CUDA did not overwrite or otherwise remove CUDA 7.0, but installed it in /usr/local/cuda7.5 leaving /usr/local/cuda, which got picked up by $LD_LIBRARY_PATH. I removed the old files using

sudo apt remove cuda-command-line-tools-7-0 cuda-core-7-0 cuda-cublas-7-0 cuda-cublas-dev-7-0 cuda-cudart-7-0 cuda-cudart-dev-7-0 cuda-cufft-7-0 cuda-cufft-dev-7-0 cuda-curand-7-0 cuda-curand-dev-7-0 cuda-cusolver-7-0 cuda-cusolver-dev-7-0 cuda-cusparse-7-0 cuda-cusparse-dev-7-0 cuda-documentation-7-0 cuda-driver-dev-7-0 cuda-license-7-0 cuda-misc-headers-7-0 cuda-npp-7-0 cuda-npp-dev-7-0 cuda-nvrtc-7-0 cuda-nvrtc-dev-7-0 cuda-repo-ubuntu1404-7-0-local cuda-samples-7-0 cuda-toolkit-7-0 cuda-visual-tools-7-0

Now it seems to work.

[bhack]

See #1166

[tboquet]

I have the same problem using a docker container and the nvidia-docker tool with an image based on the cuda:7.5-cudnn4-devel image. I tried to use @esafak suggestion and I endup with:

root@9ad6f042d266:/usr/local/cuda/lib64# readlink -f libcudart.so.7.0
/usr/local/cuda-7.5/targets/x86_64-linux/lib/libcudart.so.7.5.18

root@9ad6f042d266:/usr/local/cuda/lib64# readlink -f libcudart.so
/usr/local/cuda-7.5/targets/x86_64-linux/lib/libcudart.so.7.5.18

But i still have:

...
  File "/opt/conda/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: libcudart.so.7.0: cannot open shared object file: No such file or directory

If I run python in /usr/local/cuda/lib64 I have errors related to #808:

I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:99] Couldn't open CUDA library libcuda.so. LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:121] hostname: 9ad6f042d266
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:146] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:257] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  352.63  Sat Nov  7 21:25:42 PST 2015
GCC version:  gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04)
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:150] kernel reported version is: 352.63
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1051] LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64:
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1052] failed to find libcuda.so on this system: Failed precondition: could not dlopen DSO: libcuda.so; dlerror: libcuda.so: cannot open shared object file: No such file or directory
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so locally

Did you manage to find the problem?

Tx!

[ruffsl]

@tboquet , your unable to find libcuda.so might be related to the bottom of this rabbit hole: #970 (comment)

[tboquet]

@ruffsl, tx, I will try to build the devel version and post my results.

[vrv]

Release 0.7.1 wheels now are built assuming cuda-7.5 and cudnn v4. If you want earlier versions supported, you can build from sources and set the appropriate library paths / versions during ./configure.

[zheng-xq]

If you build from the latest TF, it would ask you for Cuda SDK and Cudnn versions. You can point it to a different version.

[monajalal]

@fivejjs how did you do so? can you please write the detailed command?

[brando90]

why does ldconfig /usr/local/cuda/lib64 solve it @zdx3578

I currently cannot sudo in the cluster Im using so that didnt work, any alternatives?

[philoniare]

I have just installed CUDA v8 and the issue still persists. I got the following error:

ImportError: libcudart.so.7.5: cannot open shared object file: No such file or directory

In case anyone else encounters the same issue, the solution is to create symlink.

1. Go to the cuda directory: cd /usr/local/cuda/lib64
2. Create the symlink: sudo ln -s libcudart.so libcudart.so.7.5
   Then, you'll be able to import tensorflow. To check if it is using the GPU, try running:

sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

[Euphemiasama]

Thank you @philoniare I too have CUDA 8 and it works now !