keck91/drboxguide

Installation Guide for caffe / DRBox from Scratch

Guide to install caffe with CUDA 8.0, cuDNN 5.1, Python 2.7 under Ubuntu 16.04 with GPU support in order to use DRBox

I was new to Linux and Deep-Learning. After three weeks I finally managed to install CUDA, cuDNN and the caffe framework to use the repository DRBox by liulei01. I combined different install guides and the solution below worked for me. The makefile.config that I used is also attached. You can use this guide for an installation of caffe only, too. In that case, use the BVLC repository instead of DRBox. I hope that it will save you some time and energy. Let me know if it worked for you.
I used the following guides:
https://medium.com/@zhanwenchen/install-cuda-and-cudnn-for-tensorflow-gpu-on-ubuntu-79306e4ac04e
https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide
https://github.com/adeelz92/Install-Caffe-on-Ubuntu-16.04-Python-3
https://medium.com/@mengjiunchiou/build-opencv-caffe-with-cuda-9-0-on-ubuntu-16-04-b2794a41612d \

Install Ubuntu 16.04.5 LTS (Xenial Xerus)

Download 64-bit desktop image from Ubuntu website
Install Ubuntu

Install CUDA 8.0

Install NVIDIA driver version 384

sudo apt-get install nvidia-384 nvidia-modprobe

reboot

Download CUDA runfile installer from NVIDIA website
Choose CUDA 8.0
Extract File

chmod +x cuda_8.0.61_375.26_linux.run
./cuda_8.0.61_375.26_linux.run --extract=$HOME

Execute CUDA installer

sudo ./cuda-linux64-rel-8.0.61-21551265.run

Install sample tests

sudo ./cuda-samples-linux-8.0.61-21551265.run

Configure the runtime library

sudo bash -c "echo /usr/local/cuda/lib64/ > /etc/ld.so.conf.d/cuda.conf"
sudo ldconfig

It is also recommended for Ubuntu users to append string /usr/local/cuda/bin to system file /etc/environment so that nvcc will be included in $PATH

sudo apt-get install vim
sudo vim /etc/environment

add :/usr/local/cuda/bin (including the ":") at the end of the PATH-string (inside the quotes)
You can use "i" do edit the text and "wq" to save and exit. Next, reboot

reboot

Test installation

cd /usr/local/cuda-8.0/samples
sudo make

After completion (takes a while), run tests

cd /usr/local/cuda/samples/bin/x86_64/linux/release
./deviceQuery

Install cuDNN 5.1

Go to cuDNN download page and select cuDNN version 5.1 for CUDA 8.0
Download all 3 .deb-files (runtime library, developer library & code samples)
Install them in the same order

sudo dpkg -i libcudnn5_5.1.10-1+cuda8.0_amd64.deb

sudo dpkg -i libcudnn5-dev_5.1.10-1+cuda8.0_amd64.deb

sudo dpkg -i libcudnn5-doc_5.1.10-1+cuda8.0_amd64.deb

Configure CUDA and cuDNN libraries

vim ~/.bashrc

# put the following line in the end or your .bashrc file
export LD_LIBRARY_PATH="LD_LIBRARY_PATH=${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}/usr/local/cuda/extras/CUPTI/lib64"

source ~/.bashrc

Install OpenCV

This part is entirely taken from Medium. Install dependencies (note: "libtiff4-dev" may not be available. Use "libtiff5-dev" instead)

sudo apt-get install build-essential

sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev

sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev

sudo apt-get install libtiff4-dev libopenexr-dev libeigen2-dev yasm libopencore-amrnb-dev libtheora-dev libvorbis-dev libxvidcore-dev

sudo apt-get install python-tk libeigen3-dev libx264-dev libqt4-dev libqt4-opengl-dev sphinx-common texlive-latex-extra libv4l-dev default-jdk ant libvtk5-qt4-dev

Get opencv-2.4.9 using wget

cd

wget http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.4.9/opencv-2.4.9.zip

unzip opencv-2.4.9.zip

cd opencv-2.4.9

mkdir build

cd build

proceed like explained on Medium. I followed the instructions, even though I use CUDA 8.0 and not CUDA 9.0.

Install caffe/drbox

Update and upgrade

sudo apt-get update
sudo apt-get upgrade

Install dependencies

sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev
sudo apt-get install libatlas-base-dev
sudo apt-get install libopenblas-dev
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev

Download drbox repository
Note: If you just want the original caffe repository, clone from BVLC directly and use "make pycaffe" instead of "make py".

sudo apt-get install git
git clone https://github.com/liulei01/DRBox.git
cd DRBox

Add caffe python path to the $PYTHONPATH

export PYTHONPATH=/path/to/caffe/python

Copy Makefile.config

cp Makefile.config.example Makefile.config

Edit Makefile.config

vim Makefile.config

# Comment CPU_ONLY
# CPU_ONLY = 1
# Change CUDA directory
CUDA_DIR := /usr/local/cuda-8.0
#In CUDA_ARCH, delete before *30 for compatibility
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
             -gencode arch=compute_35,code=sm_35 \
             -gencode arch=compute_50,code=sm_50 \
             -gencode arch=compute_52,code=sm_52 \
             -gencode arch=compute_61,code=sm_61
# Edit PYTHON_INCLUDE
PYTHON_INCLUDE := /usr/include/python2.7 \
		/usr/lib/python2.7/dist-packages/numpy/core/include
# Uncomment Python layer
WITH_PYTHON_LAYER := 1
# Edit INCLUDE_DIRS & LIBRARY_DIRS
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include \
                /usr/include/hdf5/serial/
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial     

Close and save file
Link some other files

cd /usr/lib/x86_64-linux-gnu

sudo ln -s libhdf5_serial.so.10.1.0 libhdf5.so

sudo ln -s libhdf5_serial_hl.so.10.0.2 libhdf5_hl.so 

Install python requirements

sudo apt-get install python-pip
cd python
for req in $(cat requirements.txt); do pip install --no-cache-dir $req; done
cd ..

Edit Makefile

vim Makefile

and replace

NVCCFLAGS += -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

with this

NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CXX) -Xcompiler -fPIC $(COMMON_FLAGS)

Also, open the file CMakeLists.txt and add the following line:

# ---[ Includes
set(${CMAKE_CXX_FLAGS} "-D_FORCE_INLINES ${CMAKE_CXX_FLAGS}")

Then

make clean

make all

make test

Note: Despite the runtest failed with DRBox, everything works fine.
If you only want caffe and not DRBox, the runtest should pass, though!
Use "make runtest" and "make pytest".

make py

Test with Python

python
import caffe
caffe.__version__

I also included caffe in my PATH in order to use the module caffe in command line mode

vim ~/.bashrc
# add the following line
export PATH=path/to/caffe/build/tools:$PATH

Comments

After importing caffe to python, I keep getting warnings about binary incompatibility:

[...]
RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
[...]

Issue #6678 covered this. I ignored them, too

Update

After "import caffe" I got error messages:

ImportError: libcublas.so.8.0: cannot open shared object file: No such file or directory

ImportError: No module named skimage.io

I installed a few packages according to 5716 and 8.

sudo apt-get install python-matplotlib python-numpy python-pil python-scipy
sudo apt-get install python-skimage

In bashrc I added

export CAFFE_ROOT=/home/username/caffe
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

drboxguide's Issues

cannot find -l　boost_thread

Here is what I met when I do 'make ':

CXX .build_release/src/caffe/proto/caffe.pb.cc
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so.1.0.0-rc3
/usr/bin/ld: cannot find -l　boost_thread
collect2: error: ld returned 1 exit status
Makefile:568: recipe for target '.build_release/lib/libcaffe.so.1.0.0-rc3' failed

I have no idea about how to find the boost_thread.

I found libboost_thread is in usr/lib ,and added it to ./bashrc
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib
I also tried reinstall boost,then copy to usr/lib

but question still resist.How can I solve this?
Thank you in advance!

Drbox algorithm evaluation code

Hello, do you have Drbox algorithm evaluation code?