Boost and Eigen 3.3 on Ubuntu 16.04 LTS

[opennmt]

Boost and Eigen 3.3 on Ubuntu 16.04 LTS

[guillaumekln]

If you want to make the nvcc warnings quiet, you can pass this flag to cmake:

-DCUDA_NVCC_FLAGS="-gencode=arch=compute_61,code=compute_61"

where 61 is the compute capability of your GPU.

Intel MKL is optional.

[guillaumekln]

Did you try to compile with static libraries?

cmake -DBUILD_SHARED_LIBS=OFF [...]

[loretoparisi]

[UPDATE]

I have found a solution for eigen3:

RUN curl http://security.ubuntu.com/ubuntu/pool/universe/e/eigen3/libeigen3-dev_3.3.2-1_all.deb -o libeigen3-dev_3.3.2-1_all.deb && dpkg -i libeigen3-dev_3.3.2-1_all.deb

so using dpkg solves the install dir issue:

-- Could NOT find Boost
-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3")
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5") 
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY) 

The problem with boost is still present, so of course it still exits with error later on.
Reading the apt-get logs I can see that it installs some packages of boost but not all the recommended ones:

Reading state information...
The following additional packages will be installed:
  libboost1.58-dev
Suggested packages:
  libboost-doc libboost1.58-doc libboost-atomic1.58-dev
  libboost-chrono1.58-dev libboost-context1.58-dev libboost-coroutine1.58-dev
  libboost-date-time1.58-dev libboost-exception1.58-dev
  libboost-filesystem1.58-dev libboost-graph1.58-dev
  libboost-graph-parallel1.58-dev libboost-iostreams1.58-dev
  libboost-locale1.58-dev libboost-log1.58-dev libboost-math1.58-dev
  libboost-mpi1.58-dev libboost-mpi-python1.58-dev
  libboost-program-options1.58-dev libboost-python1.58-dev
  libboost-random1.58-dev libboost-regex1.58-dev
  libboost-serialization1.58-dev libboost-signals1.58-dev
  libboost-system1.58-dev libboost-test1.58-dev libboost-thread1.58-dev
  libboost-timer1.58-dev libboost-wave1.58-dev libboost1.58-tools-dev
  libmpfrc++-dev libntl-dev
The following NEW packages will be installed:
  libboost-dev libboost1.58-dev pkg-config
0 upgraded, 3 newly installed, 0 to remove and 9 not upgraded.

Not sure if this can be the issue here.

[loretoparisi]

[UPDATE]

Ok this is my last finding to solve the boost issue:

# OpenNMT dependencies: boot, eigen, intel mkl
RUN apt-get update && apt-get install -y \
  libboost-dev \
  libboost-all-dev \
  pkg-config

It seems that libboost-all-dev will install all needed dependencies not only the boost library alone, so that now I came out with

-- Found Eigen3: /usr/include/eigen3 (Required is at least version "3.3") 
... 
-- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "6.5") 
-- Could NOT find MKL (missing:  MKL_INCLUDE_DIR MKL_INTEL_LP64_LIBRARY MKL_GF_LP64_LIBRARY MKL_GNU_THREAD_LIBRARY MKL_CORE_LIBRARY) 
-- Boost version: 1.58.0

I just have some complains about the architecture from nvcc when building

[ 27%] Building C object lib/TH/CMakeFiles/TH.dir/THGeneral.c.o
[ 30%] Building C object lib/TH/CMakeFiles/TH.dir/THFile.c.o
[ 33%] Building C object lib/TH/CMakeFiles/TH.dir/THDiskFile.c.o
[ 36%] Linking C shared library libTH.so
[ 36%] Built target TH
[ 39%] Building NVCC (Device) object CMakeFiles/onmt.dir/src/cuda/onmt_generated_Kernels.cu.o
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).

and the missing Intel MKL, but this is a separate thing.

[loretoparisi]

@guillaumekln thanks it make sense, by example when I have built the Dockerfile for darkflow I add to change this manually in the Cmake.

Somewhere I have used a command to detected the architecture automatically, I'm searching this right now...

[loretoparisi]

Closing this since it has been solved btw.

[loretoparisi]

@guillaumekln re-opening since I get

root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# echo "hello this is me" | ./translate --model /root/nmt/onmt_baseline_wmt15-all.en-de_epoch13_7.19_release.t7 --beam_size 5 --batch_size 1 -
Illegal instruction (core dumped)

where the built file was

root@18b64b3f8bc0:/mxmlib/CTranslate/build/cli# file translate 
translate: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=9116d7fc8728d04bb935c6781256aa9c486e6358, not stripped

My cmake options were

cmake -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&

I have added the DWITH_OPENMP and it seems to work

cmake -DWITH_OPENMP=OFF -DCMAKE_CXX_FLAGS="-march=native" -DEIGEN_ROOT="/usr/include/eigen3" .. &&

[guillaumekln]

What is your g++ --version?

[loretoparisi]

@guillaumekln

0.0.90.190 ubuntu@ai-tools-nmt-i-04fc6623dcd3531f8:~$ g++ -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.4' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 

[guillaumekln]

Not sure about this.

• Are you compiling and testing on the same machine?
• Are you compiling with a compiler other than g++?

[loretoparisi]

@guillaumekln I think the problem were the linked libraries

root@ac0117818001:/mxmlib/CTranslate/build/cli# ldd translate 
    linux-vdso.so.1 =>  (0x00007fff2bbda000)
    libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)
    libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007f7cab9b9000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7cab637000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7cab421000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cab057000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7caae3a000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7caac36000)
    librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7caaa2d000)
    libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007f7ca7a12000)
    libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
    libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7ca72e7000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f7ca70c5000)
    /lib64/ld-linux-x86-64.so.2 (0x0000557438c78000)

So I think I was missing

libTH.so => /mxmlib/CTranslate/build/lib/TH/libTH.so (0x00007f7ca75f0000)
libOpenNMTTokenizer.so => /mxmlib/CTranslate/build/lib/tokenizer/libOpenNMTTokenizer.so (0x00007f7ca77f9000)
libonmt.so => /mxmlib/CTranslate/build/libonmt.so (0x00007f7cabc49000)

or libboost compiled on a different machine.

Question: no way to get rid of boost and eigen i.e. not using them?

[guillaumekln]

Why would you like to drop these dependencies?

[loretoparisi]

@guillaumekln I wonder if it would be possibile to make the binaries standalone i.e. like fasttext does. This would be great and make everything more portable. Thank you!

[loretoparisi]

@guillaumekln thanks now I have only the boost dependency:

root@e195516272c0:/mxmlib/CTranslate/build/cli# ldd translate 
	linux-vdso.so.1 =>  (0x00007ffc8f1cc000)
	libboost_program_options.so.1.58.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.58.0 (0x00007ffa56042000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa55e24000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffa55c20000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ffa55a18000)
	libcublas.so.8.0 => /usr/local/cuda/lib64/libcublas.so.8.0 (0x00007ffa529fc000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ffa5267a000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffa52371000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffa5215a000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa51d91000)
	/lib64/ld-linux-x86-64.so.2 (0x00005637afd00000)

[guillaumekln]

boost is only used for managing command line options. It should be easy to write your own translation client without this dependency.

See cli/translate.cc.