bigballon/pytorch-cpp

PyTorch C++ inference with LibTorch

This demo will demonstrate how to use LibTorch to build your C++ application.

[UPDATE 2019/01/18] : Init the repo, test with PyTorch1.0.
[UPDATE 2020/02/22] : Thanks for Ageliss and his PR, which update this demo to fit LibTorch1.4.0 and OpenCV4.0.
[UPDATE 2020/04/15] : Retest this tutorial with OpenCV4.3/PyTorch1.4/LibTorch1.4, update readme for beginner.
[UPDATE 2020/04/25] : Update CMakeLists.txt to fit C++14, retest with CUDA10.0/PyTorch1.5/LibTorch1.5.
[UPDATE 2020/11/08] : Improve readme, retest with PyTorch1.7/ CUDA10.2+cuDNNv7.6.5 and CUDA11.0+cuDNNv8.0.4

Contents

• Contents
• Preparation
  • Step 0: Dependencies
  • Step 1: JIT Model
  • Step 2: Cpp Program
  • Step 3: CMakeLists
• Build
• Usage

Preparation

Step 0: Dependencies

Make sure LibTorch and OpenCV have been installed correctly.

• Install OpenCV: for Linux, for Mac OS
• Get LibTorch: download LibTorch package from the official website, then unpack it, for example:

cd path_to_your_workspace
wget https://download.pytorch.org/libtorch/cu102/libtorch-cxx11-abi-shared-with-deps-1.7.0.zip
unzip libtorch-cxx11-abi-shared-with-deps-1.7.0.zip

Step 1: JIT Model

Export torch script file, we use resnet18/resnet50 in this demo. (see model_trace.py)

Step 2: Cpp Program

Write C++ application program. (see prediction.cpp)

PS: module->to(at::kCUDA) and input_tensor.to(at::kCUDA) will switch your model & tensors to GPU mode, comment out them if you just want to use CPU.

Step 3: CMakeLists

Write a CMakeLists.txt. (check cppdocs for more details)

Build

• run model_trace.py, you will get a converted model resnet50.pt.
• compile your cpp program, you need to use -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch, for example:

mkdir build
cd build
# change "/home/bigballon/libtorch" to your libtorch path
cmake -DCMAKE_PREFIX_PATH=/home/bigballon/libtorch ..
make

PS: If you get the compile error: error: undefined reference to `cv::imread(std::string const&, int)', check issues 14684 and issues 14620 for more details.

Usage

classifier <path-to-exported-script-module> <path-to-lable-file>
# example:
# ./classifier ../resnet18.pt ../label.txt

> ./classifier ../resnet18.pt ../label
== Switch to GPU mode
== Model [../resnet18.pt] loaded!
== Label loaded! Let's try it
== Input image path: [enter Q to exit]
../pic/dog.jpg
== image size: [976 x 549] ==
== simply resize: [224 x 224] ==
    ============= Top-1 =============
    Label:  beagle
    With Probability:  97.0629%
    ============= Top-2 =============
    Label:  Walker hound, Walker foxhound
    With Probability:  1.30952%
    ============= Top-3 =============
    Label:  English foxhound
    With Probability:  0.434456%

../pic/shark.jpg
== image size: [800 x 500] ==
== simply resize: [224 x 224] ==
    ============= Top-1 =============
    Label:  tiger shark, Galeocerdo cuvieri
    With Probability:  67.672%
    ============= Top-2 =============
    Label:  hammerhead, hammerhead shark
    With Probability:  16.4908%
    ============= Top-3 =============
    Label:  great white shark, white shark, man-eater, man-eating shark
    With Probability:  15.7808%
== Input image path: [enter Q to exit]
Q

> ./classifier ../resnet50.pt ../label
== Switch to GPU mode
== Model [../resnet50.pt] loaded!
== Label loaded! Let's try it
== Input image path: [enter Q to exit]
../pic/dog.jpg
== image size: [976 x 549] ==
== simply resize: [224 x 224] ==
    ============= Top-1 =============
    Label:  beagle
    With Probability:  99.1227%
    ============= Top-2 =============
    Label:  Walker hound, Walker foxhound
    With Probability:  0.469356%
    ============= Top-3 =============
    Label:  English foxhound
    With Probability:  0.110916%
== Input image path: [enter Q to exit]
../pic/shark.jpg
== image size: [800 x 500] ==
== simply resize: [224 x 224] ==
    ============= Top-1 =============
    Label:  tiger shark, Galeocerdo cuvieri
    With Probability:  92.2599%
    ============= Top-2 =============
    Label:  great white shark, white shark, man-eater, man-eating shark
    With Probability:  5.94252%
    ============= Top-3 =============
    Label:  hammerhead, hammerhead shark
    With Probability:  1.77417%
== Input image path: [enter Q to exit]
Q

Take it easy!! 💌

pytorch-cpp's Issues

Error on run

./classifier resnet50.pt label.txt
terminate called after throwing an instance of 'c10::Error'
  what():  INVALID_ARGUMENT:main_module.submodules[7][2][5]: Cannot find field. (deserialize at /pytorch/torch/csrc/jit/import.cpp:95)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f9b5bad3fe1 in /home/shayan/libtorch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f9b5bad3dfa in /home/shayan/libtorch/lib/libc10.so)
frame #2: <unknown function> + 0x668841 (0x7f9b69760841 in /home/shayan/libtorch/lib/libtorch.so.1)
frame #3: torch::jit::load(std::istream&, c10::optional<c10::Device>) + 0xe5 (0x7f9b69760f15 in /home/shayan/libtorch/lib/libtorch.so.1)
frame #4: torch::jit::load(std::string const&, c10::optional<c10::Device>) + 0x6c (0x7f9b6976103c in /home/shayan/libtorch/lib/libtorch.so.1)
frame #5: main + 0x81 (0x55c9231d5861 in ./classifier)
frame #6: __libc_start_main + 0xe7 (0x7f9b5b150b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #7: _start + 0x2a (0x55c9231d75aa in ./classifier)

Segmentation fault

After loading my model to inference all images, a segmention fault occur, how to resolve this problem? My error is as follows:
File: ./test-mnist/p2p/P2P-xunlei-unk-udp-2965_192.168.253.9_12345_113.100.16.221_12345_17.png, Label: p2p, Probability: 99.9866%
File: ./test-mnist/p2p/P2P-xunlei-unk-udp-3034_192.168.253.9_12345_180.166.38.134_0_17.png, Label: p2p, Probability: 99.9999%

Thread 1 "classifier" received signal SIGSEGV, Segmentation fault.
0x00007fffec81723e in ?? () from /usr/local/cuda-10.2/lib64/libcudart.so.10.2
Missing separate debuginfos, use: yum debuginfo-install libgcc-8.3.1-4.5.el8.x86_64 libgomp-8.3.1-4.5.el8.x86_64 libstdc++-8.3.1-4.5.el8.x86_64 zlib-1.2.11-10.el8.x86_64
(gdb) bt
#0 0x00007fffec81723e in ?? () from /usr/local/cuda-10.2/lib64/libcudart.so.10.2
#1 0x00007fffec81c70b in ?? () from /usr/local/cuda-10.2/lib64/libcudart.so.10.2
#2 0x00007fffec8492d0 in cudaStreamDestroy () from /usr/local/cuda-10.2/lib64/libcudart.so.10.2
#3 0x00007fff68f0551d in cudnnDestroy () from /usr/local/lib/libtorch_cuda.so
#4 0x00007fff68207a05 in at::cuda::(anonymous namespace)::DeviceThreadHandlePool<cudnnContext*, &at::native::(anonymous namespace)::createCuDNNHandle, &at::native::(anonymous namespace)::destroyCuDNNHandle>::~DeviceThreadHandlePool() () from /usr/local/lib/libtorch_cuda.so
#5 0x00007fff63e66677 in __cxa_finalize () from /lib64/libc.so.6
#6 0x00007fff65a68a83 in __do_global_dtors_aux () from /usr/local/lib/libtorch_cuda.so
#7 0x00007fffffffe130 in ?? ()
#8 0x00007ffff7de4106 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC
(gdb)

I met this error when cmake.

Hi, @BIGBALLON

I met this error when cmake.

Kerb:build Apple$ cmake -DCMAKE_PREFIX_PATH=~/libtorch CMakeLists.txt ..
-- The C compiler identification is AppleClang 10.0.1.10010046
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
-- Check for working CXX compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Found torch: /Users/Apple/libtorch/lib/libtorch.dylib
CMake Error at CMakeLists.txt:6 (find_package):
By not providing "FindOpenCV.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "OpenCV", but
CMake did not find one.

Could not find a package configuration file provided by "OpenCV" with any
of the following names:

OpenCVConfig.cmake
opencv-config.cmake

Add the installation prefix of "OpenCV" to CMAKE_PREFIX_PATH or set
"OpenCV_DIR" to a directory containing one of the above files. If "OpenCV"
provides a separate development package or SDK, be sure it has been
installed.
-- Configuring incomplete, errors occurred!

What's wrong to me?

My machine: MacOSX Mojave, Libtorch, OpenCV installed.
My opencv is here:
Kerb:build Apple$ pkg-config --cflags --libs /usr/local/Cellar/opencv/4.1.0_2/lib/pkgconfig/opencv4.pc
-I/usr/local/Cellar/opencv/4.1.0_2/include/opencv4/opencv -I/usr/local/Cellar/opencv/4.1.0_2/include/opencv4 -L/usr/local/Cellar/opencv/4.1.0_2/lib -lopencv_gapi -lopencv_stitching -lopencv_aruco -lopencv_bgsegm -lopencv_bioinspired -lopencv_ccalib -lopencv_dnn_objdetect -lopencv_dpm -lopencv_face -lopencv_freetype -lopencv_fuzzy -lopencv_hfs -lopencv_img_hash -lopencv_line_descriptor -lopencv_quality -lopencv_reg -lopencv_rgbd -lopencv_saliency -lopencv_sfm -lopencv_stereo -lopencv_structured_light -lopencv_phase_unwrapping -lopencv_superres -lopencv_optflow -lopencv_surface_matching -lopencv_tracking -lopencv_datasets -lopencv_text -lopencv_dnn -lopencv_plot -lopencv_videostab -lopencv_video -lopencv_xfeatures2d -lopencv_shape -lopencv_ml -lopencv_ximgproc -lopencv_xobjdetect -lopencv_objdetect -lopencv_calib3d -lopencv_features2d -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_flann -lopencv_xphoto -lopencv_photo -lopencv_imgproc -lopencv_core

Thanks in advance .
Best,
from @bemoregt.

Why CPP is slower than Python?

I tested the pretrained ResNet34 with original model and jit traced model, with 400 images, the image preprocess of Python and C++ is the same, which is just like the way you did.

On my Mac.
Python costs 34 seconds, CPU usage >90%.
C++ costs 73 seconds, CPU usage < 40%.
why? why? why?

Python(pytorch)和C++(libtorch)相同模型（同网络结构同权重trace转化而来）相同样本推理结果是否不同

Linker problem: undefined reference to `cv::imread(std::string const&, int)'

Thanks for the great sample.
I can't seem to this project to link if I include both OpenCV and Torch.
When both libraries are included in the cmake file, I get the error:
PyTorch-CPP/prediction.cpp:22: undefined reference to `cv::imread(std::string const&, int)'
collect2: error: ld returned 1 exit status

If I remove all references to Torch, the project builds, links, and runs.

Any suggestions would be appreciated.
I asked this question on StackOverflow, but I haven't gotten any responses.

Here the CMakeError.txt file:
erforming C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
Change Dir: /home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_3c4f7/fast && /usr/bin/make -f CMakeFiles/cmTC_3c4f7.dir/build.make CMakeFiles/cmTC_3c4f7.dir/build
make[1]: Entering directory '/home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_3c4f7.dir/src.c.o
/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD -o CMakeFiles/cmTC_3c4f7.dir/src.c.o -c /home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp/src.c
Linking C executable cmTC_3c4f7
/home/tim/.local/share/JetBrains/Toolbox/apps/CLion/ch-0/201.7223.86/bin/cmake/linux/bin/cmake -E cmake_link_script CMakeFiles/cmTC_3c4f7.dir/link.txt --verbose=1
/usr/bin/cc -DCMAKE_HAVE_LIBC_PTHREAD -rdynamic CMakeFiles/cmTC_3c4f7.dir/src.c.o -o cmTC_3c4f7
/usr/bin/ld: CMakeFiles/cmTC_3c4f7.dir/src.c.o: in function main': src.c:(.text+0x46): undefined reference to pthread_create'
/usr/bin/ld: src.c:(.text+0x52): undefined reference to pthread_detach' /usr/bin/ld: src.c:(.text+0x63): undefined reference to pthread_join'
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_3c4f7.dir/build.make:87: cmTC_3c4f7] Error 1
make[1]: Leaving directory '/home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_3c4f7/fast] Error 2

Source file was:
#include <pthread.h>

void* test_func(void* data)
{
return data;
}

int main(void)
{
pthread_t thread;
pthread_create(&thread, NULL, test_func, NULL);
pthread_detach(thread);
pthread_join(thread, NULL);
pthread_atfork(NULL, NULL, NULL);
pthread_exit(NULL);

return 0;
}

Determining if the function pthread_create exists in the pthreads failed with the following output:
Change Dir: /home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/make cmTC_6b9bf/fast && /usr/bin/make -f CMakeFiles/cmTC_6b9bf.dir/build.make CMakeFiles/cmTC_6b9bf.dir/build
make[1]: Entering directory '/home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp'
Building C object CMakeFiles/cmTC_6b9bf.dir/CheckFunctionExists.c.o
/usr/bin/cc -DCHECK_FUNCTION_EXISTS=pthread_create -o CMakeFiles/cmTC_6b9bf.dir/CheckFunctionExists.c.o -c /home/tim/.local/share/JetBrains/Toolbox/apps/CLion/ch-0/201.7223.86/bin/cmake/linux/share/cmake-3.16/Modules/CheckFunctionExists.c
Linking C executable cmTC_6b9bf
/home/tim/.local/share/JetBrains/Toolbox/apps/CLion/ch-0/201.7223.86/bin/cmake/linux/bin/cmake -E cmake_link_script CMakeFiles/cmTC_6b9bf.dir/link.txt --verbose=1
/usr/bin/cc -DCHECK_FUNCTION_EXISTS=pthread_create -rdynamic CMakeFiles/cmTC_6b9bf.dir/CheckFunctionExists.c.o -o cmTC_6b9bf -lpthreads
/usr/bin/ld: cannot find -lpthreads
collect2: error: ld returned 1 exit status
make[1]: *** [CMakeFiles/cmTC_6b9bf.dir/build.make:87: cmTC_6b9bf] Error 1
make[1]: Leaving directory '/home/tim/samples/PyTorch-CPP/cmake-build-debug/CMakeFiles/CMakeTmp'
make: *** [Makefile:121: cmTC_6b9bf/fast] Error 2

I looked for the cause of the error "/usr/bin/ld: cannot find -lpthreads".
I don't think that is relevant. The linker command is looking for 'pthread', not 'pthreads:
$ cat link.txt
/usr/bin/c++ -g -rdynamic CMakeFiles/classifier.dir/prediction.cpp.o -o classifier -Wl,-rpath,/usr/local/lib:/home/tim/tools/libtorch/lib /usr/local/lib/libopencv_dnn.so.4.3.0 /usr/local/lib/libopencv_gapi.so.4.3.0 /usr/local/lib/libopencv_highgui.so.4.3.0 /usr/local/lib/libopencv_ml.so.4.3.0 /usr/local/lib/libopencv_objdetect.so.4.3.0 /usr/local/lib/libopencv_photo.so.4.3.0 /usr/local/lib/libopencv_stitching.so.4.3.0 /usr/local/lib/libopencv_video.so.4.3.0 /usr/local/lib/libopencv_videoio.so.4.3.0 /home/tim/tools/libtorch/lib/libtorch.so /home/tim/tools/libtorch/lib/libc10.so /usr/local/lib/libopencv_imgcodecs.so.4.3.0 /usr/local/lib/libopencv_calib3d.so.4.3.0 /usr/local/lib/libopencv_features2d.so.4.3.0 /usr/local/lib/libopencv_flann.so.4.3.0 /usr/local/lib/libopencv_imgproc.so.4.3.0 /usr/local/lib/libopencv_core.so.4.3.0 -Wl,--no-as-needed,/home/tim/tools/libtorch/lib/libtorch_cpu.so -Wl,--as-needed /home/tim/tools/libtorch/lib/libc10.so -lpthread -Wl,--no-as-needed,/home/tim/tools/libtorch/lib/libtorch.so -Wl,--as-needed

libpthread is in "/usr/lib/x86_64-linux-gnu":
$ ls /usr/lib/x86_64-linux-gnu/libpth*.*
/usr/lib/x86_64-linux-gnu/libpthread-2.31.so /usr/lib/x86_64-linux-gnu/libpthread.so
/usr/lib/x86_64-linux-gnu/libpthread.a /usr/lib/x86_64-linux-gnu/libpthread.so.0

And that directory is found by "ld":
ld --verbose | grep SEARCH_DIR
SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64"); SEARCH_DIR("=/usr/local/lib64"); SEARCH_DIR("=/lib64"); SEARCH_DIR("=/usr/lib64"); SEARCH_DIR("=/usr/local/lib"); SEARCH_DIR("=/lib"); SEARCH_DIR("=/usr/lib"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/x86_64-linux-gnu/lib");

how to do it on win10

create a vs2015proj on win10 using libtorch1.5 always meet the error "You need C++14 to compile PyTorch testtorch c:\Users\44457\Desktop\libtorch\include\c10\util\C++17.h 24 "
"
and i can't just change the cmakelist.txt from 11 to 14,it doesn't work

Question: Have you run multiple forward() calls on the same module using CUDA?

I have a similar libtorch setup (trace and save torchscript CUDA-flavor PT file in python, use torch::jit::load to load file in cpp), but I have noticed on forward() calls after the first forward() call, an exception is thrown. I do not see this in the CPU (non-CUDA) variant. Have you tried looping your forward calls to see if they work repeatedly?
Thanks,
Rob