jolibrain/caffe

Caffe: a Fast framework for deep learning. Custom version with built-in sparse inputs, segmentation, object detection, class weights, and custom layers

caffe's Issues

MultiBoxLossLayerTest with CPUDevice fails in unit tests

Current device name: Graphics Device
[==========] Running 4 tests from 4 test cases.
[----------] Global test environment set-up.
[----------] 1 test from MultiBoxLossLayerTest/0, where TypeParam = caffe::CPUDevice<float>
[ RUN      ] MultiBoxLossLayerTest/0.TestConfGradient

INFO - 15:32:54 - Opened lmdb /tmp/caffe_test.3bb4-0cf4
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0414 15:32:54.470300  7141 annotated_data_layer.cpp:62] output data size: 3,3,20,20
INFO - 15:32:54 - Opened lmdb /tmp/caffe_test.3bb4-0cf4
INFO - 15:32:54 - Data layer prefetch queue emptychecking gradients

INFO - 15:32:54 - Creating layer / name=_smooth_l1_loc / type=SmoothL1Loss
INFO - 15:32:54 - Creating layer / name=_softmax_conf / type=SoftmaxWithLoss
E0414 15:32:54.972417  7141 test_gradient_check_util.hpp:206] Exhaustive Mode.
E0414 15:32:54.972452  7141 test_gradient_check_util.hpp:208] Exhaustive: blob 0 size 1
E0414 15:32:54.972458  7141 test_gradient_check_util.hpp:210] Exhaustive: blob 0 data 0
INFO - 15:32:54 - Creating layer / name=_softmax_conf / type=Softmaxunknown file: Failure
C++ exception with description "./include/caffe/llogging.h:153 / Fatal Caffe error" thrown in the test body.
[  FAILED  ] MultiBoxLossLayerTest/0.TestConfGradient, where TypeParam = caffe::CPUDevice<float> (1342 ms)
[----------] 1 test from MultiBoxLossLayerTest/0 (1342 ms total)

[----------] 1 test from MultiBoxLossLayerTest/1, where TypeParam = caffe::CPUDevice<double>
[ RUN      ] MultiBoxLossLayerTest/1.TestConfGradient

INFO - 15:32:54 - Opened lmdb /tmp/caffe_test.6365-407e
I0414 15:32:55.059880  7141 annotated_data_layer.cpp:62] output data size: 3,3,20,20
INFO - 15:32:55 - Opened lmdb /tmp/caffe_test.6365-407e
INFO - 15:32:55 - Data layer prefetch queue emptychecking gradients

INFO - 15:32:55 - Creating layer / name=_smooth_l1_loc / type=SmoothL1Loss
INFO - 15:32:55 - Creating layer / name=_softmax_conf / type=SoftmaxWithLoss
E0414 15:32:55.063814  7141 test_gradient_check_util.hpp:206] Exhaustive Mode.
E0414 15:32:55.063829  7141 test_gradient_check_util.hpp:208] Exhaustive: blob 0 size 1
E0414 15:32:55.063835  7141 test_gradient_check_util.hpp:210] Exhaustive: blob 0 data 0
INFO - 15:32:55 - Creating layer / name=_softmax_conf / type=Softmaxunknown file: Failure
C++ exception with description "./include/caffe/llogging.h:153 / Fatal Caffe error" thrown in the test body.
[  FAILED  ] MultiBoxLossLayerTest/1.TestConfGradient, where TypeParam = caffe::CPUDevice<double> (92 ms)
[----------] 1 test from MultiBoxLossLayerTest/1 (92 ms total)

GPU memory leak with cudnn

CUDNN version of some caffe operations leak a lot of memory.

• LEaked memory is allocated during cudnnCreate(&handle_)
  for instance https://github.com/jolibrain/caffe/blob/master/src/caffe/layers/cudnn_conv_layer.cpp#L53

• the corresponding https://github.com/jolibrain/caffe/blob/master/src/caffe/layers/cudnn_conv_layer.cpp#L251 , while correctly executed, does not release memory for other processes (memory still is reusable by same process)

this is detected since cuda 10.1 cudd 7.6.5 on ubuntu 18.04 , nvidia-driver 440.33.01 , but may have been present before

Build error with CUDA 8.0

src/caffe/3rdparty/ctc/reduce.cu(44): error: identifier "__shfl_down_sync" is undefined
          detected during:
            instantiation of "T CTAReduce<NT, T, Rop>::reduce(int, T, CTAReduce<NT, T, Rop>::Storage &, int, Rop) [with NT=128, T=float, Rop=ctc_helper::add<float, float>]" 
(76): here
            instantiation of "void reduce_rows<NT,Iop,Rop,T>(Iop, Rop, const T *, T *, int, int) [with NT=128, Iop=ctc_helper::negate<float, float>, Rop=ctc_helper::add<float, float>, T=float]" 
(124): here
            instantiation of "void ReduceHelper::impl(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]" 
(139): here
            instantiation of "ctcStatus_t reduce(Iof, Rof, const T *, T *, int, int, __nv_bool, cudaStream_t) [with T=float, Iof=ctc_helper::negate<float, float>, Rof=ctc_helper::add<float, float>]" 
(149): here