I can train smoothly，but when I test, there is a errror:

[s9xie]

I can train smoothly，but when I test, there is a errror:

[tyl569]

Hello,
I can train smoothly，but when I test, there is a errror:
I1225 18:53:09.767861 16202 net.cpp:299] Memory required for data: 1097870692
F1225 18:53:10.263083 16202 syncedmem.cpp:19] Check failed: error == cudaSuccess (29 vs. 0) driver shutting down
*** Check failure stack trace: ***
Aborted (core dumped)

Thank you

[tyl569]

Is this because GPU is not enough memory？

[rribani]

I'm facing a very similar problem, but in my case it happens only with the latest caffe version:

syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

With the old caffe it works. I'm using cuda 9.0.

[wangjiaod]

I meet the same problem,please tell the answaer

[aditipanda]

I am new to Caffe, and am also facing this problem.

I read in another git thread (alexgkendall/caffe-segnet#57 (comment)) that calling caffe.reset_all() might help, but have no idea where to place this line of code.

Secondly, this git thread also speaks about MATLAB interface of Caffe. Now in the installation instruction file of HED, it is written "Python and/or MATLAB(optional)". So that means MATLAB is optional. Accordingly, I also trained the network on my GPU which does not have MATLAB. So does this mean by any chance that this training isn't optimal, as the MATLAB interface could not be used?

Also, while testing, since we are using MATLAB to create a network by the command "caffe.Net(model_name, weights)", it means MATLAB interface is compulsory for testing.

But in that case it should also be compulsory for training, as we are using other commands like "caffe.set_gpu(0)". They are also for MATLAB interface. Correct me if I am wrong.

I am thinking I have to re-train the network after installing MATLAB on my Ubuntu workstation.

Was any of you able to solve it? @wangjiaod @rribani @tyl569