mathrho/action_recog

fork from https://github.com/yjxiong/caffe

action_recog's Issues

Out of memory!

Hi,

I tried to run your example/action_recognition/train_action_recognition_rgb.sh but it gives the following error (even with 100gb of memory):
...
I0420 17:29:54.015863 46833 net.cpp:299] This network produces output loss
I0420 17:29:54.015892 46833 net.cpp:548] Collecting Learning Rate and Weight Decay.
I0420 17:29:54.015907 46833 net.cpp:311] Network initialization done.
Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-172954.46833!I0420 17:29:54.015909 46833 net.cpp:312] Memory required for data: 576518148
I0420 17:29:54.016170 46833 solver.cpp:47] Solver scaffolding done.
I0420 17:29:54.016252 46833 solver.cpp:363] Solving CUHK_Action_SpatialNet
I0420 17:29:54.016257 46833 solver.cpp:364] Learning Rate Policy: step
I0420 17:29:54.019449 46833 solver.cpp:428] Iteration 0, Testing net (#0)
I0420 17:29:54.317395 46836 cudnn_conv_layer.cpp:179] Optimized cudnn conv
I0420 17:29:54.466910 46833 cudnn_conv_layer.cpp:179] Optimized cudnn conv
I0420 17:29:54.518374 46834 cudnn_conv_layer.cpp:179] Optimized cudnn conv
I0420 17:30:23.613687 46835 solver.cpp:485] Test net output #0: accuracy = 0.00352941
I0420 17:30:23.613732 46835 solver.cpp:485] Test net output #1: loss = 4.61512 (* 1 = 4.61512 loss)
I0420 17:30:23.814564 46833 solver.cpp:485] Test net output #0: accuracy = 0.00352941
I0420 17:30:23.814605 46833 solver.cpp:485] Test net output #1: loss = 4.61512 (* 1 = 4.61512 loss)
I0420 17:30:23.989828 46836 solver.cpp:485] Test net output #0: accuracy = 0.00352941
I0420 17:30:23.989862 46836 solver.cpp:485] Test net output #1: loss = 4.61512 (* 1 = 4.61512 loss)
Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46836!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46836!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46836!F0420 17:30:24.498685 46836 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f0e6bb64b7d google::LogMessage::Fail()
@ 0x7f0e6bb66c7f google::LogMessage::SendToLog()
@ 0x7f0e6bb6476c google::LogMessage::Flush()
@ 0x7f0e6bb6751d google::LogMessageFatal::~LogMessageFatal()
@ 0x7f0e6c0058eb caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f0e6bffc013 caffe::Blob<>::mutable_gpu_diff()
@ 0x7f0e6c0370f4 caffe::CuDNNConvolutionLayer<>::Backward_gpu()
@ 0x7f0e6c01cffc caffe::Net<>::BackwardFromTo()
@ 0x7f0e6c01d241 caffe::Net<>::Backward()
@ 0x7f0e6bee7e9d caffe::Solver<>::Step()
@ 0x7f0e6bee87bf caffe::Solver<>::Solve()
@ 0x40790e train()
@ 0x405d16 main
@ 0x7f0e6b07aec5 (unknown)
@ 0x4062cd (unknown)
I0420 17:30:24.525815 46834 solver.cpp:485] Test net output #0: accuracy = 0.00352941
I0420 17:30:24.525871 46834 solver.cpp:485] Test net output #1: loss = 4.61512 (* 1 = 4.61512 loss)
Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46834!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46834!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20160420-173024.46834!F0420 17:30:24.535877 46834 syncedmem.cpp:51] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f59a63cdb7d google::LogMessage::Fail()
@ 0x7f59a63cfc7f google::LogMessage::SendToLog()
@ 0x7f59a63cd76c google::LogMessage::Flush()
@ 0x7f59a63d051d google::LogMessageFatal::~LogMessageFatal()
@ 0x7f59a686e8eb caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f59a6864f62 caffe::Blob<>::mutable_gpu_data()
@ 0x7f59a689f440 caffe::CuDNNConvolutionLayer<>::Forward_gpu()
@ 0x7f59a6887349 caffe::Net<>::ForwardFromTo()
@ 0x7f59a6887797 caffe::Net<>::ForwardPrefilled()
@ 0x7f59a6750e95 caffe::Solver<>::Step()
@ 0x7f59a67517bf caffe::Solver<>::Solve()
@ 0x40790e train()
@ 0x405d16 main
@ 0x7f59a58e3ec5 (unknown)
@ 0x4062cd (unknown)
*** Aborted at 1461166225 (unix time) try "date -d @1461166225" if you are using GNU date ***
PC: @ 0x7ffdf10e2ae3 (unknown)
*** SIGTERM (@0x32670000b6ef) received by PID 46835 (TID 0x7fac5a220a40) from PID 46831; stack trace: ***
@ 0x7fac58b8fd40 (unknown)
@ 0x7ffdf10e2ae3 (unknown)
@ 0x7fac58c6192d (unknown)
@ 0x7fac38d8f1de (unknown)
@ 0x7fac387447ab (unknown)
@ 0x7fac38721e33 (unknown)
@ 0x7fac3871a050 (unknown)
@ 0x7fac3871a943 (unknown)
@ 0x7fac386883e2 (unknown)
@ 0x7fac3868853a (unknown)
@ 0x7fac3866c005 (unknown)
@ 0x7fac58928e92 (unknown)
@ 0x7fac5890d306 (unknown)
@ 0x7fac5892f328 (unknown)
@ 0x7fac59b2cfce caffe::caffe_gpu_memcpy()
@ 0x7fac59b05a4e caffe::SyncedMemory::gpu_data()
@ 0x7fac59afbe02 caffe::Blob<>::gpu_data()
@ 0x7fac59b56e3b caffe::InnerProductLayer<>::Forward_gpu()
@ 0x7fac59b1e349 caffe::Net<>::ForwardFromTo()
@ 0x7fac59b1e797 caffe::Net<>::ForwardPrefilled()
@ 0x7fac599e7e95 caffe::Solver<>::Step()
@ 0x7fac599e87bf caffe::Solver<>::Solve()
@ 0x40790e train()
@ 0x405d16 main
@ 0x7fac58b7aec5 (unknown)
@ 0x4062cd (unknown)
*** Aborted at 1461166225 (unix time) try "date -d @1461166225" if you are using GNU date ***
PC: @ 0x7fff08bfdae3 (unknown)
*** SIGTERM (@0x32670000b6ef) received by PID 46833 (TID 0x7f3c0a48ca40) from PID 46831; stack trace: ***
@ 0x7f3c08dfbd40 (unknown)
@ 0x7fff08bfdae3 (unknown)
@ 0x7f3c08ecd92d (unknown)
@ 0x7f3be8ffb1de (unknown)
@ 0x7f3be89b07ab (unknown)
@ 0x7f3be898de33 (unknown)
@ 0x7f3be8986050 (unknown)
@ 0x7f3be8986943 (unknown)
@ 0x7f3be88f43e2 (unknown)
@ 0x7f3be88f453a (unknown)
@ 0x7f3be88d8005 (unknown)
@ 0x7f3c08b94e92 (unknown)
@ 0x7f3c08b79306 (unknown)
@ 0x7f3c08b9b328 (unknown)
@ 0x7f3c09d98fce caffe::caffe_gpu_memcpy()
@ 0x7f3c09d71a4e caffe::SyncedMemory::gpu_data()
@ 0x7f3c09d67e02 caffe::Blob<>::gpu_data()
@ 0x7f3c09dc2e3b caffe::InnerProductLayer<>::Forward_gpu()
@ 0x7f3c09d8a349 caffe::Net<>::ForwardFromTo()
@ 0x7f3c09d8a797 caffe::Net<>::ForwardPrefilled()
@ 0x7f3c09c53e95 caffe::Solver<>::Step()
@ 0x7f3c09c547bf caffe::Solver<>::Solve()
@ 0x40790e train()
@ 0x405d16 main
@ 0x7f3c08de6ec5 (unknown)
@ 0x4062cd (unknown)

mpirun noticed that process rank 3 with PID 46836 on node summi exited on signal 6 (Aborted).

How can I solve this problem? needs more than 100gb memory?

Best,