ROI pooling Speed deteriorate after tens of thousands iterations

[longcw]

ROI pooling Speed deteriorate after tens of thousands iterations

[acgtyrant]

两位大佬之间的对话！

The dialogue between the two bosses!

[drcege]

@longcw Shouldn‘t the data order used in PyTorch be NCHW ?

[longcw]

Will this problem appear when running on only one GPU?
Since I only have a GPU that I cannot reproduce your problem.
Maybe you can offer me some debug information and I am very willing to help you.

[yikang-li]

I don't know any more information. The only problem is when I time all the operations, I find that the ROI Pooling time consuming grows a lot. Currently, when I change the -arch parameters from sm_35 to sm_52, it seems OK so far. Do you know what is the parameter for?

Best,

[longcw]

The -arch compiler option specifies the compute capability that is assumed when compiling C to PTX code.

Read more at: http://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#ixzz4aDuTT670

You can find the compute capability of your GPU in https://developer.nvidia.com/cuda-gpus

[yikang-li]

Actually, it has nothing to do with --arch parameter. The per-iter-time is still doubled after about 50000 iterations.

[yikang-li]

Did you implement ROI Pooling layer by yourself?

[longcw]

No. The kernel code or ROI Pooling is copied from CharlesShang and smallcorgi.
I have no idea about your problem because the code itself is really easy to understand. I only changed the index expression since pytorch uses an order of [batch, c, w, h]:

    // int bottom_index = (h * width + w) * channels + c;
    int bottom_index = (c * height + h) * width + w;

You can see the code here: https://github.com/longcw/faster_rcnn_pytorch/blob/master/faster_rcnn/roi_pooling/src/cuda/roi_pooling_kernel.cu

[yikang-li]

I saw a commit named 'fix the memory leak for ROI pool module'. If you don't mind, could you give me some detailed information about that?

Best,
Yikang

[longcw]

I misunderstood torch.autograd.Function and used it as a Module when I first time implemented the ROI Pooling layer. The memory used by Function will not release after each iteration if I use the Function as a class member variable.

# faster_rcnn/roi_pooling/modules/roi_pool.py
# self.roi_pool = RoIPoolFunction(...) # wrong
# return self.roi_pool(features, rois)
return RoIPoolFunction(...)(features, rois) # right

[longcw]

I trained Faster RCNN for 100k iterations on a GTX 1080 without speed decreasing.
Maybe your problem is a bug of PyTorch in multi-gpu. Did you try to stop and restore it after 50k iterations?

[yikang-li]

Yes, currently, my strategy is to snapshot the model every 10k iters. When the speed deteriorates, I restart training from the snapshot.

So you are studying in Tsinghua? I was there in EE Dept.

[longcw]

Yes, I am in CS Dept.

[yikang-li]

Thank you very much. Sorry for the late reply due to the ICCV deadline.

Are you going to submit any paper to ICCV?

[longcw]

No. I look forward to reading your paper in ICCV 😀.