imagenet example training gets slower over time.

[pytorch]

imagenet example training gets slower over time.

[apaszke]

If that's all of your memory then the system might be swapping and the performance loss is expected. To fix it try with fewer workers or buy more RAM 😉

[zym1010]

@apaszke thanks. but seems that the code keeps assuming more and and more memory... is that expected? I mean on ImageNet dataset. When using Caffe with LMDB, it consumes all the memory, but training isn't slowed down.

[apaszke]

No. Are you using the script from the official examples repo or is it your own?

[zym1010]

the official one.

[zym1010]

I'm now trying to reboot the computers, and see if it helps.

[zym1010]

@apaszke BTW, can you have a look at https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/4? I think when I'm using Caffe, memory is not that a big issue.

[zym1010]

@apaszke I retried, and seems by checking free -m, the reason of slowing down is that too much data got cached in mem (the python programs don't take much memory), and maybe the it takes time for the OS to make room for newly read images. I'm using a CentOS 6 cluster system. Maybe the caching mechanism is poor there?

[zym1010]

@apaszke I tested the same script on another machine with Ubuntu 14.04. This time, the training speed is rock solid. So I think problem is with CentOS 6.

[VictorJiangXin]

Sorry to bother you after so long time, but I have met the same problem with you. Have you solve your problem?

[zym1010]

@VictorJiangXin not sure. At least for newer OSes (not CentOS 6), this should not be a problem.