BrokenPipeError during training process

[pxiangwu]

BrokenPipeError during training process

[pxiangwu]

Hi, I never encountered this issue before. Which version of Pytorch are you using?

I tested my code on Pytorch1.0 and 1.1. Not sure if the latest Pytorch version (e.g., 1.6) would work well.

[fyaft2012]

Hi, I use the pytorch 1.1.0, not the latest one. And it runs on Ubuntu 18.04 (I previously trained on 16.04, same problem happened), with RTX 2080, 32GB RAM.

[pxiangwu]

@fyaft2012 , could you debug the code by setting the num_worker = 0? This would be helpful to check what happens.

[fyaft2012]

Hi Wu, okay I am just debugging as what you suggest. Let's see what will happen, while the speed is much reduced after setting num_worker = 0. I will let you know whether it helps or not. Thanks!

[pxiangwu]

Sure. Let me know what happens since I am quite curious about this. From the error you showed above, it could possibly related to the multithread data loading.

Also, from the error reported, one potential work-around could be: set the size of self.cache to be 0. In this way, we will not cache the pre-loaded data in the memory (although the speed could be slowed down a little bit).

[fyaft2012]

Okay, I will try both of them. And let you know the results.

[pxiangwu]

Also, one thing I forgot to mention. You may also need to observe the CPU memory to see if it is used up by the cache. Not sure 32GB RAM is OK, since I run the code on the cluster with 1TB memory ...

If the memory is not enough, you may reduce the cache size.

[fyaft2012]

Okay, I see. Thank you!