BrokenPipeError: [Errno 32] Broken pipe

[dena]

BrokenPipeError: [Errno 32] Broken pipe

[R-Ceph]

Hi @han-x !

Could you give me an information like below to consider from various perspectives?

• server machine info (Ubuntu? OSX? Windows?)
• machine specs (#CPUs, #GPUs, RAM)
• pytorch version

Thanks!

• server machine info: Ubuntu 16.04.7 LTS
• CPU：Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
• RAM：64GB
• GPU：2080 * 2
• pytorch：Version: 1.7.0

[R-Ceph]

It seems that when running "python main.py -- worker" and "python main.py --train-server" on the same computer, the "worker" process will take up a lot of memory so that this kind of error occurs.

In my last running time, this problem came out after 129400 episodes, and the "worker" took up about 50GB mem, and the "train-server" took up about 10GB mem.

128900 129000 129100
epoch 396
win rate = 0.999 (124.8 / 125)
generation stats = 0.000 +- 0.742
loss = p:-0.000 v:0.012 ent:0.324 total:0.012
updated model(54158)
129200 129300 129400
epoch 397
win rate = 0.991 (125.8 / 127)
generation stats = -0.000 +- 0.742
loss = p:-0.000 v:0.012 ent:0.329 total:0.012
updated model(54282)
Exception in thread Thread-7:
Traceback (most recent call last):
File "/home/alex2/workspace/miniconda3/envs/torch/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/alex2/workspace/miniconda3/envs/torch/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/alex2/hx_workspare/HandyRL/handyrl/connection.py", line 190, in _receiver
data, cnt = conn.recv()
File "/home/alex2/workspace/miniconda3/envs/torch/lib/python3.7/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/home/alex2/workspace/miniconda3/envs/torch/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/home/alex2/workspace/miniconda3/envs/torch/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError

[YuriCat]

@han-x

Massively thank you for your report.
You are training a big neural network, aren't you?
In current implementation, each Gather process store all models after starting workers.
Rewrite worker.py for storing only latest model is currently the best way to reduce memories.

We are considering enabling to select training scheme and avoid storing unused models.

[R-Ceph]

Thanks for your reply! But I just used the original GeeseNet...
I will try the method you said, thanks again LoL

[YuriCat]

@han-x

Thanks. That's a strange case...
Since the original GeeseNet was about 500KB, it will occupy only 100MB when in 200 epochs.

It's nothing wrong 129k episodes occupied 10GB in trainer process.
You can increase compress_steps if you want to save memory, while it will slow down the speed of making batches and sometimes slow down the training speed as a result.

[ikki407]

Hi @han-x !

Could you give me an information like below to consider from various perspectives?

• server machine info (Ubuntu? OSX? Windows?)
• machine specs (#CPUs, #GPUs, RAM)
• pytorch version

Thanks!

[ikki407]

I have noticed a possible cause from your stacktraces. Are you using the codes of current master branch? I think there are some differences between your script and script in master branch.

The similar error happened before and we solved it in #145.

Could you check it? And update your code if old code is used. Thanks.