Encounter a strange issue

[tonghanwang]

Encounter a strange issue

[kkkclearlove]

经过观察pymarl及其旧版代码，我找到了问题所在：
首先先介绍我的环境，rtx3090-torch1.7.1-cuda11。需要修改原代码中的两处，他们分别位于：
src/utils/logging.py:
line 48:item = "{:.4f}".format(np.mean([x[1] for x in self.stats[k][-window:]]))
修改为：import torch as th
item = "{:.4f}".format(th.mean(th.tensor([x[1] for x in self.stats[k][-window:]])))
src/learner/q_learner:
line 71: if self.args.double_q:
# Get actions that maximise live Q (for double q-learning)
mac_out[avail_actions == 0] = -9999999
cur_max_actions = mac_out[:, 1:].max(dim=3, keepdim=True)[1]
target_max_qvals = th.gather(target_mac_out, 3, cur_max_actions).squeeze(3)
改为：if self.args.double_q:
# Get actions that maximise live Q (for double q-learning)
mac_out_detach = mac_out.clone().detach()
mac_out_detach[avail_actions == 0] = -9999999
cur_max_actions = mac_out_detach[:, 1:].max(dim=3, keepdim=True)[1]
target_max_qvals = th.gather(target_mac_out, 3, cur_max_actions).squeeze(3)
如上修改后本仓库即可在我的上述环境中运行。

[youpengzhao]

I guess it may be due to the version of pytorch. I use the torch whose version is 1.5, different from 0.4. However, I run the pymarl code successfully under pytorch 1.5 ago. So I don't know where the problem is. In fact, I delete the code of computing communication loss, which is different from original pymarl, but the error still exists. I think it's very strange and hope for your answer. Thank you very much!

[TonghanWang]

This issue is indeed quite strange. From the shape of the tensor ([32,16,2,3]), I guess there is something wrong with the messages. Since the error is caused by some in-place operations, I guess it is because of the _communication function.

But I guess the most convenient way to deal with this problem is to use torch 0.4.1.post2, given that NDQ is based on a earlier version of PyMARL. :)

[kkkclearlove]

我也遇到了类似问题，请问老铁解决了吗？

[youpengzhao]

Sorry for not updating this issue. I solved this issue but forgot to update the result on github. Thanks for kkkclearlove. The code of NDQ indeed has some problems under the pytorch 1.x. The gradient of some tensor may be changed without cloning, so the issue happens. Maybe using pytorch 0.4 can avoid this issue or you can make adjustment just as kkkclearlove said.