Getting nan results when training

[William4553]

Getting nan results when training

[TFboys-lzz]

Thank you for your interest in our work！
I check the code released and make sure that the operations {division, log, and sqrt} have added an eps.
But, You can try to clamp the input of the POW operation to a range greater than 0, i.e., sine = torch.sqrt((1.0 - torch.pow(cosine.(0.0,1.0), 2)).clamp(0.0001, 1.0)) in 152 lines of utils.py, because torch.pow(a) = nan when a < 0.
If the above operation cannot address your problem, you can try to use the torch.nn.utils.clip_grad_norm_ operation to clip gradient norm of an iterable of parameters.
If the above suggestions solve your problem, I hope you can let me know so that I can update the code, thanks!!

[TFboys-lzz]

Alternatively, you can try clamping the input of torch.sqrt() to a larger scale, e.g., sine = torch.sqrt((1.0 - torch.pow(cosine.(0.0,1.0), 2)).clamp(0.1, 1.0)).
Because, the grad of torch.sqrt 1/torch.sqrt(x), if x = 0.0001, the grad = 10000

[William4553]

Thanks for the response and for the advice. I found that line 35 of loss.py is causing my problem. When I removed the weight parameter from loss = F.cross_entropy(input=pred,target=label,weight=weight), I no longer got the issue with nan. But when the weight parameter is there, loss has the value nan.

[TFboys-lzz]

Thanks for your reply!
I will check the code again.

[TFboys-lzz]

Thanks to Feng Wei from Zhejiang University. We found that the instability of the training may be due to the exceptionally small (close to zero) weights of the background classes.