train occur NAN

[thinklab-sjtu]

train occur NAN

[yangxue0827]

#4

[adamzhu1990]

now, I try it on the DOTA1.0 all the same😂

[yangxue0827]

Do you use the configuration file I provided？ @adamzhu1990

[adamzhu1990]

yes, I use the cfgs_res50_dota_r3det_v2.py as default configuration file

[zsh1234567]

First of all, thanks to the author for the code.
I have used the parameters given by the author, the results I got are as follows.
However,"I find the refine_reg_loss_stage3:0.00",Its value is 0, is this normal?

2020-04-19 20:51:29: global_step:2560 current_step:2560
per_cost_time:1.864s
cls_loss:0.979
reg_loss:1.603
refine_cls_loss:1.234
refine_reg_loss:1.127
refine_cls_loss_stage3:0.001
refine_reg_loss_stage3:0.000
total_losses:4.946

[yangxue0827]

The reason for this (NAN after 'bbox_transform.py:98: RuntimeWarning: divide by zero encountered in log') is that the model predicts a very small box (w and h are close to 0), which leads to a problem with the encoding and decoding of the bounding box. I will upload the repaired code and it is still being debugged. @adamzhu1990 @zsh1234567