Issue with training FCN_Vgg16_32s

[aurora95]

Issue with training FCN_Vgg16_32s

[interactivetech]

I studied the loss during training, and from 4-9 steps, the loss increased rapidly to a large number until reaching nan. I had a feeling this was due to a bad learning rate, so I investigated what was the learning rate other repos set to train FCN_VGG16-32s. It seems that people use 1e-10.

Now the learning rate is behaving appropriately.

Here are some links of additional details where people set the learning rate to 1e-10:

Pytorch Repo: https://github.com/wkentaro/pytorch-fcn/blob/cfad9e594e2bb5a327ba838a7103abeb74e190c8/torchfcn/ext/fcn.berkeleyvision.org/siftflow-fcn32s/solver.prototxt#L10

Pytorch Repo: https://github.com/ZijunDeng/pytorch-semantic-segmentation/blob/4a1721f9a3284788336430efb140288096c6dd09/train/voc-fcn/train.py#L27

[interactivetech]

Update, 1e-10 as a learning rate was not working as well as I hoped it would. I am getting the best training results setting 'base_lr=1e-5' and weight_decay='1e-5'