loss: nan - val_loss nan

[kr-colab]

loss: nan - val_loss nan

[jradrion]

Hi @stsmall,

While it isn't terribly common, we have seen this behavior before. We're not entirely sure what is causing it or how to fix it. That being said, the model weights are updated as training progresses, so training terminating with nan will still allow for valid predictions, provided that previous epochs completed without loss = nan. In the example you provided, the weights used for prediction will be taken from the training epoch with the smallest lost prior to epoch 309.

One thing I will add to my to-do list is terminating training immediately upon a nan, to avoid additional epochs (i.e. wasted time). I'll see if I can implement this soon.

Best,
Jeff

[stsmall]

Hi Jeff,
Thanks! It usually ended up terminating after 10-30 more nan.
thanks,
@stsmall

[jradrion]

I added code to terminate training upon loss = Nan.