the loss is NAN when train model

[preddy5]

the loss is NAN when train model

[preddy5]

Hey @WHQCHINA
This tends to happen when you have a high learning rate, could try training with a lower running rate and check if that resolves the issue.

[WHQCHINA]

Hey @WHQCHINA
This tends to happen when you have a high learning rate, could try training with a lower running rate and check if that resolves the issue.

@preddy5 The result is NAN use your way. find function training_step run failed

[WHQCHINA]

@preddy5 what is the true label about train data. can you give me to your train data?

[preddy5]

Hey @WHQCHINA
were you able to replicate the results with the provided emoji dataset?

[WHQCHINA]

Hey @WHQCHINA
were you able to replicate the results with the provided emoji dataset?

The train loss still is NAN use dataset in train. And print log in training_step function, that is print function not run.

[preddy5]

@WHQCHINA which version of pytorch-lightning are you using?

[WHQCHINA]

@WHQCHINA which version of pytorch-lightning are you using?

version of pytorch-lightning update to 0.9.0, it's work, thanks!