Getting cls_logits NaN of Inf during training

[AceMcAwesome77]

Getting cls_logits NaN of Inf during training

[KumoLiu]

Hi @AceMcAwesome77,

The error message you're encountering, "cls_logits is NaN or Inf.", is letting you know that at some point in your training, the cls_logits tensor contains a Not a Number (NaN) or Infinity (Inf).
This can occur due to various reasons. This could come from a learning rate that's too high, instabilities in your numerical operations, uninitialized variables, or it could also be a problem with the specific data you're inputting into the model. It's a sign that the model is diverging, and gradients are getting out of control, which can also stem from exploding or vanishing gradients.
You indeed can attempt to mitigate this issue using gradient clipping, which can help ensure gradients never exceed a certain threshold. However, applying gradient clipping doesn't guarantee to resolve the root cause of the problem.
I would recommend looking at your training process more holistically. Inspect the learning rate, look for possible issues in the data, try normalizing the inputs, or use different weight initialization techniques.

Hope it helps, thanks.