loss explode question

[ivanvovk]

loss explode question

[junkeon]

Thank you for your advice, i will try and report it!

[ivanvovk]

Thank you for your feedback. And what was your batch size?

[ivanvovk]

No-no, thank you very much, its enough information. I'm just trying to figure out why I had no loss explode problems with default setting on my server. However, I am glad that you managed to finish training, finally. Since the issue is solved, I'm closing it.

[ivanvovk]

Hello, sure. Try decrease LR scheduler gamma a bit in the first place. From 0.95 to 0.9 with default initial LR (1e-3). I used 0.95 and it was kinda borderline between normal training and training with loss explosion. And please report if it helps or not.

[ivanvovk]

Any updates, @junkeon ?

[junkeon]

As your advice, i decreased gamma varies from 0.9 to 0.5 and there are all loss explode
But it seems okay with lr = 5e-4 and gamma = 0.9 on LJ speech data
In that setting, i got l1 loss slight above 0.1 and the synthesized quality is fine

[junkeon]

sorry for late the previous feedback...
I can't remember the exact the number, it might be 48, the default setting
I can give you the number tomorrow (i am not allowed to access the server from outside), can't i?