Got Nan in loss calculation

[begeekmyfriend]

Got Nan in loss calculation

[begeekmyfriend]

By the way, a256 is all right and a128 is always bad

[BohanZhai]

Would you mind tell me the batch size in your model?

[begeekmyfriend]

By the way the evaluation under configuration of a256 and c256 is all right under my checkpoint with 1577 epochs training. I am using my own mandarin data corpus. Therefore I think it is the configuration leads to the training failure.
eval_a256_c256.zip

[BohanZhai]

In my opinion, the large batch size may cause NaN. Also, because the dataset is changed, so the learning rate may change too.

[begeekmyfriend]

Maybe you are right I will try this. Thanks for your reply and your brillliant jobs. There seems better cost performance than WaveGlow.

[gongchenghhu]

@begeekmyfriend I have also got negative loss value. Have you solved it? Could you tell me how to solve this problem.

[begeekmyfriend]

The loss is essentially negative since it is composed of log liklihoods.

[gongchenghhu]

@begeekmyfriend Thank you very much! I trained on Chinese biaobei dataset on a single GPU. I don't know how many epochs it takes to get a good result. Now I have trained 230 epoch about 23689 step. The result on 22000step is not very good. Do you have any idea?
23000step-a256c256-result.zip

[begeekmyfriend]

In my hunble opinion the evaluation is not as good as that of WaveRNN. The SqueezeWave is light weight model for fast inference adapting for edge devices. It really achieve real time synthesis as a vocoder but it costs you the loss of a bit fidelity. You can wait until 1~2K epochs for better evaluation.

[gongchenghhu]

@begeekmyfriend Thanks for your quickly reply，and I will wait until 1~2K epochs.

[wizardk]

@begeekmyfriend Hi, did you compare the speed of SqueezeWave and WaveRNN without batched on single CPU?

[begeekmyfriend]

SqueezeWave is much more fast but with less fidelity because of less training sampling points.

[wizardk]

The reason for the NAN should refer to these : https://github.com/NVIDIA/waveglow/issues/95 and https://github.com/NVIDIA/waveglow/issues/123