loss nan and cost nan while running my own corpus using librispeech sets

[rwth-i6]

loss nan and cost nan while running my own corpus using librispeech sets

[yanghongjiazheng]

It happens on pretrain epoch 38 step 3379

[Spotlight0xff]

Hi,
what config are you using? this one?
Also what TF/CUDA/CUDNN versions do you have?

[albertz]

What TF version do you use? Can you try with TF 2.3? (Maybe related)

Note that the learning rate warmup is only for the first 10 epochs (or 15 epochs after my later change). Warmup is not the same as pretrain. Check learning_rates in your config, which defines the warmup. Do you already have it for 15 epochs? You might try to increase it even more. Or also use a newer config, like the one linked by @Spotlight0xff .

[yanghongjiazheng]

what config are you using? this one?

I use this one

Also what TF/CUDA/CUDNN versions do you have?

TF version is 1.8
CUDA version is 0.9.0

[yanghongjiazheng]

warmup step in my config is still 10.

[yanghongjiazheng]

So the nan problem is due to the heavy training datas?

[yanghongjiazheng]

I encountered this problem after adding 3000h training datas. When I used the same configuration training on 2000h corpus, the nan problem did not happen.

[albertz]

I would recommend to use some of our newer configs, and increase the learning rate warmup.

[Spotlight0xff]

Also you should update your TF and CUDA.

[christophmluscher]

This issue seems outdated. I will close. If necessary feel free to reopen.