one-leaf/tensorflow

一些机器学习的实践

tensorflow's Issues

[Potential NAN bug] Loss may become NAN during training

Hello~

Thank you very much for sharing the code!

I try to run the script tensorflow/book/deeplearning/code/06-6.py. After some iterations, it is found that the training loss becomes NAN. After carefully checking the code, I found that the following code may trigger NAN in loss:

In tensorflow/book/deeplearning/code/06-6.py

# 通过设置log前的最小值不让归0，防止出现 log(0) 未定义
tf.clip_by_value(self.pred, 1e-15, 1.0)
cross_entropy = tf.reduce_mean(-tf.reduce_sum(self.y*tf.log(self.pred), reduction_indices=1)) 

I noticed that tf.clip_by_value(self.pred, 1e-15, 1.0) was used to avoid log(0). However, tf.clip_by_value will not change the value in self.pred. We need to get the return value of the clip function and re-assign it to self.pred
Making the following changes may get rid of NAN:

self.pred = tf.clip_by_value(self.pred, 1e-15, 1.0)
cross_entropy = tf.reduce_mean(-tf.reduce_sum(self.y*tf.log(self.pred), reduction_indices=1)) 

Hope to hear from you ~

Thanks in advance! : )