Loss Nan

[stick-to]

Loss Nan

[snowadmirer]

The random crop algorithm is inconsiderate. There could be no valid targets in its output. To fix this bug, modify the random crop code part or add some post processing code to loss calculating.

I simply added some post processing code. If there is no valid targets in an images, the offset loss and size loss shall be zero, the keypoints loss could be calculated the same way by avoiding division by zero.

random crop部分的数据增强代码有问题，会crop出没有valid targets的图片，修改这一部分的代码或者在计算loss时加入后处理。即使图片中没有valid targets，loss也是可以计算的，将size loss和offset loss设置为0，在计算keypoints loss时避免除0

Centernet.py line 207：
offset_loss = tf.reduce_mean(tf.abs(offset_gt - offset))
size_loss = tf.reduce_mean(tf.abs(size_gt - size))
offset_loss = tf.cond(tf.cast(slice_index, tf.bool), lambda: offset_loss, lambda: tf.zeros_like(offset_loss, tf.float32))
size_loss = tf.cond(tf.cast(slice_index, tf.bool), lambda: size_loss, lambda: tf.zeros_like(size_loss, tf.float32))

Centernet.py line 250:
num_g = tf.maximum(num_g, tf.ones_like(num_g, dtype=tf.int32))
keypoints_loss = tf.reduce_sum(keypoints_pos_loss) / tf.cast(num_g, tf.float32) + tf.reduce_sum(keypoints_neg_loss) / tf.cast(num_g, tf.float32)

[nuaasxr]

training VOC2007 model, result is good
but i change the dataset, image set have one class, 950 images,
when training, the result 4~5 iters the loss is nan, i check the dataset no problem, and change the learning rate and batch_size, but can't help to loss, always loss nan, has anyone met it before? how can i fixed it? thx

yes, me too.
how are you now?