question on getting zero kl-divergence

[yanndubs]

question on getting zero kl-divergence

[YannDubs]

It is a bit hard to say without knowing what you do exactly. Here's what I would do to know where the issue comes from
1/ Make sure the KL is not 0 if you remove it from the loss. I.e. use a weight of 0 (no annealing, 0 from the start).
2/ Compare the KL (in 1) with the reconstruction loss in the same setting. Maybe the reconstruction loss is neglectable compared to the KL.
3/ If the reconstruction loss is neglectable you are either using a very strange task or not computing it correctly. Specifically, make sure you are averaging over the correct values when computing the loss (i.e. the number of pixels, the batch, the channels ...).