yongfeiyan/gumbel_softmax_vae

PyTorch implementation of a Variational Autoencoder with Gumbel-Softmax Distribution

gumbel_softmax_vae's Issues

latent dim

Hi, what does the latent_dim mean in your code? Could it be changed to other numbers? I can understand that categorical_dim means 10 categories for 10 digits, but I'm confused about the latent_dim. Thanks!

hard Gumbel softmax = ST Gumbel softmax?

Some questions on KLD

In your code the KL divergence is calculated by:
KLD = torch.sum(qy * (log_qy - 1. / categorical_dim), dim=-1).mean()
I think, for the 1. / categorical_dim, it should be replaced by the torch.log(1. / categorical_dim), otherwise, it is not the KL divergence.

mistake in Gumbel softmax sample function.

The gumbel_softmax_sample function is logits + gumbel_sample. But it should have been F.log_softmax(logits) + gumbel_sample according to the paper. Is this not a mistake?

Bernoulli Variables

Hi @YongfeiYan, thanks for sharing your project with us.

I would like to know how do I modify the implementation to use Bernoulli variables. I need the network to generate codes consisting of 0s and 1s.

Thanks.

Unnecessary parameter

Gumbel_Softmax_VAE/gumbel_softmax_vae.py

Line 83 in 7d3df6c

def __init__(self, temp):

Is this temp parameter needed? Isn't temp passed in during the forward pass?

KL divergence term

I was wondering what exactly this line in the KLD calculation does:
log_ratio = torch.log(qy * categorical_dim + 1e-20)

In the definition of the ELBO loss, the KLD should be computed between the variational distribution q(z|x) and the prior p(z). How come you did not simply use the pytorch implementation of KLD (kl_div)?

Temperature reset after each epoch

Hey, sorry I haven't run your code to check, but here does this reset the temperature parameter after each epoch?

Can KL divergence be negative?

I'm using your loss function code in my project and getting negative values...

Is it normal to get negative KLD value?