Shapes are inconsistent in your implementation

[jaanli]

Shapes are inconsistent in your implementation

[jaanli]

You're right - the distribution for q(z) and p(z) have the same shape for a single datapoint.

p(z) does not need to have a batch size dimension, as the kl function supports broadcasting across the batch size dimension - so it's effectively as if every element in the batch has a p(z) of dimension latent_dim.

Hope this helps!

[jaanli]

Thanks!

I'm not sure I follow:

• the kl divergence has an analytic form in this case (we do not use samples from p_z to compute it, so I don't see an issue with the shape)

• p_z does not produce a 1D tensor, it produces a tensor of shape latent_dim

Am I missing something?

[lenhhoxung86]

Hi,
What I mean is that the ways to create q_z and p_z are different.
q_z = distributions.Normal(loc=q_mu, scale=q_sigma)
Here q_mu and q_sigma have the same shape of [batch_size, latent_dim], but it's not the case for p_z.

Here is an example:

>>> x = tf.distributions.Normal(loc=np.zeros(3, dtype=np.float32),scale=np.ones(3,dtype=np.float32))
>>> sess.run(x.sample())
array([ 1.4619417, -0.7553768,  2.2952332], dtype=float32)
>>> 
>>> y = tf.distributions.Normal(loc=np.zeros((2,3), dtype=np.float32),scale=np.ones((2,3),dtype=np.float32))
>>> sess.run(y.sample())
array([[ 0.73629814,  1.0238795 ,  0.00905563],
       [-1.0864298 , -1.0123701 ,  0.7973023 ]], dtype=float32)

Of course, we calculate the kl divergence based on the distributions x and y but I think the samples created by the two distributions should have the same shape, aren't they?