cwkx/gon

Gradient Origin Networks - a new type of generative model that is able to quickly learn a latent representation without an encoder

Gradient Origin Networks

This paper has been accepted at ICLR 2021.

This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points. The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.

(GON)

(Variational GON)

(Implicit GON)

The code is available in GON.py and licensed under the MIT license. For more information, please visit the Project Page. Here is a link to the paper. The implicit GON version uses a SIREN (Implicit Neural Representations with Periodic Activation Functions, Sitzmann et al., 2020).

Citation

If you find this useful, please cite:

@inproceedings{bond2020gradient,
   title     = {Gradient Origin Networks},
   author    = {Sam Bond-Taylor and Chris G. Willcocks},
   booktitle = {International Conference on Learning Representations},
   year      = {2021},
   url       = {https://openreview.net/pdf?id=0O_cQfw6uEh}
}

gon's Issues

Where can I obtain the validation reconstruction loss?

Hi,

In the paper, I see there is a comparison between different models. Where can I find the corresponding code that calculates the validation reconstruction loss? Thank you!

Ease of training on Color images ?

Hello,
Thank you for your innovation & sharing it with others. I see that all the training codes use single channel image datasets. (though a cifar10 result is shown in the paper).
Is it is as easy to train it on 3 channel (color images) ? Or are there any caveats ?
Please share your know how.

thanks !

GON for large images (64x64)?

Thanks for the great work.

I try your GON with additional layers to generate large images from CeleA at 64x64 resolution.
However, the results are blurred and distorted.
I have tried adding a smooth Conv layer or Residual Conv layers after the ConvTrans, but the problem persists.
Could you kindly share your GON for large images as used for Figure 9 in your paper? Thank you.

Colab error

Hi,

I would like to see the effect of the network on a single image. Is this a good use case, even though the work is targeted to a dataset?

Also, In the colab file, at L_inner, indicated below,

for step in range(501):
    # sample a batch of data
    x, t = next(train_iterator)
    x, t = x.to(device), t.to(device)
    x = x.permute(0, 2, 3, 1).reshape(batch_size, -1, n_channels)

# compute the gradients of the inner loss with respect to zeros (gradient origin)
    z = torch.zeros(batch_size, 1, num_latent).to(device).requires_grad_()
    z_rep = z.repeat(1,c.size(1),1)
    g = F(torch.cat((c, z_rep), dim=-1))
    **L_inner = ((g - x)**2).sum(1).mean()**
    z = -torch.autograd.grad(L_inner, [z], create_graph=True, retain_graph=True)[0]

what is t used for? it does not appear to be used in later parts of the file? or maybe I missed it.

`

For the inference, latent vector (z) should be saved?

Hi. Thank you for sharing great works.

I have some question related to the memory issues.

After training GON for some dataset (e.g. MNIST), latent vector (z) should be saved for the inference for reconstructing the whole dataset?

I think latent vector with the size of (number of image x nz x 1 x 1) is not small.

Is it right ?