christiancosgrove/pytorch-spectral-normalization-gan

Paper by Miyato et al. https://openreview.net/forum?id=B1QRgziT-

SN-GAN (spectral normalization GAN) in PyTorch

Based on the paper "Spectral Normalization for Generative Adversarial Networks" by Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida

ICLR 2018 preprint: https://openreview.net/forum?id=B1QRgziT-

CIFAR-10 Samples

Implementation Details

This code implements both DCGAN-like and ResNet GAN architectures. In addition, training with standard, Wasserstein, and hinge losses is possible.

To get ResNet working, initialization (Xavier/Glorot) turned out to be very important.

Training

Train ResNet generator and discriminator with hinge loss: python main.py --model resnet --loss hinge

Train ResNet generator and discriminator with wasserstein loss: python main.py --model resnet --loss wasserstein

Train DCGAN generator and discriminator with cross-entropy loss: python main.py --model dcgan --loss bce

pytorch-spectral-normalization-gan's Issues

Large D/G losses?

Hi,

I'm using recently-released PyTorch 0.4 (not sure if that's causing the funky numbers I'm getting), but I'm getting the following (with python main.py --model resnet --loss wasserstein):

disc loss tensor(-0.2347, device='cuda:0') gen loss tensor(-0.6066, device='cuda:0')
disc loss tensor(-120.7743, device='cuda:0') gen loss tensor(-1614.5465, device='cuda:0')
disc loss tensor(-121.0873, device='cuda:0') gen loss tensor(-1225.4401, device='cuda:0')
disc loss tensor(-56.4558, device='cuda:0') gen loss tensor(-2320.5115, device='cuda:0')
disc loss tensor(-45.6140, device='cuda:0') gen loss tensor(-2665.3479, device='cuda:0')
disc loss tensor(-46.4297, device='cuda:0') gen loss tensor(-3849.7197, device='cuda:0')
disc loss tensor(-39.8169, device='cuda:0') gen loss tensor(-4879.6089, device='cuda:0')
disc loss tensor(-56.9688, device='cuda:0') gen loss tensor(-5421.9688, device='cuda:0')
disc loss tensor(-3.2100, device='cuda:0') gen loss tensor(-4737.8677, device='cuda:0')
disc loss tensor(-36.7729, device='cuda:0') gen loss tensor(-4344.2520, device='cuda:0')
disc loss tensor(-55.6719, device='cuda:0') gen loss tensor(-6263.5303, device='cuda:0')
disc loss tensor(-62.0518, device='cuda:0') gen loss tensor(-7915.4751, device='cuda:0')
disc loss tensor(-0.5933, device='cuda:0') gen loss tensor(-7315.9282, device='cuda:0')
disc loss tensor(-26.8652, device='cuda:0') gen loss tensor(-10451.8770, device='cuda:0')
disc loss tensor(-48.6777, device='cuda:0') gen loss tensor(-8293.3584, device='cuda:0')

Is this meant to happen?

Thanks!

About the G loss.

I get confused about the G loss.

In the code of Wasserstein(https://github.com/martinarjovsky/WassersteinGAN/blob/f81eafd2aa41e93698f203732f8f395abc70be02/main.py#L212) the author use

errG = netD(fake)

where fake = netG(z).

However, in your implementation, the G loss is

gen_loss = -discriminator(generator(z)).mean().

Theoretically, I believe that the G loss should be -D(G(z)) because the G is expected to be able
to 'cheat' the D.

Should I apply only to the discriminator?

Should I apply only to the discriminator?

Do not apply spectral normalization to both generator and discriminator?

reproduce resnet results on CIFAR10

Hey,
I'm trying to reproduce the published results of the resnet network on the CIFAR10 dataset.
As published in the paper, they were able to achieve a score of 8.22 using resnet network, however I couldn't achieve above 7.5+- in this net. Am I missing something ?
I tried the 3 available loss options, only the models trained on bce were converged, while the other collapsed.
Thanks

32x32 vs 64x64 issue

When trying to train the model on a custom dataset with an image size of 64x64 pixels the discriminator loss drops to 0 after one epoch. Doing the same with a 32x32 resolution works fine. Given that CIFAR-10 is 32x32 can it be that that is the maximum resolution this particular architecture allows even though the generated example images are 64x64?

Would it make sense to apply batch normalisation along with spectral normalisation to the paper?

In the paper Inductive Guided Filter:Real Time Deep Image Matting with Weakly Annotated Masks on Mobile Devices, Both spectral normalisation and batch normalisation is being applied to the layers of the discriminator.

I was wondering would it make any sense to apply both, and If yes, How would it be implemented. They have not mentioned the order in which both the normalisations are applied .

How does _u and _v update?

Thanks for your clear implementation.
I encounter the problem about the _u and _v update policy. I've noticed that in your implementation, _u is updated before op's inference phase, does _u need back propagate update by the gradient?
Another problem is, should I update the gradient created by w_bar to original weight directly? I found you mentioned the point, but I think update to w_bar seems more reasonable, and during the next iteration taking w_bar as original weight, am I right?

How to use in 3D conv?

The paper and the code are both for the 2D convolution of the sn limit w, then how to deal with w in the 3D convolution?

GrayScale data

Hi there,
I am trying to apply this SNGAN implementation on grayscale cell images, my data size is quite enough large ~ 100,000 images. I have used resnet architecture models edited by adding an extra layer to generate/discriminate 64 pxl images and gan loss. By training the model about 15K iterations (~ 10 epochs) I could not recognize visual improvement for the generated samples as they suffer from checkboard and grid artifacts.
The training curves are shown below:

I am not sure if I have to train for a longer time (more epochs), however, the training curves and the visual samples indicate abnormal case!
Any advice, please.

pytorch version?

Nice code! One question: Does it support PyTorch v0.4？

does it use w_bar when update u and v?

in the function _update_u_v(self), does it use w_bar rather than w to update u and v? I mean, should i replace w = getattr(self.module, self.name + '_bar') with w = getattr(self.module, self.name).

spectral_normalization_nondiff.py

Thanks for your code . I'm quite confused with the nondiff version, what does it for , and what's the difference with the diff version. Thanks

sn-wgan results

Hi, I'm trying to apply spectral normalization to wasserstein gans. I've failed to make it work in my project so tried your repository to get more intuition of how to train them. However I had no training progress for about a day of training.

In original WGAN paper they seem to use 25 epoches for training. I've waited for 130 so far and got the following results (with your code)

Some introspection in discriminator gave me interesting insights. Indeed WGAN is Lipshitz with constant ~2.4 according to gradient norm histograms. However sn-gan or regular gan seem to have gradients with larger average norm. Below are gradient norms for trained for 100 epoches sn-gan and sn-wgan and gan. On cifar dataset (and mnist for regular gan), I used 5 discriminator iterations per generator update.

Seems like sn-gan has better gradients for generator according to histograms. For regular GAN I see there are small gradients even for images from generator.

Did you manage to get satisfactory results for SN-WGAN and how if yes?

My current intuition says me that devil is in gradient or their bias (I don't think I have that biase as I use 1024 batch size). Convergence might be too slow because of these gradients.

slef.u can't be updated?

The self.u in SpectralNorm seems never update during training. And u in spectral_norm will be reinitialized with u = Variable(W.data.new((W.data.shape[1]))).data.normal_(0,1) in every epoch. The type of self.u is None all the time.
Any way to figure this out?
Thanks!

Can it be used directly on a linear layer?

Can it be used directly on a linear layer? e.g., SpectralNorm(Linear(dim, item)

w_g is not defined in model.py

The title says it all :)

Didn't try to work it out from the code, but with w_g = 4 it seems to work (dcgan, bce).

Spectral Normalization for Recurrent Layers

Hi,

Just would like to know how to refactor 'spectral_normalization.py' such that it applies spectral normalization to the weights of a recurrent layer (e.g., GRU). Is it correct to change the 'name' argument of the init method of the SpectralNorm class so to indicate 'w_ih' and 'w_hh' instead of 'weight'?

Best

How can I generate bigger than 64x64 with this source?

Hi, @christiancosgrove

How can I generate bigger than 64x64 with this source?

I want to get a size of 128x128 or above sizeed image.

If Spectral Norm is good for safe training of GAN, I think that bigger sized generation is also possible..., Right?

Thanks in advance~
from @bemoregt.