minhnhat93/tf-sndcgan

Tensorflow Implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (ICML 2017 workshop)

tf-SNDCGAN

Tensorflow implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (https://www.researchgate.net/publication/318572189_Spectral_Normalization_for_Generative_Adversarial_Networks, ICML 2017)

The implementation is based on the author's original code at: https://github.com/pfnet-research/chainer-gan-lib

This implementation works for tensorflow default data format "NHWC"

Spectral Normalization for Generative Adversarial Networks:

This method enforces Lipschitz-1 condition on the Discrminator of Wasserstein-GAN by normalizing its weight matrices with their own respective maximum singular value. This can be used together with Gradient Penalty in the paper "Improved Training of Wasserstein GAN".

The author uses a fast approximation method to compute the maximum singular value of weight matrices.

Quick run:

Keras is required for loading Cifar10 data set

python3 train.py

How to use spectral normalization:

# Import spectral norm wrapper
from libs.sn import spectral_normed_weight
# Create weight variable
W = tf.Variable(np.random.normal(size=[784, 10], scale=0.02), name='W', dtype=tf.float32)
# name of tf collection used for storing the update ops (u)
SPECTRAL_NORM_UPDATE_OPS = "spectral_norm_update_ops"
# call wrapping function, W_bar will be the spectral normed weight matrix
W_bar = spectral_normed_weight(W, num_iters=1, update_collection=SPECTRAL_NORM_UPDATE_OPS)
# Get the update ops
spectral_norm_update_ops = tf.get_collection(SPECTRAL_NORM_UPDATE_OPS)
...
# During training, run the update ops at the end of the iteration
for iter in range(max_iters):
    # Training goes here
    ...
    # Update ops at the end
    for update_op in spectral_norm_update_ops:
        sess.run(update_op)

For an example, see the file test_sn_implementation.py

Training curve:

Generated image samples on Cifar10:

Inception score:

After using in place batch norm update and use the optimal training parameters from the paper, I was able to match their claimed Inception score at 100k iteration: 7.4055686 +/- 0.087728456

The official github repostiory has an inception score of 7.41

Issues:

• GPU under-utilization: The original implementation of the author in chainer uses 80%+ GPU most of the time. On an NVIDIA GTX 1080TI, their implementation run at nearly 3 iterations/s. This implementation use less than 50% GPU and run at less than 2 iterations/s. Solved. It was the global_step assignment that makes tensorflow create new assign node for graph each iteration, slow down the execution. This also made the graph become very large over time leading to gigantic event files. GPU utilization is now around 85+%

• No Fréchet Inception Distance (https://arxiv.org/abs/1706.08500) evaluation yet.

tf-sndcgan's Issues

How do I generate a lot of images?

How do I generate a lot of images?

python train.py

if max_iter = 100000, then only 64 images are generated

I want to generate a lot of features, but also a lot of images.

how ?

I want to apply to tf.layers.con2d

If I want to apply to tf.layers.con2d, what should I do?
I want to put it in the kernel initializer.

for example

    x = tf.layers.conv2d(inputs=x, filters=channels,
                                      kernel_size=kernel, 
                                      kernel_initializer=spectral_normed_weight(),
                                      strides=stride)

Should I just use tf.nn.conv2d... ?

Should I apply only to the discriminator?

Should I apply only to the discriminator?

Do not apply spectral normalization to both generator and discriminator?

Why do you use `update collection`?

While skimming through your code, I don't understand why do you need update_collection variable here,
https://github.com/minhnhat93/tf-SNDCGAN/blob/master/train.py#L39

If i don't add update collection in the code.
just write like below

    with tf.control_dependencies([u.assign(u_hat)]):
        w_norm = tf.reshape(w_norm, w_shape)

d_real and d_loss both use the above procedure.
Is there anything wrong?
thx

Data format?

While reading

tf-SNDCGAN/libs/sn.py

Lines 15 to 17 in 6dce068

	W_reshaped = tf.reshape(W, [-1, W_shape[-1]])
	if u is None:
	u = tf.get_variable("u", [1, W_shape[-1]], initializer=tf.truncated_normal_initializer(), trainable=False)

, I noticed you use W_shape[-1]. Did you assume data format is NHWC? So in the case I use NCHW, it should be changed to W_shape[1]?

Input Error for Evaluation

There comes a error when doing evaluation:
Cannot feed value of shape (100, 32, 32, 3) for Tensor 'ExpandDims:0' at libs/inception_score/model.py line 45.

Thanks for sharing.

warning in `spectral_normed_weight`

Hi! I've been using your implementation of SNGAN and I have some questions. If I follow the training procedure intrain.py, then the warning in the spectral_normed_weight function will print

Setting update_collection to None will make u being updated every W execution. This maybe undesirable. Please consider using a update collection instead.

I wonder if the warning was meaningful. I also found that the way you used the collection api is different in train.py and test_sn_implementation.py. Which way should I follow?

Thanks!