shijx12/deepsim

tensorflow implementation of Generating Images with Perceptual Similarity Metrics based on Deep Networks

DeepSim

This is a tensorflow implementation of the paper Generating Images with Perceptual Similarity Metrics based on Deep Networks by Alexey Dosovitskiy, Thomas Brox.

This repo is based on CharlesShang's work TFFRCNN. I really appreciate their great work.

I mainly use the data load module of their work in ./deepSimGAN/util.py. You can remove all codes outside the ./deepSimGAN directory if you rewrite the DataFetcher class in the ./deepSimGAN/util.py script.

requirement

• python 2.7
• tensorflow >= 1.1.0
• python-opencv >= 3.2.0
• numpy >= 1.11.3
• tqdm

Training

To train your own deepsim model, you need to:

1. Prepare dataset and pretrained-model for encoder training.
2. Train your encoder and save the fine-tuned checkpoint.
3. Prepare dataset for generator and discriminator training.
4. Load fine-tuned encoder and train the generator and discriminator.

Prepare dataset and pretrained model for encoder

1. Download the training, validation, data and VOCdevkit to the target directory named VOCdevkit, such as /data/VOCdevkit. We use $VOCdevkit to refer to it and use $DeepSim to refer to this repo's root directory.

   cd $VOCdevkit
   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar
   wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCdevkit_18-May-2011.tar

2. Extract all of these tars

   tar xvf VOCtrainval_11-May-2012.tar
   tar xvf VOCdevkit_18-May-2011.tar

3. It should have this basic structure

   $VOCdevkit/                           # development kit
   $VOCdevkit/VOCcode/                   # VOC utility code
   $VOCdevkit/VOC2012                    # image sets, annotations, etc.
   # ... and several other directories ...

4. Create symlinks for the PASCAL VOC dataset

   cd $DeepSim/data
   ln -s $VOCdevkit VOCdevkit2012

5. Download pre-trained model VGG16 and put it in the path $DeepSim/data/pretrain_model/VGG_imagenet.npy

Train the encoder

The codes of encoder net deninition and training are in deepSimGAN/EncoderNet.py. You can use following command to start your training:

cd $DeepSim
python deepSimGAN/EncoderNet.py --weight_path data/pretrain_model/VGG_imagenet.npy --logdir output/encoder

All of checkpoints and summaries are stored in the given logdir. You can use tensorboard to monitor the training process

tensorboard --logdir output/encoder --host 0.0.0.0 --port 6006

Prepare dataset for generator and discriminator training

You can just reuse the Pascal VOC 2012.

If you want to use other datasets, remember to rewrite the class DataFetcher in ./deepSimGAN/util.py

Train your deepSimNet

The code of deepSimNet definition is in deepSimGAN/deepSimNet.py and the code of training is in deepSimGAN/main.py. You can use following command to start your training:

python deepSimGAN/main.py --encoder output/encoder --logdir output/deepsim

There are many other arguments which can be specified to influence your training. Please refer to the argument parser in deepSimGAN/main.py.

deepsim's Issues

coding error in nextbatch

Line 33-34 in util.py, function next batch
if self.resize: im = np.stack([cv2.resize(im[:,:,i], (cfg.RESIZED_SIZE, cfg.RESIZED_SIZE)) for i in range(im.shape[2])], axis=2)

the image index I at line 30 is overwritten by this new looping variable i.
Please check.

Also, can you confirm if the generator is working for the PASCAL VOC dataset, I see non-realistic repeated patterns in the outputs when I ran the code?

Reagarding Generating Images using Generated checkpoints.

Hi,
can you please help me with how to use the generated checkpoints to generate images?
Also, I have trained the model for 200 epochs only for now, do you think that's the problem that I keep getting an error whenever I try to restore the deepsim model, stating the graph opened and precious graph has different shape.
[512,1] lhs
[512,2] rhs.

Thanks

Can you provide trained model?

I want to use some layers of this network, training i will cost much time so i wonder if the author can provide trained model parameters, thanks!