dvlab-research/outpainting_srn

Wide-Context Semantic Image Extrapolation, CVPR2019

Wide-Context Semantic Image Extrapolation

by Yi Wang, Xin Tao, Xiaoyong Shen, Jiaya Jia.

Introduction

This repository gives the Tensorflow implementation of the method in CVPR 2019 paper, 'Wide-Context Semantic Image Extrapolation'. This method can expand semantically sensitive objects (face, body) / scenes beyond image boundary.

Partial Results

Faces

Bodies

Scenes

More results

Network structure

Key components

• Small-to-large scheme
• Context normalization
• Relative spatial variant loss

Prerequisites

• Python3.5 (or higher)
• Tensorflow 1.6 (or later versions, excluding 2.x) with NVIDIA GPU or CPU
• OpenCV
• numpy
• scipy
• easydict

Installation

git clone https://github.com/shepnerd/outpainting_srn.git
cd outpainting_srn/

Testing

• Download the pretrained models through the following links (CelebA-HQ_256, Paris streetview, Cityscapes), and unzip and put them into checkpoints/.

• To test images in a folder, we call test.py with the opinion --dataset_path and --load_model_dir. We give some CelebA-HQ_256 examples in the imgs/celebahq-256/. For example:

  python test.py --dataset celebahq-256 --data_file ./imgs/celebahq-256/ --load_model_dir ./checkpoints/celebahq-srn-subpixel --random_mask 1

  or write / modify test.sh according to your own needs, then execute this script as (Linux platform):

  sh ./test.sh

The visual evaluations will be saved in the folder ./test_results/.

Training

• Training the model with the reconstruction loss (relative spatial variant loss) firstly (set the opinion --pretrain_network 1), then fine-tuning the model with all losses (--pretrain_network 0 and --load_model_dir [Your model path]) after the first stage converges. To pretrain the network,

  python train.py --dataset [DATASET_NAME] --data_file [DATASET_TRAININGFILE] --gpu_ids [NUM] --pretrain_network 1 --batch_size 16

  where [DATASET_TRAININGFILE] indicates a file storing the full paths of the training images. A simple example is given as:

  python train.py --dataset celebahq-256 --data_file ../celebahq-256_train.txt --gpu_ids 0 --img_shapes 256,256 --pretrain_network 1 --batch_size 8

• Then finetune the network,

  python train.py --dataset [DATASET_NAME] --data_file [DATASET_TRAININGFILE] --gpu_ids [NUM] --pretrain_network 0 --load_model_dir [PRETRAINED_MODEL_PATH] --batch_size 8

Datasets

All used datasets (CelebA-HQ, CUB200, Dog, DeepFashion, Paris-Streetview, Cityscape, and Places2) and their corresponding train/test splits are given in the paper.

Todo

• Other pretrained models
• ...

Disclaimer

• The training and evaluation performance on large-scale datasets (with a variety of categories) is unstable due to possible mode collapse in adversarial training.
• The used ID-MRF is a simplified version based on contextual loss. The step of excluding s is omitted for computational efficiency.

Citation

If our method is useful for your research, please consider citing:

@inproceedings{wang2019srn,
  title={Wide-Context Semantic Image Extrapolation},
  author={Wang, Yi and Tao, Xin and Shen, Xiaoyong and Jia, Jiaya},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={1399--1408},
  year={2019}
}

@inproceedings{wang2018image,
  title={Image Inpainting via Generative Multi-column Convolutional Neural Networks},
  author={Wang, Yi and Tao, Xin and Qi, Xiaojuan and Shen, Xiaoyong and Jia, Jiaya},
  booktitle={Advances in Neural Information Processing Systems},
  pages={331--340},
  year={2018}
}

Acknowledgments

Our code is built upon Image Inpainting via Generative Multi-column Convolutional Neural Networks and pix2pixHD.

Recent related work

Contact

Please send email to [email protected].

outpainting_srn's Issues

Is there any progress in this repo?

I try to just change the location of the mask, using the inpainting method. But the effect is not good. Can you give me some suggestions?

Paris-streetview pretrained model seems not work

------------ Options -------------
d_cnum: 64
data_file: ./imgs/celebahq-256/
dataset: paris_streetview
dataset_path: ./imgs/celebahq-256/
date_str: test_20211116-223942
fa_alpha: 0.5
feat_expansion_op: subpixel
g_cnum: 64
img_shapes: [256, 256, 3]
load_model_dir: ./checkpoints/cityscapes-srn-subpixel-gc64
mask_shapes: [128, 128]
model: srn
model_folder: test_20211116-223942_paris_streetview_srn_subpixel_s256x256_gc64_rand-mask_seed-1
model_prefix: snap
random_crop: True
random_mask: True
random_seed: False
saving_path: ./test_results/test_20211116-223942_paris_streetview_srn_subpixel_s256x256_gc64_rand-mask_seed-1
seed: 1
test_dir: ./test_results
test_num: -1
use_cn: True
-------------- End ----------------
The total number of testing images is 8, and we take 8 for test.
WARNING:tensorflow:From /Users/image outpainting/outpainting_srn/net/ops.py:104: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
Traceback (most recent call last):
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1589, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 3 in both shapes must be equal, but are 256 and 128. Shapes are [3,3,64,256] and [3,3,64,128]. for 'Assign_30' (op: 'Assign') with input shapes: [3,3,64,256], [3,3,64,128].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/image outpainting/outpainting_srn/test.py", line 79, in <module>
    vars_list))
  File "/Users/image outpainting/outpainting_srn/test.py", line 78, in <lambda>
    assign_ops = list(map(lambda x: tf.assign(x, tf.contrib.framework.load_variable(config.load_model_dir, x.name)),
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
    validate_shape=validate_shape)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
    use_locking=use_locking, name=name)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1756, in __init__
    control_input_ops)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1592, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 256 and 128. Shapes are [3,3,64,256] and [3,3,64,128]. for 'Assign_30' (op: 'Assign') with input shapes: [3,3,64,256], [3,3,64,128].

pretrained model

could you share other pretrained model like place2 or cityspace. if you are convenient,please share it. or can you send to my address [email protected]?? thank you very much!!

CPU Support

Hi,
It seams that this test.py does not support CPU.
When running on Mac, it always crashes.

Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1741, in
main()
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/danieltian/Desktop/Code/github_try/outpainting_srn/test.py", line 14, in
"nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 1103, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 66, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 46, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

How to resume training

when i was training a lot time ,and i have stop it ,can you tell me how to resume training? And the petrain_network is equal to 1or 0?

Pretrained model for Places2 dataset

Nice work!

Could you please share the pretrained model on the Places2 dataset?

assert self.opt.model in ['srn', 'srn-hr']

Traceback (most recent call last):
File "test.py", line 39, in
config = TestOptions().parse()
File "/home/wt/outpainting_srn/options/test_options.py", line 66, in parse
assert self.opt.model in ['srn', 'srn-hr']
AssertionError

decoding data

Hey, may I ask for help? because the code is implemented using older versions of tensorflow so i tried to run it using the newer version. However, i am facing problem in data.py file.

File "C:\Users\hp\OneDrive\Desktop\outpainting_srn-master\data\data.py", line 20, in next
im_gt =tf.io.decode_image(tf.io.read_file(self.data_queue[0]), channels=3)

TypeError: 'TensorSliceDataset' object is not subscriptable

thank you in advance.

Pretrained model for DeepFashion dataset

Hi @shepnerd, thank you for releasing the public implementation of your paper. I tried several experiments to replicate the results shown in the paper for the DeepFashion dataset, but I wasn't able to. I was wondering if you can share the checkpoints for the model trained on the DeepFashion dataset? That would be really helpful. Thanks

Additionally trained pretrained network yields bad results.

I'm planning to use this network for other domain and I'm trying to train the network by myself with train.py.
First of all, in order to clarify whether my way of training is correct, I proceeded training starting from pretrained checkpoint of CelebA which you provide in Google Drive, using CelebA dataset.
Here, I expected the results of test.py don't drastically change because I thought the provided
pretrained checkpoint was well converged one.
However, additionally trained network yielded bad results.
I think some parameters need to be tuned but I'm not sure how to fix this.

Dataset:
Acquired from the CelebA homepage (http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).
dataset/CelebA/Img/img_align_celeba.zip at the GoogleDrive (https://drive.google.com/drive/folders/0B7EVK8r0v71pWEZsZE9oNnFzTm8)

Commands:

$ python train.py --dataset celebahq-256 --data_file ~/projects/resize/celeba_train.txt --gpu_ids 0,1 --pretrain_network 0 --batch_size 8 --load_model_dir ./checkpoints/celebahq-srn-subpixel/
$ python test.py --dataset celebahq-256 --data_file ./imgs/celebahq-256/ --load_model_dir ./checkpoints/20200607-200756_celebahq-256_srn_subpixel_contextual_b8_s256x256_gc64_dc64_rand-mask/ --random_mask 1

Result

Training from scratch, with pretrained VGG19

I am trying to replicate the result of the dog dataset. To train from scratch, I rebuild it using PyTorch, and directly copy pytorch/model/{layer,loss}.py and pytorch/util/ from inpainting_gmcnn to compute mrf loss.

But whenever mrf_loss is included, the generator seems to quickly black out the image. I wonder if those files from inpainting_gmcnn is compatible to calculate mrf_loss? And if there is any detailed explanation about how the vgg19 features are used to compare?

Currently, the model is trained without mrf loss, the texture is bad. Below are my changed settings,
batch_size = 6
lrG = 1e-5
lrD = 5e-5
mrf_alpha = 0
others stay the same

My casual implementation: https://github.com/zcemycl/Pytorch_Outpainting_SRN

ValueError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [5,5,3,32] and [5,5,3,64]. for 'Assign' (op: 'Assign') with input shapes: [5,5,3,32], [5,5,3,64].

I got this error while doing the testing. Please help me out here.
Traceback (most recent call last):
File "test.py", line 79, in
vars_list))
File "test.py", line 78, in
assign_ops = list(map(lambda x: tf.assign(x, tf.contrib.framework.load_variable(config.load_model_dir, x.name)),
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 280, in assign
validate_shape=validate_shape)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 58, in assign
use_locking=use_locking, name=name)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3273, in create_op
compute_device=compute_device)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3313, in _create_op_helper
set_shapes_for_outputs(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2501, in set_shapes_for_outputs
return _set_shapes_for_outputs(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2474, in _set_shapes_for_outputs
shapes = shape_func(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2404, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn
require_shape_fn)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [5,5,3,32] and [5,5,3,64]. for 'Assign' (op: 'Assign') with input shapes: [5,5,3,32], [5,5,3,64].

ValueError: attempt to get argmax of an empty sequence

train problem

for example,if i use celebahq,i should follow the cmd：python train.py --dataset celebahq --data_file /data/yiwang/download/celeba_hd_256/train.txt --gpu_ids 0 --max_iters 80000 --g_cnum 64 --pretrain_network 1 --batch_size 8 --model srn --feat_expansion_op subpixel --random_mask 1 --random_crop 0 --use_cn 1

what's the train.txt? how should i get it when the dataset is Paris-Streetview?

The paris-streetview pretrained model with the Private links

Hi Yi Wang ,Can you share the paris-streetview pretrained model with the Private links.My email is [email protected],Thanks a lot,Thanks for your amazing work!

can you share the paris-streetview pretrained model with the Private links.My email is [email protected]