dvlab-research/outpainting_srn

Wide-Context Semantic Image Extrapolation, CVPR2019

outpainting_srn's Issues

pretrained model

could you share other pretrained model like place2 or cityspace. if you are convenient,please share it. or can you send to my address [email protected]?? thank you very much!!

Paris-streetview pretrained model seems not work

------------ Options -------------
d_cnum: 64
data_file: ./imgs/celebahq-256/
dataset: paris_streetview
dataset_path: ./imgs/celebahq-256/
date_str: test_20211116-223942
fa_alpha: 0.5
feat_expansion_op: subpixel
g_cnum: 64
img_shapes: [256, 256, 3]
load_model_dir: ./checkpoints/cityscapes-srn-subpixel-gc64
mask_shapes: [128, 128]
model: srn
model_folder: test_20211116-223942_paris_streetview_srn_subpixel_s256x256_gc64_rand-mask_seed-1
model_prefix: snap
random_crop: True
random_mask: True
random_seed: False
saving_path: ./test_results/test_20211116-223942_paris_streetview_srn_subpixel_s256x256_gc64_rand-mask_seed-1
seed: 1
test_dir: ./test_results
test_num: -1
use_cn: True
-------------- End ----------------
The total number of testing images is 8, and we take 8 for test.
WARNING:tensorflow:From /Users/image outpainting/outpainting_srn/net/ops.py:104: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
Traceback (most recent call last):
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1589, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 3 in both shapes must be equal, but are 256 and 128. Shapes are [3,3,64,256] and [3,3,64,128]. for 'Assign_30' (op: 'Assign') with input shapes: [3,3,64,256], [3,3,64,128].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/image outpainting/outpainting_srn/test.py", line 79, in <module>
    vars_list))
  File "/Users/image outpainting/outpainting_srn/test.py", line 78, in <lambda>
    assign_ops = list(map(lambda x: tf.assign(x, tf.contrib.framework.load_variable(config.load_model_dir, x.name)),
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 219, in assign
    validate_shape=validate_shape)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 60, in assign
    use_locking=use_locking, name=name)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1756, in __init__
    control_input_ops)
  File "/Users/leiyiming/opt/anaconda3/envs/outpainting_srn/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1592, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimension 3 in both shapes must be equal, but are 256 and 128. Shapes are [3,3,64,256] and [3,3,64,128]. for 'Assign_30' (op: 'Assign') with input shapes: [3,3,64,256], [3,3,64,128].

assert self.opt.model in ['srn', 'srn-hr']

Traceback (most recent call last):
File "test.py", line 39, in
config = TestOptions().parse()
File "/home/wt/outpainting_srn/options/test_options.py", line 66, in parse
assert self.opt.model in ['srn', 'srn-hr']
AssertionError

The paris-streetview pretrained model with the Private links

Hi Yi Wang ,Can you share the paris-streetview pretrained model with the Private links.My email is [email protected],Thanks a lot,Thanks for your amazing work!

Pretrained model for Places2 dataset

Nice work!

Could you please share the pretrained model on the Places2 dataset?

can you share the paris-streetview pretrained model with the Private links.My email is [email protected]

Pretrained model for DeepFashion dataset

Hi @shepnerd, thank you for releasing the public implementation of your paper. I tried several experiments to replicate the results shown in the paper for the DeepFashion dataset, but I wasn't able to. I was wondering if you can share the checkpoints for the model trained on the DeepFashion dataset? That would be really helpful. Thanks

Is there any progress in this repo?

I try to just change the location of the mask, using the inpainting method. But the effect is not good. Can you give me some suggestions?

How to resume training

when i was training a lot time ,and i have stop it ,can you tell me how to resume training? And the petrain_network is equal to 1or 0?

ValueError: attempt to get argmax of an empty sequence

CPU Support

Hi,
It seams that this test.py does not support CPU.
When running on Mac, it always crashes.

Traceback (most recent call last):
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1741, in
main()
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1735, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/pydevd.py", line 1135, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/Applications/PyCharm CE.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/danieltian/Desktop/Code/github_try/outpainting_srn/test.py", line 14, in
"nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 1103, in argmax
return _wrapfunc(a, 'argmax', axis=axis, out=out)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 66, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 46, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: attempt to get argmax of an empty sequence

train problem

for example,if i use celebahq,i should follow the cmd：python train.py --dataset celebahq --data_file /data/yiwang/download/celeba_hd_256/train.txt --gpu_ids 0 --max_iters 80000 --g_cnum 64 --pretrain_network 1 --batch_size 8 --model srn --feat_expansion_op subpixel --random_mask 1 --random_crop 0 --use_cn 1

what's the train.txt? how should i get it when the dataset is Paris-Streetview?

Additionally trained pretrained network yields bad results.

I'm planning to use this network for other domain and I'm trying to train the network by myself with train.py.
First of all, in order to clarify whether my way of training is correct, I proceeded training starting from pretrained checkpoint of CelebA which you provide in Google Drive, using CelebA dataset.
Here, I expected the results of test.py don't drastically change because I thought the provided
pretrained checkpoint was well converged one.
However, additionally trained network yielded bad results.
I think some parameters need to be tuned but I'm not sure how to fix this.

Dataset:
Acquired from the CelebA homepage (http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).
dataset/CelebA/Img/img_align_celeba.zip at the GoogleDrive (https://drive.google.com/drive/folders/0B7EVK8r0v71pWEZsZE9oNnFzTm8)

Commands:

$ python train.py --dataset celebahq-256 --data_file ~/projects/resize/celeba_train.txt --gpu_ids 0,1 --pretrain_network 0 --batch_size 8 --load_model_dir ./checkpoints/celebahq-srn-subpixel/
$ python test.py --dataset celebahq-256 --data_file ./imgs/celebahq-256/ --load_model_dir ./checkpoints/20200607-200756_celebahq-256_srn_subpixel_contextual_b8_s256x256_gc64_dc64_rand-mask/ --random_mask 1

Result

Training from scratch, with pretrained VGG19

I am trying to replicate the result of the dog dataset. To train from scratch, I rebuild it using PyTorch, and directly copy pytorch/model/{layer,loss}.py and pytorch/util/ from inpainting_gmcnn to compute mrf loss.

But whenever mrf_loss is included, the generator seems to quickly black out the image. I wonder if those files from inpainting_gmcnn is compatible to calculate mrf_loss? And if there is any detailed explanation about how the vgg19 features are used to compare?

Currently, the model is trained without mrf loss, the texture is bad. Below are my changed settings,
batch_size = 6
lrG = 1e-5
lrD = 5e-5
mrf_alpha = 0
others stay the same

My casual implementation: https://github.com/zcemycl/Pytorch_Outpainting_SRN

ValueError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [5,5,3,32] and [5,5,3,64]. for 'Assign' (op: 'Assign') with input shapes: [5,5,3,32], [5,5,3,64].

I got this error while doing the testing. Please help me out here.
Traceback (most recent call last):
File "test.py", line 79, in
vars_list))
File "test.py", line 78, in
assign_ops = list(map(lambda x: tf.assign(x, tf.contrib.framework.load_variable(config.load_model_dir, x.name)),
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 280, in assign
validate_shape=validate_shape)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 58, in assign
use_locking=use_locking, name=name)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3273, in create_op
compute_device=compute_device)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3313, in _create_op_helper
set_shapes_for_outputs(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2501, in set_shapes_for_outputs
return _set_shapes_for_outputs(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2474, in _set_shapes_for_outputs
shapes = shape_func(op)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2404, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn
require_shape_fn)
File "/media/disk/gds/miniconda3/envs/out/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Dimension 3 in both shapes must be equal, but are 32 and 64. Shapes are [5,5,3,32] and [5,5,3,64]. for 'Assign' (op: 'Assign') with input shapes: [5,5,3,32], [5,5,3,64].

decoding data

Hey, may I ask for help? because the code is implemented using older versions of tensorflow so i tried to run it using the newer version. However, i am facing problem in data.py file.

File "C:\Users\hp\OneDrive\Desktop\outpainting_srn-master\data\data.py", line 20, in next
im_gt =tf.io.decode_image(tf.io.read_file(self.data_queue[0]), channels=3)

TypeError: 'TensorSliceDataset' object is not subscriptable

thank you in advance.