mcahny/deep-video-inpainting

Official pytorch implementation for "Deep Video Inpainting" (CVPR 2019)

deep-video-inpainting's Issues

Source code for Space time cubic puzzles

Hi, Thanks for your sharing.
Are you planning to share your source code of "Self-Supervised Video Representation
Learning with Space-Time Cubic Puzzles"?

Training Code

Hi, dear authors,

I'm glad to find this work and test it successfully under your guidance. The results show the magic “eclipse”. It's so interesting and I'd like to try it on my own task——defects removal of cartoons.

Therefore, I'd like to ask you to share the entire training code with me, could you? (#^.^#)
Hope for your reply~

not working for fixed region inpainting

Hello, good code. But I find it does work for fixed region inpainting , such as the logo with the fixed position on the video.

Anyone can give some advises? @mcahny

I can not run the code

I miss the problem as follows, what is wrong ? Thank you very much
File "demo_vi.py", line 166, in
outputs, _, , , _ = model(masked_inputs, masks, lstm_state, prev_feed, t)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 121, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/vinet.py", line 174, in forward
flow2 = self.flownet(f1, f2)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/flow_modules.py", line 132, in forward
corr = self.corr(x1.contiguous(), x2.contiguous())
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/correlation_package/modules/correlation.py", line 17, in forward
result = CorrelationFunction(self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)(input1, input2)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting-master/models/correlation_package/functions/correlation.py", line 30, in forward
self.pad_size, self.kernel_size, self.max_displacement,self.stride1, self.stride2, self.corr_multiply)
File "/home/vis/zhaoxiang05/video_inpainting/Deep-Video-Inpainting/python3.6/lib/python3.6/site-packages/torch/utils/ffi/init.py", line 202, in safe_call
result = torch._C._safe_call(*args, **kwargs)
torch.FatalError: sizes must be non-negative (THCTensor_resizeNd at /pytorch/aten/src/THC/THCTensor.cpp:108)

demo txt missing

Deep-Video-Inpainting/demo_vi.py

Line 26 in f7dcef0

DTset = DAVIS(DAVIS_ROOT, imset='2016/demo_davis.txt', size=(opt.crop_size, opt.crop_size))

cannot find 2016/demo_davis.txt in your project, so I can't run this demo?

Tensorflow version?

@mcahny @SeokjuLee is there a tensorflow version in works?

Will we release the entire training code?

Hi author,
It is pretty hard to train the network from train_git.py. Can you please give us a howto document or release the entire training code?

Thanks!

readme.md tutorial, how to request

How do i run this repository on google-colab???

Cannot run code!

@SeokjuLee @mcahny
Getting this error when trying to run !bash install.sh :

Compiling resample2d kernels by nvcc...
rm: cannot remove 'Resample2d_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
In file included from /usr/local/cuda/bin/../targets/x86_64-linux/include/host_config.h:50:0,
from /usr/local/cuda/bin/../targets/x86_64-linux/include/cuda_runtime.h:78,
from :0:
/usr/local/cuda/bin/../targets/x86_64-linux/include/crt/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
x86_64-linux-gnu-gcc: error: /content/Deep-Video-Inpainting/lib/resample2d_package/src/Resample2d_kernel.o: No such file or directory
Traceback (most recent call last):
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 207, in link
self.spawn(linker + ld_args)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 209, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "build.py", line 31, in
ffi.build()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 184, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 108, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/usr/local/lib/python3.6/dist-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/usr/local/lib/python3.6/dist-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.class.name, e))
cffi.VerificationError: LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Compiling correlation kernels by nvcc...
rm: cannot remove 'correlation_cuda_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
In file included from /usr/local/cuda/bin/../targets/x86_64-linux/include/host_config.h:50:0,
from /usr/local/cuda/bin/../targets/x86_64-linux/include/cuda_runtime.h:78,
from :0:
/usr/local/cuda/bin/../targets/x86_64-linux/include/crt/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
#error -- unsupported GNU version! gcc versions later than 6 are not supported!
^~~~~
x86_64-linux-gnu-gcc: error: /content/Deep-Video-Inpainting/models/correlation_package/src/correlation_cuda_kernel.o: No such file or directory
Traceback (most recent call last):
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 207, in link
self.spawn(linker + ld_args)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 909, in spawn
spawn(cmd, dry_run=self.dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 36, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python3.6/distutils/spawn.py", line 159, in _spawn_posix
% (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 51, in _build
dist.run_command('build_ext')
File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
cmd_obj.run()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 339, in run
self.build_extensions()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
self._build_extensions_serial()
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
self.build_extension(ext)
File "/usr/lib/python3.6/distutils/command/build_ext.py", line 558, in build_extension
target_lang=language)
File "/usr/lib/python3.6/distutils/ccompiler.py", line 717, in link_shared_object
extra_preargs, extra_postargs, build_temp, target_lang)
File "/usr/lib/python3.6/distutils/unixccompiler.py", line 209, in link
raise LinkError(msg)
distutils.errors.LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "build.py", line 31, in
ffi.build()
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 184, in build
_build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 108, in _build_extension
outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
File "/usr/local/lib/python3.6/dist-packages/cffi/api.py", line 723, in compile
compiler_verbose=verbose, debug=debug, **kwds)
File "/usr/local/lib/python3.6/dist-packages/cffi/recompiler.py", line 1526, in recompile
compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 22, in compile
outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
File "/usr/local/lib/python3.6/dist-packages/cffi/ffiplatform.py", line 58, in _build
raise VerificationError('%s: %s' % (e.class.name, e))
cffi.VerificationError: LinkError: command 'x86_64-linux-gnu-gcc' failed with exit status 1

An exiting work! Waiting for your code!

about the evaluation index is low

According to the mask in the Fuseformer, I measured psnr=26.9 in the youtube-vos test set, far lower than 29.2 in the Fuseformer. This code only modifies the parameter pre_run = 2 and the test image size is set to 432,240. Is it the parameter setting problem? Another setting: python3.7, pytorch1.4.0+cu100. Look forward to your feedbacks and suggestions soon.

Align-and-Attend Network Implementation

Thanks again for the great code. I have a quick question about the "Align-and-Attend Network for Globally and Locally Coherent Video Inpainting" implementation. Do you or your group have any plan to release the codes related to this paper too? Thanks. :)

Can batch size only be set to 1 when you train?

Results on Davis

Thanks to the authors for sharing the code. Do you have any plan on releasing pre-computed results on the Davis dataset so that we can make a direct comparison to your results in a follow-up paper? Many thanks!

how to train the model without using pretrained path

Instead of using pretrain path how to train the model with youtubevos dataset as mentioned in the project paper

have anyone tried training?

Why your Conv-LSTM code doesn't match the pipeline?

demo_vi.py SEGMENTATION FAULT

@mcahny @SeokjuLee
I got the first step, 1) Compile Resample2d, Correlation dependencies, complete by using the versions from the flownet2 repo, but can't seem to get the demo_vi.py script to run.

It makes it to line 162, where the following line produces a segmentation fault:
outputs, , , , _ = model(masked_inputs, masks, lstm_state, prev_feed, t)
I've checked the values of the input variables, where t=0 (first frame), lstm_state=None, and the other three tensors have the following shapes:
masked_inputs.shape = torch.Size([1, 3, 5, 512, 512])
masks_.shape = torch.Size([1, 1, 5, 512, 512])
prev_feed.shape = torch.Size([1, 5, 512, 512])

I could also print the model structure if that helps.

My version of torch is 1.0.0, and CUDA is 10.0. Not sure if these versions could be the issue...

could you please tell me version of cuda and cudnn？

a better inpainting method ?

《Deep Flow-Guided Video Inpainting》might generate better inpainting results，maybe you can check
https://github.com/nbei/Deep-Flow-Guided-Video-Inpainting around July 14th when they release code :)

Can you tell me how to train this model? THANKS!!!

have a problem when bash install.sh

when bash install.sh by conda envs
I meet a issue(it seems like i need to set a PATH for CUDA but i don't know how to set it for conda envs):
Traceback (most recent call last):
File "setup.py", line 25, in
], extra_compile_args={'cxx': cxx_args, 'nvcc': nvcc_args})
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 498, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 577, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1252, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.
Traceback (most recent call last):
File "setup.py", line 25, in
], extra_compile_args={'cxx': cxx_args, 'nvcc': nvcc_args})
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 498, in CUDAExtension
library_dirs += library_paths(cuda=True)
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 577, in library_paths
if (not os.path.exists(_join_cuda_home(lib_dir)) and
File "/home/mobax/anaconda3/envs/vinet/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1252, in _join_cuda_home
raise EnvironmentError('CUDA_HOME environment variable is not set. '
OSError: CUDA_HOME environment variable is not set. Please set it to your CUDA install root.

2 questions about the codes

Dear authors,
Thank you for sharing your codes. I'm very impressed from your results.
While looking over your codes, there are 2 things I can't understand. Please look over these questions.

1. in demo_vi.py, line 167
   if opt.double_size:
   prev_mask_ = masks_[:,:,2]*0.5
   => Why do you multiply 0.5 to mask for the double_size case?

2. in vinet.py, line 164

enc_input = torch.cat([masked_img, ones, ones*mask], dim=1)

For the encoder's inputs,
Why do you insert ones channel ? I thinks ones channels has no information for inpainting.
Why do you multiply ones channel to the mask? Is the ones * mask same with just the original mask?

I appreciate in advance.

How long will it take to release your codes?

Thanks for the interesting paper. Since CVPR is already have finished, do you have an estimation for the release date? Many thanks again.

Is there any way to install the project without cuda and gpu ?

I was trying to install the project using CPU, Pytorch 0.4.0 version . But Getting some error. I think it is because of the absence of CUDA. Is there any way to run the code without CUDA and GPU?? Or is there any way to get recover from these errors??

Error:

Compiling resample2d kernels by nvcc...
rm: cannot remove 'Resample2d_kernel.o': No such file or directory
./make.sh: line 9: nvcc: command not found
In file included from /home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC/THC.h:4:0,
                 from _resample2d.c:493:
/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC/THCGeneral.h:12:18: fatal error: cuda.h: No such file or directory
compilation terminated.
Traceback (most recent call last):
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/unixccompiler.py", line 116, in _compile
    extra_postargs)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/ccompiler.py", line 909, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/spawn.py", line 36, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/spawn.py", line 159, in _spawn_posix
    % (cmd, exit_status))
distutils.errors.DistutilsExecError: command 'gcc' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/site-packages/cffi/ffiplatform.py", line 51, in _build
    dist.run_command('build_ext')
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 338, in run
    self.build_extensions()
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 447, in build_extensions
    self._build_extensions_serial()
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 472, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/command/build_ext.py", line 532, in build_extension
    depends=ext.depends)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/ccompiler.py", line 574, in compile
    self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
  File "/home/mahbubcseju/anaconda3/envs/ML/lib/python3.5/distutils/unixccompiler.py", line 118, in _compile
    raise CompileError(msg)
distutils.errors.CompileError: command 'gcc' failed with exit status 1

Confusion about previous completed frame in the training phase.

How can I get the previous completed frame or just use previous gt frame?

Can‘t run the demo code

Have you test the demo code？I have run python demo_vi.py, it threw the error No module named models. I think that there may be miss a file named __init__.py.

Another problem is that there is not VINet_final in the file vinet.py

questions on double_size and some blurred results

Dear authors,

I'm studying your paper and codes, thanks for sharing!
I have a question that,
as I know from your codes and paper, 0 indicates non-hole pixels and 1 indicates hole pixels.
but why do you need to multiply masks by 0.5 when the input size is 512?
demo_vi.py

Also, I have observed similar results of some cases shown in your paper, however, some cases are very blurred (especially in slow-moving cases)
This is an example from DAVIS (DAVIS/bear)

Is it reasonable? Or did I miss anything to get the results better?

Looking forward to your reply.

Some problem in training

Thank you very much for the given code, but I have a few questions about the details of the training.

After I have rewritten the training code according to the given training demo, I found that the results during the training process were vague and the loss was very unstable.

I used YouTube’s Vos dataset, I don’t know if it’s my implementation problem or what? So I look forward to the author can tell me more implementation details！！

Thanks！

Loss function ssim is run in the whole picture or just the masked area?

Could you please upload the training datasets and testing dataset?

Thanks for your code. It's a fantastic work for video inpainting. However, datasets about the training and testing is hard to obtain by reading your paper. Therefore, could you please upload these related code or update these datasets? Thanks a lot.

Not an issue - question around inverse pixels

in the example the boy on bike is riding and is then painted out -
is it possible to get only those pixels / painting out of the bike?
The parts that are ripped out / rotoscoped per frame / json coordinates?

what the cuda version of your test env?

python3.6 and pytorch 0.4.0,but i still get this error when running the demo. I have installed the dependencies.

Custom video

hi, thanks for such a great work. curious to run on custom video but got stuck.

1. can i run it on custom video
2. do i need to segment each frame by hand
3. how to provide path of custom video

Thanks in advance

Confusion about Feature flow learning

I am confused about Feature flow learning in section 3.2.1. You have written that ' The explicit flow supervision is only given at the finest scale (i.e. 1) and only between the consecutive two frames', but in the section 3.3, you defined the flow loss combine all flows that from other frame to current frame. Could you help me to understand this diff?

I have a problem with install.sh

Is there any other cuda+pytorch version to choose from, my GPU is 3090 and cannot support cudatoolkit=10.0

Multiplication factor of 5 in the loss function of train_git.py

I need a clarification regarding the following:
In line 96 of train_git.py, which calculates HOLE_loss, the L1 loss is multiplied by 5 before accumulating in HOLE_loss variable.
Is this because there are 5 input frames?

Line 96, for reference

HOLE_loss += 5*criterion_L1(
            frame_o1*frame_m[0][:,:,midx,:,:].expand_as(frame_o1), 
            frame_i[0]*frame_m[0][:,:,midx,:,:].expand_as(frame_o1))

If yes, if I were to use just 1 past and 1 future frame (3 input frames), can I change that factor to 3?

Source codes of Space-Time Cubic Puzzles

Hi, thanks for your sharing. Do you have plan to release the implementation of "Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles" ?
Thanks !