sniklaus/sepconv-slomo

an implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

sepconv-slomo's Issues

Request of your train code model.

I'm intersted in your work,but there is no how to train the model.If you can offer it,I'll very thank you.

nn.Upsampling is deprecated. Use nn.functional.interpolate instead.

Hello!I got this prompt when I ran your run.py! Why is this happening?And how can I solve it?

Perceptual loss

Hi, in your paper you mentioned the use of perceptual loss, did you apply any pre-processing to the training set as basically I'm getting a contrast issue despite performing normalisation using the imageNet mean and standard deviation values. I've tried it without doing this, but the output is more or less the same
I'd appreciate it if you had any ideas as to where I'm going wrong in this case
Thanks

NVidia only?

Too sad...

vaule of interpolation

I have noticed that when interpolating an image or video does it x2, is there a way to increase the amount of interpolation for example x4 or x8?

The training videos

Hi, thanks for your sharing. I like your work.
Recently I want to retrain your model, and I download some YouTube videos. But I train the model which can not achieve the PSNR 35.73 dB in the Middlebury. So can you offer your training videos list?
Thanks for you very much!

Need your help to analyze the reason

Hello, I divided a complete video into odd-frame video and even-frame video, using odd frames to interpolate, then I converted the YUV video losslessly into mp4 format and interpolated it, and converted the interpolated mp4 into YUV format to calculate the PSNR result, but this effect is not very rational, I have not yet figured out the reason, What is your opinion?

no useful context

https://github.com/l2009312042/sepconv
i write the train code and backward part ref to meipuru344
thx for sniklaus ,the sepconv work is great

test

Hello! Can I test videos of any resolution based on your network? Because I tested the 4K video display is not very good。
out_0.zip

A question about the Instance normalization

Hello, your latest work "Revisiting Adaptive Convolutions for Video Frame Interpolation" mentions that input normalization can improve the PSNR. So I apply it to my training of softmax-splatting, while it seems to be divergent after several steps. Here is my implemention，is there any problems? Or the instance normalization is not suitable for the frame interpolation based on optical flow estimation, since the distribution of input data has changed after instance normalization. Looking forward to your reply, thanks a lot.

`def instance_norm(I0, I1):
input = torch.cat((I0, I1), dim=2)
mean1 = input.mean(axis=(2,3),keepdim=True)
mean = mean1.expand(input.shape)
std1 = input.std(axis=(2,3),keepdim=True)
std = std1.expand(input.shape)
input_hat = (input-mean)/std
return input_hat[:, :, :int(input_hat.size(2) / 2), :],
input_hat[:, :, int(input_hat.size(2) / 2):, :], mean1, std1

def instance_norm_reverse(result, mean, std):
std1=std.expand(result.shape)
mean1=mean.expand(result.shape)
output = result*std1+mean1
output[output>1.0] = 1.0
output[output<0.0] = 0.0
return output`

Not working on grayscale images

Hi, I used your code in several images and it worked fine. However, when I tried on a gray scale image, I had the following error
Traceback (most recent call last): File "run.py", line 257, in tensorInputFirst = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strFirst))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0) IndexError: too many indices for array
Is there a way your code works for gray scale images.

bash install.bash failed

I am using Windows 10 with an Nvidia GTX 1060 and CUDA 8.
install.bash failed
SeparableConvolution_cuda.obj : error LNK2001: unresolved external symbol state SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol cudaGetLastError SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol cudaConfigureCall SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol cudaSetupArgument SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol cudaLaunch SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __imp_THLog1p SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __imp_THCState_getCurrentStream SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __imp___THCudaCheck SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __imp_THCudaTensor_data SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __imp_THCudaTensor_nElement SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __cudaRegisterFatBinary SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __cudaUnregisterFatBinary SeparableConvolution_kernel.o : error LNK2001: unresolved external symbol __cudaRegisterFunction
Do I do anything wrong or It's not even possible to make it working on windows?

a question about the loss function LF

Hi, I used the reimplemntation of the sepconv-slomo, but I noticed that the loss might be wrong. I guess it should be
loss = torch.nn.MSELoss()(outputFeatures, targetFeatures). Is it right?
(from https://github.com/martkartasev/sepconv/blob/master/src/loss.py):

class VggLoss(nn.Module):

def __init__(self, device=torch.device('cpu')):
    super(VggLoss, self).__init__()

    model = torchvision.models.vgg19(pretrained=True)

    self.features = nn.Sequential(
        # stop at relu4_4 (-10)
        *list(model.features.children())[:-10]
    )

    for param in self.features.parameters():
        param.requires_grad = False

def forward(self, output, target):

    output_features = self.features(output)
    target_features = self.features(target)

    loss = torch.norm(output_features - target_features, 2)

    return loss

Dose the SeparableConvolution layer need backward ?

Hi! Dose the SeparableConvolution layer need backward ? if need,could you tell me how to define it's backword function

RuntimeError: tried to construct a tensor from a nested float sequence, but found an item of type numpy.float32

I am using Ubuntu 17.04 with an Nvidia GTX 980 and CUDA 9
When I try to run the run.py script I get the following error:

alex@alex-desktop:~/pytorch-sepconv$ python run.py --model lf --first ./images/first.png --second ./images/second.png --out ./result.png
Traceback (most recent call last):
  File "run.py", line 167, in <module>
    tensorInputFirst = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strFirst))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0)
RuntimeError: tried to construct a tensor from a nested float sequence, but found an item of type numpy.float32 at index (0, 0, 0)

error in run.py

Traceback (most recent call last):
File "run.py", line 172, in
moduleNetwork = Network().cuda().eval()
File "run.py", line 124, in init
self.load_state_dict(torch.load('network-' + arguments_strModel + '.pytorch'))
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 358, in load
return _load(f, map_location, pickle_module)
File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 532, in _load
magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, '<'.

When I run the file run.py, the error above occurred. Hope someone can help me.

Train.py?

Hi,

How would someone go about training this network on new images?

Thanks

"ModuleNotFoundError: no module named torch.utils.serialization"

Hello,

I've tried to run the run.py file with my anaconda pytorch setup, however it gives me the:
"ModuleNotFoundError: no module named torch.utils.serialization"
error. I can't seem to find any information on how to fix this or really where that serialization package is from.

load_state_dict_from_url() got an unexpected keyword argument 'file_name'

can you how to deal with this problem？
thank you very much！

No module named '_ext'

Applause to everyone involved in making this, it is amazing.

Anyway this is the first time I'm using python, please be gentle.
When I try to run "run.py" I get this following error:

Traceback (most recent call last):
File "run.py", line 13, in
from SeparableConvolution import SeparableConvolution # the custom SeparableConvolution layer
File "D:\Users-\Desktop\Sepconv\pytorch-sepconv-master\SeparableConvolution.py", line 3, in
import _ext.cunnex
ModuleNotFoundError: No module named '_ext'

Not sure what I'm missing, I'm using python 3.6.

How to generate the Separable kernels?

Hi, thank you for your idea and codes. But I could not understand the Separable kernel. The kernels mentioned in the paper are 1D, but in my opinion, the output of the sub-network should be 51x128x128. I am very confused

One issue about SeparableConvolution.py

Thank you for your great works!
The funtion :SeparableConvolution_cuda_forward(input,vertical,horizontal,output) which can be seen in SeparableConvolution.py. I want to ask if vertical's 51channels just multiply by each column in 51*51 patches of input and horizontal's 51channels multiply by each row of patches ,and then add them to 1D new pixels ?
Looking forward to your reply!

Segmentation Fault

Install.bash ran with error

In file included from /home/john/anaconda3/lib/python3.7/site-packages/torch/lib/include/THC/THCGeneral.h:5:0,
                 from /home/john/anaconda3/lib/python3.7/site-packages/torch/lib/include/THC/THC.h:4,
                 from src/SeparableConvolution_kernel.cu:1:
/home/john/anaconda3/lib/python3.7/site-packages/torch/lib/include/TH/THAllocator.h:6:10: fatal error: ATen/Allocator.h: No such file or directory
 #include <ATen/Allocator.h>
          ^~~~~~~~~~~~~~~~~~

but it still proceeded with downloads. Then, when run run.py, it gives a Segmentation Fault.

CUDA version 10, Pytorch 0.4.1. Ubuntu 18.04

EDIT: works fine on Ubuntu 16.04, Pytorch 0.4.0, CUDA 8.0. Initially faced some errors in install.bash, but changed python install.py to python3 install.py (inside install.bash) and works fine. Also had to add CUDA to PATH for PyTorch to recognize it properly. Will see if it works again on recent versions.

Assertion Error in Video Frame interpolation

Hi, Mr. Niklaus!

Thank you for your great work! When I use your code to interpolate my own video(1280*720, 30FPS,40s), an error occurs as shown in Figure. But I successfully generate the output reslut of car-turn.mp4.

I really appreciate it if anyone can help me .

Thanks in advance!

Parameters

some question about code

Hello, if I want to interpolates one additional frame between two given frames in a video, is the modification code as follows? Because the result of my interpolation is poor, so I want you to help me see if it is the place where I modified，thank you very much ！！
if name == 'main':
if arguments_strOut.split('.')[-1] in [ 'bmp', 'jpg', 'jpeg', 'png' ]:
tensorFirst = torch.FloatTensor(numpy.array(PIL.Image.open(arguments_strFirst))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0))
tensorSecond = torch.FloatTensor(numpy.array(PIL.Image.open(arguments_strSecond))[:, :, ::-1].transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0))

	tensorOutput = estimate(tensorFirst, tensorSecond)

	PIL.Image.fromarray((tensorOutput.clamp(0.0, 1.0).numpy().transpose(1, 2, 0)[:, :, ::-1] * 255.0).astype(numpy.uint8)).save(arguments_strOut)

elif arguments_strOut.split('.')[-1] in [ 'avi', 'mp4', 'webm', 'wmv' ]:
	import moviepy
	import moviepy.editor

	strTempdir = tempfile.gettempdir() + '/' + str.join('', [ random.choice('abcdefghijklmnopqrstuvwxyz0123456789') for intCount in range(20) ]); os.makedirs(strTempdir + '/')

	intFrames = 0
	tensorFrames = [ None, None, None]####modified

	for intFrame, numpyFrame in enumerate(numpyFrame[:, :, ::-1] for numpyFrame in moviepy.editor.VideoFileClip(filename=arguments_strVideo).iter_frames()):
		tensorFrames[2] = torch.FloatTensor(numpyFrame.transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0))####modified

		if tensorFrames[0] is not None:
			tensorFrames[1] = estimate(tensorFrames[0], tensorFrames[2])####modified
			

			PIL.Image.fromarray((tensorFrames[0].clamp(0.0, 1.0).numpy().transpose(1, 2, 0)[:, :, ::-1] * 255.0).astype(numpy.uint8)).save(strTempdir + '/' + str(intFrames).zfill(5) + '.png'); intFrames += 1
			PIL.Image.fromarray((tensorFrames[1].clamp(0.0, 1.0).numpy().transpose(1, 2, 0)[:, :, ::-1] * 255.0).astype(numpy.uint8)).save(strTempdir + '/' + str(intFrames).zfill(5) + '.png'); intFrames += 1
			PIL.Image.fromarray((tensorFrames[2].clamp(0.0, 1.0).numpy().transpose(1, 2, 0)[:, :, ::-1] * 255.0).astype(numpy.uint8)).save(strTempdir + '/' + str(intFrames).zfill(5) + '.png'); intFrames += 1
			
		# end

		tensorFrames[0] = torch.FloatTensor(numpyFrame.transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0))
	# end

	moviepy.editor.ImageSequenceClip(sequence=strTempdir + '/', fps=25).write_videofile(arguments_strOut)

	shutil.rmtree(strTempdir + '/')

# end

end

Out of memory... how to increase memory?

So I got it to work (I guess) but I'm having an issue with larger images. Anything below 720p works fine, but at or above 720p it gives me an out of memory error. Is there a way to extend this? I don't mind if it takes longer to process; or would I have to get a GPU with more VRAM? My current one is a 970gtx with 3.5GB of VRAM.
Or would high VRAM not make any difference?

Error in run.py

When I try to run the python file run.py after using downloading the bash file, I get the following error message:

Traceback (most recent call last):
File "run.py", line 172, in
moduleNetwork = Network().cuda().eval()
File "run.py", line 124, in init
self.load_state_dict(torch.load('./network-' + arguments_strModel + '.pytorch'))
File "/home/souvik207/.local/lib/python2.7/site-packages/torch/serialization.py", line 356, in load
f = open(f, 'rb')
IOError: [Errno 2] No such file or directory: './network-lf--first.pytorch'

What could be causing this?
Is there a way to overcome this error?

in the run .py , the input tensor has been paded twice?

input size is 1,3,720,960, first pad to 1,3,896(720 + 25+ 25 + 126),1024(960 + 25 + 25+ 14)
and then pad to 1,3,946(896 + 252),1074(1024 +252),is that ok?

How to setup on windows?

Hello,

So first off, this is concerning specifically the video supported extension found here:
https://github.com/dagf2101/pytorch-sepconv

There do not seem to be any instructions on how to install it on windows and the lack of information for what libraries are used does not help either. I already have pytorch setup, but I do not know how to proceed further, could you help me? When I open up the install.bash file it mentions nvcc, do I just install that? But then it has these various configuration parameters...
I also have anaconda installed (similiar to pip).

CVPR 2018 paper

Great works! Can't wait for the code of your new CVPR 2018 paper~

Make it work with pytorch installations from wheel packages

I have decided to spare myself the "pleasure" of building pytorch from source (necessary if you want to have the CUDA support on Linux). Thus, I have used the ready made wheel package (see here). The problem is that the pytorch it installs has the following version string: 1.3.1+cu100. This makes your code complain when checking the pytorch version (I link the appropriate line from run.py).

https://github.com/sniklaus/pytorch-sepconv/blob/76bc80fa9584801e6f0a32996253e317a46c1370/run.py#L24

By substituting your code with the one below, the installation with wheel package passes the assertion (and the same is true for standard pytorch installations):
assert(int(str('').join(torch.__version__.split('.')[0:3]).split('+')[0]) >= 41)

I am sorry for not making this a pull request.

problem with compilation

I installed in ubuntu16.04. When I run the bash install.bash , i get the following error.

huwenchao@huwenchao-B350GT3:~/cv/pytorch-sepconv-master$ bash install.bashIn file included from /home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/THC/THCGeneral.h:5:0,
                 from /home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/THC/THC.h:4,
                 from src/SeparableConvolution_kernel.cu:1:
/home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/TH/THAllocator.h:6:28: fatal error: ATen/Allocator.h: No such file or directory
 #include <ATen/Allocator.h> ^
compilation terminated.
gcc: error: /home/huwenchao/cv/pytorch-sepconv-master/src/SeparableConvolution_kernel.o: No such file or directory
Traceback (most recent call last):
  File "install.py", line 32, in <module>
    objectExtension.build()
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 189, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 111, in _build_extension
    outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/api.py", line 697, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/recompiler.py", line 1520, in recompile
    compiler_verbose, debug)
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

I am a novice to this,please help me.

Installing on macOS not possible

Hi

I'm trying to get things to function, but I keep getting errors. I believe this is due to nVidia's cuDNN libraries/primitives not being available for macOS.

Is there a way around this?

Kind regards.

resolution

Hello! When the input video is 'car-turn.mp4' , a frame interpolation sequence is obtained every two frames, and the output is the interpolated video. Why does the resolution of the video remain 640360 not 1280720, but the number of frames changes from 81 to 320?

some question about softmax-splatting

Hello, what should I do if I want to use your softmax-splatting network to test an N-frame video?

training data set frame rate

Excuse me, what's the frame rate of videos in your training data set, 60fps ?

SyntaxError in run.py

The latest run.py doesn't work on Python 3.7.4.
Is the run.py written for the other Python version?

Cloned Commit:
commit c455bc8
Date: Mon Aug 12 19:04:13 2019 -0700

Error:
$ python ./run.py
File "./run.py", line 199
for intFrame, numpyFrame in enumerate(numpyFrame[:, :, ::-1] for numpyFrame in moviepy.editor.VideoFileClip(filename=arguments_strVideo).iter_frames():

^ (←indicate to ":" at the end of line)
SyntaxError: invalid syntax

Python Version:
Python 3.7.4 (default, Aug 24 2019, 12:36:21)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux

Upsample error?

Hi,
I receive this error when I try running the command 'run.py --model lf --first ./images/first.png --second ./images/second.png --out ./result.png':

File "run.py", line 163, in
moduleNetwork = Network().cuda()
File "run.py", line 88, in init
torch.nn.Upsample(scale_factor=2, mode='bilinear'),
AttributeError: 'module' object has no attribute 'Upsample'

Could this have to do with the PyTorch version?

Thanks!

YUV

Hello! Can we use the format of the video is YUV ?

see_you_again_540.mp4

Could you provide the video "project/videos/see_you_again_540.mp4"?

How to train a model

I want to train a model for anime but I don’t how to train a workable model.How can I do?

Flow calculation

Hi, according to your paper, PWC-Net is used for flow calculation, I have tried to find where exactly this is used in the run.py file, however, I can't seem to find it. Can you tell me where this is used?
Thanks

Request_of_Pre-trained model and the script to train the model.

@sniklaus ,Thanks for your sharing! Your work is very great!
Could you please offer the pre-trained model and the script to train the network?

Request of training dataset

@sniklaus
Thanks for your sharing! The work is quite amazing!
Could you please offer the training dataset?

Thanks a lot!

Loss function in paper = backward() into SeparableConvolution.py ?

Hi !

I'm currently trying to retrain your network only with fast moving sport images.

To make the backpropagation, I need to implement the backward() function.

Do I just need to implement the loss function described in the paper section 3.1.1 ?

The question might seem trivial, but I prefer to ask the question :)

Thanks a lot,
Adrien

Feature reconstruction loss

Hi,

Did you post your code for feature reconstruction loss in this repo? It seems that the loss is very effective and important but I could not find the implementation.

Fangjun

AttributeError: module 'torch._C' has no attribute '_cuda_getDevice'

AttributeError: module 'torch._C' has no attribute '_cuda_getDevice'

I have cuda, but it's not finding it somehow?

pre-trained networks

could you please tell me that where is pre-trained networks?

Output is just black

I installed and run the code successfully on floydhub, but my output photo is just whole balck.
There is no warning message.
Please help!