sniklaus/pytorch-spynet

a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet's Issues

getopt error

066636a introduced an error if run.py is run with
python run.py --model sintel-final --first ./images/first.png --second ./images/second.png --out ./out.flo
taken from current README.md

Is there a training function that goes with this?

Reimplementation in Tensorflow

Thank you for the wonderful work.
Relying on this, our team has also pulled out similar work in tensorflow. The architecture is available at: https://github.com/tukilabs/Video-Compression-Net/blob/master/utils/network.py (class SpyNetwork) and the pre-trained model is available at: https://github.com/tukilabs/Video-Compression-Net/blob/master/checkpoints/spymodel.pkl .

Thanks!

How to open the .flo output file?

Can you provide me your flow rendering code?

I write a simple flow-rendering function based on opencv tutorial. The result looks like this:

It looks the same but the color palette is ugly to me. Can you share your flow rendering code with me so I can generate visualized results just like on your github page?

Thanks!

pre_trained model

wget --timestamping http://content.sniklaus.com/github/pytorch-spynet/network-chairs-clean.pytorch
wget --timestamping http://content.sniklaus.com/github/pytorch-spynet/network-chairs-final.pytorch

What is the difference between the pre_train models ?

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Hi! I meet this problem when run run.py.
my environment: CUDA 8 + cudnn 7.1+python 3.7. Thank you~
Details as following:
Traceback (most recent call last):
File "/media/data2/xly/net_code/pytorch-liteflownet-master/run.py", line 377, in
tensorOutput = estimate(tensorFirst, tensorSecond)
File "/media/data2/xly/net_code/pytorch-liteflownet-master/run.py", line 363, in estimate
tensorFlow = torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)
File "/home/xly/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/data2/xly/net_code/pytorch-liteflownet-master/run.py", line 332, in forward
tensorFlow = self.moduleSubpixel[intLevel](tensorFirst[intLevel], tensorSecond[intLevel], tensorFeaturesFirst[intLevel], tensorFeaturesSecond[intLevel], tensorFlow)
File "/home/xly/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/data2/xly/net_code/pytorch-liteflownet-master/run.py", line 226, in forward
tensorFeaturesSecond = Backward(tensorInput=tensorFeaturesSecond, tensorFlow=tensorFlow * self.dblBackward)
File "/media/data2/xly/net_code/pytorch-liteflownet-master/run.py", line 61, in Backward
return torch.nn.functional.grid_sample(input=tensorInput, grid=(Backward_tensorGrid[str(tensorFlow.size())] + tensorFlow).permute(0, 2, 3, 1), mode='bilinear', padding_mode='zeros')
File "/home/xly/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2597, in grid_sample
return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

error in demo

Hello! I tried your demo but got this error.
line 151, in forward

variableFlow=torch.autograd.Variable(data=torch.zeros(variableFirst[0].size(0),2,math.floor(variableFirst[0].size(2) / 2.0), math.floor(variableFirst[0].size(3) / 2.0)).cuda(), volatile=not self.training)

TypeError: torch.zeros received an invalid combination of arguments - got (int, int, float, float), but expected one of:

• (int ... size)

I changed to "data=torch.zeros(variableFirst[0].size(0)/2,math.floor....)" but still not work. It seems there's something wrong with the ''size''. I'm new to deep learning, can you give some suggestion?

Thanks!

Hello! How to solve this problem？

Traceback (most recent call last):
File "/home/lhl/pytorch-spynet-master/run.py", line 131, in
moduleNetwork = Network().cuda().eval()
File "/home/lhl/pytorch-spynet-master/run.py", line 100, in init
self.load_state_dict(torch.load('./network-' + arguments_strModel + '.pytorch'))
File "/home/public/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 358, in load
return _load(f, map_location, pickle_module)
File "/home/public/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 532, in _load
magic_number = pickle_module.load(f)
_pickle.UnpicklingError: invalid load key, '<'.

From first image to the second

Hi, this code is really helpful. I'm wondering how can I predict the second image from the first one + flow.

I tried

second_image_pred = backwarp(first_image, flow). But the result has strange color shift and the motion is not modeled well.

Looking forward to your reply. Thanks!

backwarping for multiple GPUs

Hi, this is an amazing work and saved me from alot of work. I just want to ask that why you have written backwarping outside the Network class while you are also defining preprocess and basic inside the class. is there any theoretical background which I am missing right now?

Note. Actually I am trying to use your code to run on multiple GPUS and this function is creating a problem.

thanks.

return torch.nn.functional.grid_sample(input=tensorInput, grid=(Backward_tensorGrid[str(tensorFlow.size())] + tensorFlow).permute(0, 2, 3, 1),\
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

PIL and pytorch are in conflict

I am using windows and anaconda. I installed pytorch using conda but when I was going to install PIL, I got an error that says pytorch and PIL dependencies are in conflict.
How should I install both to run the code

Could you please provide the training scripts?

The current repo only includes the inference scripts, so could we get the training thing?
I am interested in the implementation of the separate and independent training process of each residual flow level.

Weights for the network in pytorch.

It would be nice to have the network weights in pytorch, to completely remove the torch dependencies. The licensing of spynet allows it for non-commercial purposes such as this.

backwarp function

I was comparing the backwarp function between pytorch-spynet and pytorch-pwc and they are different. Does it matter which one is used? Furthermore, in the model I am developing, I am going to need to warp the image according to the optical flow, do we just use either one of the 2 backwarp functions used in the 2 models. So for example if I used pytorch-spynet then I would use the backwarp of pytorch-spynet, and vice versa for pytorch-pwc?

Error when running the command

I tried to run the command for a test with the 2 pictures provided (I installed pytorch on Anaconda) and I got the following error:

Traceback (most recent call last):
  File "run.py", line 166, in <module>
    moduleNetwork = Network().cuda()
  File "run.py", line 133, in __init__
    self.moduleBasic = torch.nn.ModuleList([ Basic(intLevel) for intLevel in range(6) ])
  File "run.py", line 133, in <listcomp>
    self.moduleBasic = torch.nn.ModuleList([ Basic(intLevel) for intLevel in range(6) ])
  File "run.py", line 100, in __init__
    self.moduleBasic[intConv * 2].weight.data.copy_(torch.utils.serialization.load_lua('./models/modelL' + str(intLevel + 1) + '_' + arguments_strModel  + '-' + str(intConv + 1) + '-weight.t7'))
RuntimeError: The expanded size of the tensor (7) must match the existing size (0) at non-singleton dimension 3

Do you know what could be the reason for this ? thank you !!

Question about code

Thank you for sharing your code.
I have a question with this.

What the list ([ 80, 73, 69, 72 ]) mean?
https://github.com/sniklaus/pytorch-liteflownet/blob/dae3b132452b05d4258d1da75eaa1021c5b2d720/run.py#L381

Thanks in advance

Training Architecture

How the every basic-module(a 5 layers convolution network) trains? I don't find your train part in your files.

How to train spynet using pytorch?

How to train spynet using PyTorch?

This project only has test code for spynet, could you release train spynet code?
Thank you very much!

TypeError: grid_sample() got an unexpected keyword argument 'align_corners'

It was triggered when running to

tensorFlow = torch.nn.functional.interpolate(input=moduleNetwork(tensorPreprocessedFirst, tensorPreprocessedSecond), size=(intHeight, intWidth), mode='bilinear', align_corners=False)

How can l solve this problem? Is that a cause of low pytorch version?
Looking forward to your reply.

Invalid arguments in run.py in line 151.

I'm getting the following error when I run:
python run.py --model sintel-final --first ./images/first.png --second ./images/second.png --out ./out.flo

Traceback (most recent call last):
File "run.py", line 210, in
tensorOutput = estimate(tensorInputFirst, tensorInputSecond)
File "run.py", line 192, in estimate
tensorOutput.resize_(2, intHeight, intWidth).copy_(moduleNetwork(variableInputFirst, variableInputSecond).data[0])
File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 357, in call
result = self.forward(*input, **kwargs)
File "run.py", line 151, in forward
variableFlow = torch.autograd.Variable(data=torch.zeros(variableFirst[0].size(0), 2, math.floor(variableFirst[0].size(2) / 2.0), math.floor(variableFirst[0].size(3) / 2.0)).cuda(), volatile=not self.training)
TypeError: torch.zeros received an invalid combination of arguments - got (int, int, float, float), but expected one of:

• (int ... size)
  didn't match because some of the arguments have invalid types: (int, int, float, float)
• (torch.Size size)

I was able to fix it by casting both math.floor calls to int.