vt-vl-lab/flownet2.pytorch

Off-the-shelf FlowNet module in PyTorch-0.3.0

flownet2.pytorch's Issues

correlation function runs very slow

Hi, I'm trying to train FlowNet2 separately. It's very slow when I train the flownetC, It's about 5.5s a batch (batch size = 8) on TiTan X, I don't know whether there are some problems, I want to know how fast you train, and could you give some advice?
Thanks for your help!

Issue in compiling install.sh

Hello,
Thank you for providing the code. I was trying to compile the install.sh file but Im getting the following error:-

in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1

I have edited my mask.sh file with the nvcc path as
nvcc -c -o correlation_cuda_kernel.o correlation_cuda_kernel.cu -x cu -Xcompiler -fPIC -arch=sm_52 -I ${TORCH}/utils/ffi
My nvcc --version gives me:-

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176

Could you kindly assist me with this issue? Thanks a lot

failed to extract the flow of a image with arbitrary size, e.g [600, 801]

thanks for you code, but I encountered some problems, when I put 2 image with size [600, 801] to the flownet2C network, it didn't work, the error message is something like this:
hope receive your reply, thanks:->

"
RuntimeError Traceback (most recent call last)
in ()
----> 1 flownet2(tmp)

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
355 result = self._slow_forward(*input, **kwargs)
356 else:
--> 357 result = self.forward(*input, **kwargs)
358 for hook in self._forward_hooks.values():
359 hook_result = hook(self, input, result)

/home/zlq/faster-rcnn.pytorch/FlowNet2_src/models/flownet2.pyc in forward(self, inputs)
74
75 # flownetc
---> 76 flownetc_flow2 = self.flownetc(x)[0]
77 flownetc_flow = self.upsample1(flownetc_flow2 * self.div_flow)
78

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in call(self, *input, **kwargs)
355 result = self._slow_forward(*input, **kwargs)
356 else:
--> 357 result = self.forward(*input, **kwargs)
358 for hook in self._forward_hooks.values():
359 hook_result = hook(self, input, result)

/home/zlq/faster-rcnn.pytorch/FlowNet2_src/models/components/FlowNetC.pyc in forward(self, x)
108 out_deconv5 = self.deconv5(out_conv6)
109
--> 110 concat5 = torch.cat((out_conv5, out_deconv5, flow6_up), 1)
111
112 flow5 = self.predict_flow5(concat5)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 20 and 19 in dimension 2 at /pytorch/torch/lib/THC/generic/THCTensorMath.cu:111
"

How to use flownet-S?

The demo.py give an example of how to use the model flownet2, the flow output works well, I can normalize the flow by dividing the height H and width 'W', but when I replace flownet2 with flownetS, the range of flow output seems doesn't make sense, should I multiply the output by a scalar or something?
maybe 2.5?

torch.utils.ffi is deprecated

I got the error that ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
Maybe use pytorch preversion can fix it,but I want run it at 1.10.1,how may I fix the build.py to cpp_extension?
I see the offical document for https://pytorch.org/docs/stable/cpp_extension.html .
But the original code have the parameters headers,extra_object,....
How to trasform it to cpp_extension?

fine tunning

I used the code from your source on linux but it didnt work. Hopefully, your code worked and the demo was working fine.
Can I use this off the shell version to fine-tune the models using my own input images?
Do they need to be in dataset format?
Can I feed in new images with different size (the size is multiple of 32 so theoretically it should work)?

modifying install.sh for only cpu

Hi,
I am using a laptop with only cpu available. I believe I have to modify the *.sh files as was specified in the repository's README. Problem is that I don't know how to modify it, and the way to do it was not specified. I would really appreaciate your help to get this FlowNet implementation to work on my cpu only device. Thanks a lot for your time.

Error upon running install.sh:

Compiling correlation kernels by nvcc...
rm: cannot remove 'correlation_cuda_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
./make.sh: line 12: nvcc: command not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch.utils.ffi
  File "/home/cangozpi/miniconda3/envs/carla/lib/python3.7/site-packages/torch/utils/ffi/__init__.py", line 1, in <module>
    raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
Compiling resample2d kernels by nvcc...
rm: cannot remove 'Resample2d_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
./make.sh: line 10: nvcc: command not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch.utils.ffi
  File "/home/cangozpi/miniconda3/envs/carla/lib/python3.7/site-packages/torch/utils/ffi/__init__.py", line 1, in <module>
    raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
Compiling channelnorm kernels by nvcc...
rm: cannot remove 'ChannelNorm_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory
./make.sh: line 10: nvcc: command not found
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch.utils.ffi
  File "/home/cangozpi/miniconda3/envs/carla/lib/python3.7/site-packages/torch/utils/ffi/__init__.py", line 1, in <module>
    raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.

free(): invalid pointer

Can anyone give me some pointers on how to debug

$ python3 demo.py
torch.Size([1, 3, 2, 384, 512])
free(): invalid pointer
Aborted (core dumped)

Spatial Size Constraint

Why does the spatial size have to be a multiple of 64?

Trouble propagating a custom gradient through FlowNet2 model

I am trying to back-propagate a custom gradient tensor through the FlowNet2 model. I know that this is possible in PyTorch using the following methodology:

model = Net()
model = torch.load('./mnist_saved_model.pth')
model.eval()
output = model(img)
output.backward(custom_grad)

I am trying to replicate this with FlowNet2. Here is the relevant code snippet:

### Initialize Flownet  model
flownet_saved_model = "/root/checkpoints/FlowNet2_train-checkpoint.pth.tar"
flownet_model = FlowNet2()
pretrained_dict = torch.load(flownet_saved_model)['state_dict']
model_dict = flownet_model.state_dict()
pretrained_dict = {k: v for k, v in pretrained_dict.items() if k in model_dict}
model_dict.update(pretrained_dict)
flownet_model.load_state_dict(model_dict)
flownet_model.cuda()
flownet_model.eval()

...

### Push image pair through flownet in forward pass
curr_ims = torch.from_numpy(curr_ims)
curr_ims_v = Variable(curr_ims.cuda().float(), requires_grad=True)
curr_flownet_out = flownet_model(curr_ims_v)
    
### Inject a custom gradient tensor for this image pair
custom_grad = torch.FloatTensor(np.random.rand(1,2,128,192)).cuda()
curr_flownet_out.backward(custom_grad)
...

However, when I attempt to run this, I encounter an error at line:
curr_flownet_out.backward(custom_grad)

Traceback (most recent call last):
  File "test.py", line 220, in <module>
    curr_flownet_out.backward(custom_grad)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/variable.py", line 167, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/__init__.py", line 99, in backward
    variables, grad_variables, retain_graph)
  File "/usr/local/lib/python2.7/dist-packages/torch/autograd/function.py", line 91, in apply
    return self._forward_cls.backward(self, *args)
  File "/root/pytorch_flownet2/FlowNet2_src/models/components/ops/channelnorm/functions/channelnorm.py", line 25, in backward
    grad_input1.data, ctx.norm_deg)
AttributeError: 'ChannelNormFunctionBackward' object has no attribute 'norm_deg'

Any ideas as to how I can successfully use PyTorch's autograd feature to propagate a custom gradient tensor through FlowNet2?

Thanks!

Does the size of referenced image must be a multiplier of 64？

I run the demo with my own images (size: 1080x1920). Then some problems are encounted

Traceback (most recent call last):
File "demo.py", line 30, in <module> pred_flow = flownet2(ims_v).cpu().data
 File "/data/zwzhou/miniconda2/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/zwzhou/DownloadCode/PYTORCH/pytorch_flownet2/FlowNet2_src/models/flownet2.py", line 76, in forward   flownetc_flow2 = self.flownetc(x)[0]
  File "/data/zwzhou/miniconda2/envs/pytorch/lib/python3.5/site-ckages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/zwzhou/DownloadCode/PYTORCH/pytorch_flownet2/FlowNet2_src/models/components/FlowNetC.py", line 120, in forward  concat3 = torch.cat((out_conv3_1, out_deconv3, flow4_up), 1)
RuntimeError: inconsistent tensor sizes at /opt/conda/conda-bld/pytorch_1512382878663/work/torch/lib/THC/generic/THCTensorMath.cu:157

when I try other image pairs (576x768), it can run without issues.

What should I do to process this images (1080x1920), just pad the images to be a multiplier of 64 with zero?

Zip files of pretrained models damaged

It seems that the zip files of pretrained models have been damaged.
Converted Caffe Pre-trained Models
I downloaded them from google drive, and then I can't unzip them neither using unzipping software, nor the 'tar' command. They all show that the zip files have been broken. Is it the problem of my machine?

warp_image function runs very slow

Hi ! Thanks for your code , It seems that warp_image in flowlib.py run very slow. I wonder if exits other methods to speed it up