haofeixu/aanet

[CVPR'20] AANet: Adaptive Aggregation Network for Efficient Stereo Matching

aanet's Issues

代码在CPU上运行

您好，请问这份代码是否有在纯CPU上运行过呢？

why do we can't multiply scale in file model.py when interpoate the pred_disp?

Hi, I have two questions to ask you.
1、In your model.py # line 97, why don't multiply the resize scale as do in refinement module?
Is that mean, we have finished training, This is the last result, so we don't multiply the scale any more when we do interpolate?
2、why don't use the full resolution to construct multi-scale cost volume. (i.e. H, H/2, H/3), is that only for save GPU memory?
Thanks,

match_similarity参数

    # Matching similarity or matching cost
    cost_volume = cost_volume if self.match_similarity else -cost_volume

这个操作没有理解，可以解释一下吗？为什么要加负号？,PSMNet, GANet都不用加，Stereonet源代码也没有加负号

Running time mentioned in Table 3

I noticed that in table3, you mentioned EPE/Time of StereoNet is 1.1 and 0.015.
I also read the original paper of StereoNet, maybe "0.015s" using the fast(16x, single) version and "1.1 "using the best(8x,multi) version. Can you explain it more?

checkpoint_dir

请问checkpoint_dir在哪啊

performance without pseudo-labels

Hi,

Great work and thanks for sharing the code. May I ask if you have the numbers for the performance of your model when trained with only the sparse KITTI ground truth or in other words without the pseudo-labels provided by other network?

Some questions about inference on PSMnet

Because of the version problems, I haven't been able to run your code. I'm trying to read your code to get a better understanding of your work.
I'm a little confused when i try to understand the construction of PSMNet-AA. You use the PSMNetFeature to extract feartures to construct cost volumes. Although you make a volume list, it actually has only one cost volume. So when you use AdaptiveAggregation, default num_scales = 3, how could you use CSA In this case? I think there would be an error when running assert len(self.branches) == len(x), because len(self.branches) = num_scales = 3, but len(x) = 1.
So where did I get it wrong? Looking forward to your reply and you pointing out my mistake.
Extremely grateful！

Failed build deform_conv

Hi, I meet error when compile deform_conv, is there any suggestions to solve it?

➜ deform_conv bash build.sh
running build_ext
/home/vrlab/anaconda3/lib/python3.6/site-packages/torch/utils/cpp_extension.py:305: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
building 'deform_conv_cuda' extension
creating build
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/src
gcc -pthread -B /home/vrlab/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/vrlab/anaconda3/lib/python3.6/site-packages/torch/include -I/home/vrlab/anaconda3/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/vrlab/anaconda3/lib/python3.6/site-packages/torch/include/TH -I/home/vrlab/anaconda3/lib/python3.6/site-packages/torch/include/THC -I:/usr/local/cuda-10.1/include -I/home/vrlab/anaconda3/include/python3.6m -c src/deform_conv_cuda.cpp -o build/temp.linux-x86_64-3.6/src/deform_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
src/deform_conv_cuda.cpp: In function ‘void shape_check(at::Tensor, at::Tensor, at::Tensor*, at::Tensor, int, int, int, int, int, int, int, int, int, int)’:
src/deform_conv_cuda.cpp:66:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK(weight.ndimension() == 4,
^~~~~~~~
src/deform_conv_cuda.cpp:66:3: note: suggested alternative: ‘DCHECK’
AT_CHECK(weight.ndimension() == 4,
^~~~~~~~
DCHECK
src/deform_conv_cuda.cpp: In function ‘int deform_conv_forward_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
src/deform_conv_cuda.cpp:194:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
^~~~~~~~
src/deform_conv_cuda.cpp:194:3: note: suggested alternative: ‘DCHECK’
AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
^~~~~~~~
DCHECK
src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_input_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
src/deform_conv_cuda.cpp:301:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK((offset.size(0) == batchSize), 3, "invalid batch size of offset");
^~~~~~~~
src/deform_conv_cuda.cpp:301:3: note: suggested alternative: ‘DCHECK’
AT_CHECK((offset.size(0) == batchSize), 3, "invalid batch size of offset");
^~~~~~~~
DCHECK
src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_parameters_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, float, int)’:
src/deform_conv_cuda.cpp:417:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
^~~~~~~~
src/deform_conv_cuda.cpp:417:3: note: suggested alternative: ‘DCHECK’
AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
^~~~~~~~
DCHECK
src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_forward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
src/deform_conv_cuda.cpp:497:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
^~~~~~~~
src/deform_conv_cuda.cpp:497:3: note: suggested alternative: ‘DCHECK’
AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
^~~~~~~~
DCHECK
src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_backward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
src/deform_conv_cuda.cpp:579:3: error: ‘AT_CHECK’ was not declared in this scope
AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
^~~~~~~~
src/deform_conv_cuda.cpp:579:3: note: suggested alternative: ‘DCHECK’
AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
^~~~~~~~
DCHECK
error: command 'gcc' failed with exit status 1

ImportError: cannot import name 'deform_conv_cuda'

I am working on anaconda (in Windows), I have followed the mentioned steps for installation, But I am getting this error.

from . import deform_conv_cuda
ImportError: cannot import name 'deform_conv_cuda'

Please help.

About Table 4 in your paper

dear authors:
when i run your script aanet_train.sh on KITTI 2015 ,i got the disparity images onKITTI_2015_testdatasets.that's cool!
above all,i want to compare with the gt images(like table 4 in your paper) and analyze the error,how can i do that?can you help me?
tks!
@haofeixu @Ilya-Muromets @trigal

FLOPs for networks

This issue is related to #27, In case you didn't notice the question in it, I propose a new one.
I have set 1x3x576x960 as input and only take conv layers(include both 2d and 3d conv and transpose conv) into account, and get 1.08T FLOPs for PSMNet using pytorch-OpCounter. Is there anything wrong in my setting? Could you please provide the FLOPs detailed information? for example the custom ops? Any reply will be appreciated!
Thanks!

Random Crop for training on custom datasets

The paper mentions about using random crops for the scene flow and KITTI datasets, but how to select a crop ratio for training on own datasets? I tried a couple of different crops but all of those gave me a dimension mismatch error in one of the layers also Why is the KITTI original image size different from the size mentioned in the script? Thanks

The problem of cuda?

The readme writes the cuda version is 10.0 that means the cudatoolkit in pytorch or the cuda version in /usr/local/cuda，i think the build work is by the /usr/local/cuda. i meet the segmentation fault, and not sure the problem comes from cuda version. The cuda version in my path is cuda9.2 and the pytorch environment follow the environment.yaml.

Question about D1 metric

Hi, thank you for your nice work.
I have a question about the calculation of the d1 metric in your code.
err_mask = (e > 3) & (e / d_gt > 0.05) (line 23 in metric.py)

In kitti benchmark, it says that a correctly estimated pixel should have the disparity error less than 3px or <5%.

But it seems that you are using 'and' instead of 'or'.

So I'm a littile confused, could you explain this?

Bad prediction results when use pretrained model

When I try to run the predict file using the pictures I took, I get very poor prediction results. Is there something wrong with me?

No such file or directory: 'checkpoints/aanet_kittimix/aanet_latest.pth

dear author:
i have a trouble when i run the scripts/aanet_train.sh,can you tell me some advise
the problem is:
[2020-11-18 13:53:43,988] => Loading pretrained AANet: checkpoints/aanet_kittimix/aanet_latest.pth
Traceback (most recent call last): File "train.py", line 243, in <module> main() File "train.py", line 166, in main utils.load_pretrained_net(aanet, args.pretrained_aanet, no_strict=(not args.strict)) File "/home/liyanzhou/pycharm_project/demo/aanet-master/utils/utils.py", line 77, in load_pretrained_net state = torch.load(pretrained_path, map_location='cuda') File "/home/liyanzhou/anaconda3/envs/python37/lib/python3.7/site-packages/torch/serialization.py", line 381, in load f = open(f, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'checkpoints/aanet_kittimix/aanet_latest.pth'

tks

the performace without deformable conv?

can you provide the performace without deformable conv?

About the warp direction

Hello @haofeixu . Thank you for the paper and the code.
In your code https://github.com/haofeixu/aanet/blob/master/nets/warp.py line 55
offset = torch.cat((-disp, torch.zeros_like(disp)), dim=1) # [B, 2, H, W]
Here you warp the right img to match the left, so the right one should do right shift operation. So I think here should be disp, not -disp. Is it right?

Question about time profiling

Thank you for your great work! I just tried your code and add --count_time to try to see the speed of the model. However, I found there is no torch.cuda.synchronize() after the running of the model. Since pytorch runs in an async way, I wonder whether this will affect the final measuring result.

I just tried to run aanet+ on a 2060s card, the speed is 68ms without synchronization, while the speed is 123ms after adding cuda synchronization.

The code is modified as follows in inference.py.

        with torch.no_grad():
        torch.cuda.synchronize()
        time_start = time.perf_counter()
        print(i, left.shape, right.shape)
        pred_disp = aanet(left, right)[-1]  # [B, H, W]
        torch.cuda.synchronize()
        inference_time += time.perf_counter() - time_start

the pre training model

Why can't I download the pre training model

Test results on Middlebury Stereo Evaluation - Version 3

Hi

I hope everything is OK. Recently I tested AANet on Middlebury Stereo Evaluation - Version 3, but I got very bad results. What could be the reason? I used every weight in pretrained model but none of them give the good results. Do I need to train AANet on Middlebury Stereo Evaluation - Version 3?

Thank you very much.

high err

I train your network without deformable conv and refinement.
The train err on sceneflow is high, about 2.0 epe.
Does it normal?

slightly confused about where the speed up came from

Hi!

After reading your paper, I am not very clear how the speed up came from.

From my understanding, traditional cost volume approach build cost volume of size BXCXDXHXW,
but in your approach you compute the dot product between two features, which results in BXDXHXW .

Is this where the speed up came from?

Thank you for your reply!

error in running aanet_predict.sh

after dubug, i find the problem comes from deconv,but no error happen in building
my Cuda version:

pytorch version:
pytorch1.2.0
cuda Driver version:

the file exists

Disparity Refinement

Thanks for your brilliant work. May I ask the performance of your AANet or AANet+ after removing the refinement module. Thanks.
By the way, I know there is no implementation of 3D deformable convolution, have you thinking about implementing it and releasing 3D ISA and CSA?

Failed to import deform_conv_cuda

Hi, @U @haofeixu ,
I meet the following trouble when I run your model.

Traceback (most recent call last):
  File "inf_time.py", line 8, in <module>
    import nets
  File "/home/lthpc/PRO/aanet-master/nets/__init__.py", line 1, in <module>
    from .aanet import AANet
  File "/home/lthpc/PRO/aanet-master/nets/aanet.py", line 4, in <module>
    from nets.feature import (StereoNetFeature, PSMNetFeature, GANetFeature, GCNetFeature,
  File "/home/lthpc/PRO/aanet-master/nets/feature.py", line 5, in <module>
    from nets.deform import DeformConv2d
  File "/home/lthpc/PRO/aanet-master/nets/deform.py", line 3, in <module>
    from nets.deform_conv import DeformConv, ModulatedDeformConv
  File "/home/lthpc/PRO/aanet-master/nets/deform_conv/__init__.py", line 1, in <module>
    from .deform_conv import (DeformConv, DeformConvPack, ModulatedDeformConv,
  File "/home/lthpc/PRO/aanet-master/nets/deform_conv/deform_conv.py", line 9, in <module>
    from . import deform_conv_cuda
ImportError: /home/lthpc/PRO/aanet-master/nets/deform_conv/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c105ErrorC1ENS_14SourceLocationERKSs

The building of your deform conv is finished, during which I have the following output:

running build_ext
building 'deform_conv_cuda' extension
creating build
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/src
gcc -pthread -B /opt/anaconda3/envs/AutoML-new_master2/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/TH -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.0/include -I/opt/anaconda3/envs/AutoML-new_master2/include/python3.7m -c src/deform_conv_cuda.cpp -o build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/core/Device.h:5:0,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/core/Allocator.h:6,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/ATen/ATen.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:4,
                 from /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
                 from src/deform_conv_cuda.cpp:4:
src/deform_conv_cuda.cpp: In function ‘void shape_check(at::Tensor, at::Tensor, at::Tensor*, at::Tensor, int, int, int, int, int, int, int, int, int, int)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:66:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.ndimension() == 4,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:66:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.ndimension() == 4,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:71:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:71:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:73:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(kW > 0 && kH > 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:73:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(kW > 0 && kH > 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:77:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((weight.size(2) == kH && weight.size(3) == kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:77:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((weight.size(2) == kH && weight.size(3) == kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:82:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(dW > 0 && dH > 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:82:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(dW > 0 && dH > 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:85:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:85:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:101:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(ndim == 3 || ndim == 4, "3D or 4D input tensor expected but got: %s",
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:101:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(ndim == 3 || ndim == 4, "3D or 4D input tensor expected but got: %s",
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:113:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(nInputPlane % deformable_group == 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:113:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(nInputPlane % deformable_group == 0,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:123:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.size(1) == nInputPlane,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:123:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.size(1) == nInputPlane,
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:127:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((inputHeight >= kH && inputWidth >= kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:127:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((inputHeight >= kH && inputWidth >= kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:130:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(2) == outputHeight && offset.size(3) == outputWidth),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:130:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(2) == outputHeight && offset.size(3) == outputWidth),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:135:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(1) == deformable_group * 2 * kH * kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:135:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(1) == deformable_group * 2 * kH * kW),
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:139:5: note: in expansion of macro ‘AT_CHECK’
     AT_CHECK(gradOutput->size(dimf) == nOutputPlane,
     ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:139:5: note: in expansion of macro ‘AT_CHECK’
     AT_CHECK(gradOutput->size(dimf) == nOutputPlane,
     ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:143:5: note: in expansion of macro ‘AT_CHECK’
     AT_CHECK((gradOutput->size(dimh) == outputHeight &&
     ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:143:5: note: in expansion of macro ‘AT_CHECK’
     AT_CHECK((gradOutput->size(dimh) == outputHeight &&
     ^
src/deform_conv_cuda.cpp: In function ‘int deform_conv_forward_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:194:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:194:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
   ^
src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_input_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:301:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), 3, "invalid batch size of offset");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:301:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), 3, "invalid batch size of offset");
   ^
src/deform_conv_cuda.cpp: In function ‘int deform_conv_backward_parameters_cuda(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, float, int)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:417:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:417:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK((offset.size(0) == batchSize), "invalid batch size of offset");
   ^
src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_forward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:497:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:497:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:498:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:498:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
src/deform_conv_cuda.cpp: In function ‘void modulated_deform_conv_cuda_backward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, bool)’:
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:579:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:579:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(input.is_contiguous(), "input tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:20: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                    ^
src/deform_conv_cuda.cpp:580:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:339:40: warning: ‘void c10::detail::deprecated_AT_CHECK()’ is deprecated (declared at /opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/c10/util/Exception.h:314) [-Wdeprecated-declarations]
     ::c10::detail::deprecated_AT_CHECK();                 \
                                        ^
src/deform_conv_cuda.cpp:580:3: note: in expansion of macro ‘AT_CHECK’
   AT_CHECK(weight.is_contiguous(), "weight tensor has to be contiguous");
   ^
/usr/local/cuda-10.0/bin/nvcc -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/TH -I/opt/anaconda3/envs/AutoML-new_master2/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.0/include -I/opt/anaconda3/envs/AutoML-new_master2/include/python3.7m -c src/deform_conv_cuda_kernel.cu -o build/temp.linux-x86_64-3.7/src/deform_conv_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=1 -std=c++11
creating build/lib.linux-x86_64-3.7
g++ -pthread -shared -B /opt/anaconda3/envs/AutoML-new_master2/compiler_compat -L/opt/anaconda3/envs/AutoML-new_master2/lib -Wl,-rpath=/opt/anaconda3/envs/AutoML-new_master2/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/src/deform_conv_cuda.o build/temp.linux-x86_64-3.7/src/deform_conv_cuda_kernel.o -L/usr/local/cuda-10.0/lib64 -lcudart -o build/lib.linux-x86_64-3.7/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so
copying build/lib.linux-x86_64-3.7/deform_conv_cuda.cpython-37m-x86_64-linux-gnu.so -> 

Problems about disparity calculation

Hi, thanks to your code and paper. I just have a question about disparity before loss calculation. In paper you said you first upsampled the disparity to the original resolution, then you use it for training loss calculation. So you first calculation disparity on three different resolution and then upsample them to the original resolution which is the same size as reference image(left image). Am I right? If I'm right, I just wonder if there is any wrong with these implementation?Another question is how did you decide the number of batch_size, because is really large.

Problems with the prediction results using your pretrained model

Quesion about the speed of AANet+

Hello, I am little curious about why the AANet+ have more training parameters than AANet (8.4M vs. 3.9M) but faster speed, it's a little weird.

How to use psmnet to make a prediction

Hi Hello:

I hope everything is OK. I am tring to use the pretrained model of psmnet to make a prediction. But I got some error as shown blow:

Traceback (most recent call last):
File "predict.py", line 199, in
main()
File "predict.py", line 162, in main
pred_disp = aanet(left, right)[-1] # [B, H, W]
File "/usr/local/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/usr/local/envs/aanet/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/usr/local/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/AANet/nets/aanet.py", line 207, in forward
left_feature = self.feature_extraction(left_img)
File "/content/drive/My Drive/AANet/nets/aanet.py", line 136, in feature_extraction
feature = self.fpn(feature)
File "/usr/local/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/content/drive/My Drive/AANet/nets/feature.py", line 213, in forward
assert len(self.in_channels) == len(inputs)
AssertionError

Could you give some advises? Any advises is appriciate!

Error with predict.py

(aanet) permanent@permanent-G3-3579:~/aanet-master$ CUDA_VISIBLE_DEVICES=0 python predict.py --data_dir demo --pretrained_aanet pretrained/aanet+_kitti15-2075aea1.pth
=> Loading pretrained AANet: pretrained/aanet+_kitti15-2075aea1.pth
=> 1 samples found in the data dir
=> Inferencing 0/1
Traceback (most recent call last):
File "predict.py", line 195, in
main()
File "predict.py", line 161, in main
pred_disp = aanet(left, right)[-1] # [B, H, W]
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/aanet-master/nets/aanet.py", line 207, in forward
left_feature = self.feature_extraction(left_img)
File "/home/permanent/aanet-master/nets/aanet.py", line 134, in feature_extraction
feature = self.feature_extractor(img)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/aanet-master/nets/resnet.py", line 181, in forward
layer3 = self.layer3(layer2)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/container.py", line 100, in forward
input = module(input)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/aanet-master/nets/deform.py", line 121, in forward
out = self.conv2(out)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/aanet-master/nets/deform.py", line 84, in forward
out = self.deform_conv(x, offset, mask)
File "/home/permanent/anaconda3/envs/aanet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/permanent/aanet-master/nets/deform_conv/deform_conv.py", line 351, in forward
self.groups, self.deformable_groups)
File "/home/permanent/aanet-master/nets/deform_conv/deform_conv.py", line 136, in forward
raise NotImplementedError
NotImplementedError

this is the error information
please help me

how to convert this to onnx

为了方便表述,我就用中文了.
我想把模型转成onnx格式的.我在inference.py中添加代码:

        with torch.no_grad():
            time_start = time.perf_counter()

            if args.export_onnx == "true":
                # 生成onnx格式的model
                print('*************export onnx model*************')
                print('left{},right{}'.format(left.shape,right.shape))
                torch.onnx.export(aanet,(left,right),"aanet.onnx",input_names=['left','right'],verbose=True)
                break
            else:
                print('left{},right{}'.format(left.shape,right.shape))
                pred_disp = aanet(left, right)[-1]  # [B, H, W]
                break 

但是输出如下:

*************export onnx model*************
lefttorch.Size([1, 3, 384, 1248]),righttorch.Size([1, 3, 384, 1248])
*************aanet forward begin************
/source/aanet/nets/deform_conv/deform_conv.py:147: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  ctx.groups, ctx.deformable_groups, ctx.with_bias)
/source/aanet/nets/deform_conv/deform_conv.py:147: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator zero_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  ctx.groups, ctx.deformable_groups, ctx.with_bias)
/source/aanet/nets/deform_conv/deform_conv.py:147: TracerWarning: There are 7 live references to the data region being modified when tracing in-place operator addmm_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  ctx.groups, ctx.deformable_groups, ctx.with_bias)
/source/aanet/nets/deform_conv/deform_conv.py:147: TracerWarning: There are 10 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  ctx.groups, ctx.deformable_groups, ctx.with_bias)
/source/aanet/nets/cost.py:48: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  cost_volume[:, i, :, :] = (left_feature * right_feature).mean(dim=1)
/source/aanet/nets/cost.py:46: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  right_feature[:, :, :, :-i]).mean(dim=1)
/source/aanet/nets/aggregation.py:394: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if exchange.size()[2:] != x_fused[i].size()[2:]:
/source/aanet/nets/estimation.py:21: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if cost_volume.size(1) == self.max_disp:
/source/aanet/nets/refinement.py:84: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if scale_factor == 1.0:
/source/aanet/nets/warp.py:51: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert disp.min() >= 0
/source/aanet/nets/warp.py:56: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  print('grid:{},offset:{}'.format(grid.shape,offset.shape))
grid:torch.Size([1, 2, 192, 624]),offset:torch.Size([1, 2, 128, 416])

报错的deform_conv.py:147如下:

        deform_conv_cuda.modulated_deform_conv_cuda_forward(
            input, weight, bias, ctx._bufs[0], offset, mask, output,
            ctx._bufs[1], weight.shape[2], weight.shape[3], ctx.stride,
            ctx.stride, ctx.padding, ctx.padding, ctx.dilation, ctx.dilation,
            ctx.groups, ctx.deformable_groups, ctx.with_bias)

tracer在记录模型的前向推理的过程里,由于调用了deform_conv_cuda.so的modulated_deform_conv_cuda_forward,导致没有正确记录所有的算子,后面导致grid和offset的维度不一致. 请问这个要怎么修改呢?

point cloud from disparity image

Hi @haofeixu. Thank you for the paper and the code.
I tested the code on my stereo image and the disparity image looked nice.
However, when I tried to create point cloud from the disparity (with the calib param) the point cloud looked kinda weird.
Do I need to scale disparity image estimated by aanet before using it to form the point cloud. As in OpenCV, the disparity image needs to be downscale by 16.0.

Here is the disparity estimated by aanet

But the point cloud is like this :(

About the Nvidia Hardware

I want to know the device you used is V100 16G or 32g?

the KITTI epe results

Hi, haofei
I want to know how to get the KITTI2015 epe results of Table1 in your paper? Just execute the second finetune step(whole set or just 160 images?) in 4.2 section or execute the two steps(the mix training set and separate training sets)? Thank you so much!

Are the deformable convs customised?

Hi, thank you for the great work and for open sourcing your code.

I wonder whether the (modulated) deformable convolutions used here have been customised / modified in anyway and whether they are any different -- in theory -- from the standard off-the-shelf implementations (e.g., the ones found in mmdetection and maskrcnn_benchmark)?

FLOPs 是如何得出的

请问下你们的各个模型的FLOPs 是如何得出的？
是只计算了CONV中的MAC次数，还是计算量CONV、BN、activate function、pooling等所有的

how to fine-tuned model to predict the disparity map, no additional training is done on Middlebury

how to fine-tuned model to predict the disparity map, no additional training is done on Middlebury ?
how to show image have color like below?

About the result of StereoNet in the paper

In your paper, the time of stereonet is 0.015s. It is really fast. Can you share your implementation? I tried to reimplement StereoNet last year but failed to get a speed performance like this. Thank you so much!

Memory consumption measurement

Hi, @U@haofeixu ,

In table.2 of your paper, there is a list of comparisons with four representative stereo models. Your method saves a lot of memory when equipped to other models. I'm confused about the number of memory consumption because mine is kind of different from yours. Could please tell me how you measure the memory consumption since I guess different measurement may cause different results.

How to get the FLOPs of network

Hi!
Thanks for sharing such an excellent work.
I am curious about the FLOPs of different networks, Which libs or tools did you use for FLOPs calculation?
Is the FLOPs calculation code accessible in this repo?

where is your paper

can't find your paper, would you give me a link?

Number of outputs

Hi, in your paper you mentioned three different scales are used (1/3, 1/6, 1/12). In term of loss function, you first bilinearly upsample your outputs and use the ground truth labels as well as the pseudo round truth labels together with the outputs to compute the loss of each resolution. Balancing term, lambda is also included. But at the end of section 4.2, you mentioned lambda1,2,3 = 1.0, lambda4 = 2/3, lambda5 = 1/3.

I am wondering why are there 5 losses if you have only 3 outputs? Maybe I have misinterpreted something in your paper.

How to deal with the disarity dimension when it can't divisible by 4?

"assert (x.size() == rem.size())" when perform predict.py

commond : "python predict.py --data_dir demo --pretrained_aanet pretrained/aanet+_sceneflow-d3e13ef0.pth --feature_type ganet --feature_pyramid --refinement_type hourglass --no_intermediate_supervision"

error info:

AssertionError: Caught AssertionError in replica 0 on device 0.
Original Traceback (most recent call last):
File "/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
output = module(*input, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/aanet/nets/aanet.py", line 213, in forward
disparity_pyramid[-1])
File "/aanet/nets/aanet.py", line 198, in disparity_refinement
disparity = self.refinementi
File "/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/aanet/nets/refinement.py", line 173, in forward
x = self.deconv4a(x, rem3)
File "/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "/aanet/nets/feature.py", line 369, in forward
assert (x.size() == rem.size())
AssertionError

debug:
(Pdb) p x.shape
torch.Size([1, 96, 66, 82])
(Pdb) p rem.shape
torch.Size([1, 96, 66, 81])
(Pdb)

bash build.sh error

env: python 3.6 torch 1.5 torchvision 0.6

error information ： deform_conv_cuda.cpp:417:3: error: ‘AT_CHECK’ was not declared in this scope

Prediction use pretrained network on CPU

Can I build deformable convolution without cuda and run pretrained network on CPU? Just want to do some tests and currently not have a nvidia card.

When I run predict.py, I got

ImportError: cannot import name 'deform_conv_cuda' from 'nets.deform_conv'

When I build deformable convolution, I got

AssertionError: Torch not compiled with CUDA enabled

Can I use pretrained model to get disparity map on Pascal VOC dataset?

Is it possible using the pretrained model to get disparity map on Pascal VOC dataset?

Is it necessary to build deformable conv?

The new pytorch version has support deformable conv

Correlation in cost volume

aanet/nets/cost.py

Lines 40 to 48 in f27ec42

	elif self.feature_similarity == 'correlation':
	cost_volume = left_feature.new_zeros(b, self.max_disp, h, w)
	for i in range(self.max_disp):
	if i > 0:
	cost_volume[:, i, :, i:] = (left_feature[:, :, :, i:] *
	right_feature[:, :, :, :-i]).mean(dim=1)
	else:
	cost_volume[:, i, :, :] = (left_feature * right_feature).mean(dim=1)

Hi there,

The paper looks great. Thanks for releasing your code! I noticed that in the "correlation" type of cost volume construction, you used the mean of the elementwise product of two feature vectors instead of the traditional dot product, which DispNetC and FlowNet (the original) used. Of course, this is just a scaled version. Can you talk about why you scaled it this way? Thank you!