ignacio-rocco/ncnet

PyTorch code for Neighbourhood Consensus Networks

Neighbourhood Consensus Networks

About

This is the implementation of the paper "Neighbourhood Consensus Networks" by I. Rocco, M. Cimpoi, R. Arandjelović, A. Torii, T. Pajdla and J. Sivic.

For more information check out the project [website] and the paper on [arXiv].

Getting started

Dependencies

The code is implemented using Python 3 and PyTorch 0.3. All dependencies should be included in the standard Anaconda distribution.

Getting the datasets

The PF-Pascal dataset can be downloaded and unzipped by browsing to the datasets/pf-pascal/ folder and running download.sh.

The IVD dataset (used for training for the InLoc benchmark) can be downloaded by browsing to the datasets/ivd/ folder and first running make_dirs.sh and then download.sh.

The InLoc dataset (used for evaluation) an be downloaded by browsing to the datasets/inloc/ folder and running download.sh.

Getting the trained models

The trained models trained on PF-Pascal (ncnet_pfpascal.pth.tar) and IVD (ncnet_ivd.pth.tar) can be dowloaded by browsing to the trained_models/ folder and running download.sh.

Keypoint transfer demo

The demo Jupyter notebook file point_transfer_demo.py illustrates how to evaluate the model and use it for keypoint transfer on the PF-Pascal dataset. For this, previously download the PF-Pascal dataset and trained model as indicated above.

Training

To train a model, run train.py with the desired model architecture and the path to the training dataset.

Eg. For PF-Pascal:

python train.py --ncons_kernel_sizes 5 5 5 --ncons_channels 16 16 1 --dataset_image_path datasets/pf-pascal --dataset_csv_path datasets/pf-pascal/image_pairs/ 

Eg. For InLoc:

python train.py --ncons_kernel_sizes 3 3 --ncons_channels 16 1 --dataset_image_path datasets/ivd --dataset_csv_path datasets/ivd/image_pairs/ 

Evaluation

Evaluation for PF-Pascal is implemented in the eval_pf_pascal.py file. You can run the evaluation in the following way:

python eval_pf_pascal.py --checkpoint trained_models/[checkpoint name]

Evaluation for InLoc is implemented in the eval_inloc.py file. You can run the evaluation in the following way:

python eval_inloc.py --checkpoint trained_models/[checkpoint name]

This will generate a series of matches files in the matches/ folder that then need to be fed to the InLoc evaluation Matlab code. In order to run the Matlab evaluation, you first need to clone the InLoc demo repo, and download and compile all the required depedencies. Then you can modify the compute_densePE_NCNet.m file provided in this repo to indicate the path of the InLoc demo repo, and the name of the experiment (the particular folder name inside matches/), and run it to perform the evaluation.

BibTeX

If you use this code in your project, please cite our paper:

@InProceedings{Rocco18b,
        author       = "Rocco, I. and Cimpoi, M. and Arandjelovi\'c, R. and Torii, A. and Pajdla, T. and Sivic, J."
        title        = "Neighbourhood Consensus Networks",
        booktitle    = "Proceedings of the 32nd Conference on Neural Information Processing Systems",
        year         = "2018",
        }

ncnet's Issues

Runtime Error (GPU issue) even when cuda is available.

I keep getting this error:

/content/drive/MyDrive/Rohit/Work/ncnet-master
ImMatchNet training script
Namespace(batch_size=16, checkpoint='', dataset_csv_path='datasets/pf-pascal/image_pairs/', dataset_image_path='datasets/pf-pascal', fe_finetune_params=0, image_size=400, lr=0.0005, ncons_channels=[16, 16, 1], ncons_kernel_sizes=[5, 5, 5], num_epochs=5, result_model_dir='trained_models', result_model_fn='checkpoint_adam')
Creating CNN model...
Downloading: "https://download.pytorch.org/models/resnet101-63fe2227.pth" to /root/.cache/torch/hub/checkpoints/resnet101-63fe2227.pth
100% 171M/171M [00:00<00:00, 195MB/s]
Trainable parameters:
1: torch.Size([5, 16, 1, 5, 5, 5])
2: torch.Size([16])
3: torch.Size([5, 16, 16, 5, 5, 5])
4: torch.Size([16])
5: torch.Size([5, 1, 16, 5, 5, 5])
6: torch.Size([1])
using Adam optimizer
Checkpoint name: trained_models/2021-07-22_06:15_checkpoint_adam.pth.tar
Starting training...
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:4044: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
  "Default grid_sample and affine_grid behavior has changed "
Traceback (most recent call last):
  File "train.py", line 281, in <module>
    log_interval=1,
  File "train.py", line 240, in process_epoch
    loss = loss_fn(model, batch)
  File "train.py", line 221, in <lambda>
    loss_fn = lambda model, batch: weak_loss(model, batch, normalization="softmax")
  File "train.py", line 171, in weak_loss
    corr4d = model(batch)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/drive/MyDrive/Rohit/Work/ncnet-master/lib/model.py", line 263, in forward
    feature_A = self.FeatureExtraction(tnf_batch['source_image'])
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/content/drive/MyDrive/Rohit/Work/ncnet-master/lib/model.py", line 84, in forward
    features = self.model(image_batch)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 443, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 440, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same or input should be a MKLDNN tensor and weight is a dense tensor

I have correctly downloaded the data and all pre-requisite files. I get the same error even when I download the trained models and save them in the specified folder. I am using it on Google Colab and the GPU is activated (checked torch.cuda.is_available() ). I had also tried changing use_cuda=True everywhere, but still, I get the same error.

Question about the correspondence calculation

Hi @ignacio-rocco , thanks for the great work! There are some parts of the code about how you recover the pixel position of matches from the correspondences that are not fully clear to me.

So as what I can see from your code, in lib/point_tnf.py

  if delta4d is not None: \# relocalization
        delta_iA,delta_jA,delta_iB,delta_jB = delta4d

        diA=delta_iA.squeeze(0).squeeze(0)[iA.view(-1),jA.view(-1),iB.view(-1),jB.view(-1)]
        djA=delta_jA.squeeze(0).squeeze(0)[iA.view(-1),jA.view(-1),iB.view(-1),jB.view(-1)]        
        diB=delta_iB.squeeze(0).squeeze(0)[iA.view(-1),jA.view(-1),iB.view(-1),jB.view(-1)]
        djB=delta_jB.squeeze(0).squeeze(0)[iA.view(-1),jA.view(-1),iB.view(-1),jB.view(-1)]

        iA=iA*k_size+diA.expand_as(iA)
        jA=jA*k_size+djA.expand_as(jA)
        iB=iB*k_size+diB.expand_as(iB)
        jB=jB*k_size+djB.expand_as(jB)

You first rescale the indices in the lower resolution to the resolution before the 4D maxpooling and add the shift coming from the maxpooling operation. So the method corr_to_matches outputs the location of the correspondences in the 200x100 resolution. And the location is represented by the portion which is pixel/total_pixel_num.

Then in the eval_inloc.py line 179-190,

if k_size>1:
    yA_=yA_*(fs1*k_size-1)/(fs1*k_size)+0.5/(fs1*k_size)
    xA_=xA_*(fs2*k_size-1)/(fs2*k_size)+0.5/(fs2*k_size)
    yB_=yB_*(fs3*k_size-1)/(fs3*k_size)+0.5/(fs3*k_size)
    xB_=xB_*(fs4*k_size-1)/(fs4*k_size)+0.5/(fs4*k_size)    
else:
    yA_=yA_*(fs1-1)/fs1+0.5/fs1
    xA_=xA_*(fs2-1)/fs2+0.5/fs2
    yB_=yB_*(fs3-1)/fs3+0.5/fs3
    xB_=xB_*(fs4-1)/fs4+0.5/fs4

I don’t understand why do you still need to do this recenter. And for me it is also not clear how this recenter works…
It would be appreciated if you could help me to understand what you are doing! Thanks!

One download-forbbiden file in ivd dataset

Dear author:
I'm trying to train the ncnet using ivd dataset. But one image in ivd dataset is forbidden to be downloaded. I opened the related website and found nothing.
Below is information of this image:
dk/Copenhagen/ChIJy84GHRFTUkYRY-FdhFM-nKI/AF1QipMuVTNnEvEMF8iYoq_RDqqyBVQS2NszJMT8jXQf.jpg https://lh5.googleusercontent.com/p/AF1QipMuVTNnEvEMF8iYoq_RDqqyBVQS2NszJMT8jXQf=s2048-k-no

Evaluation script produces no "matches" folder

After downloading pretrained models for demo, we ran:

$ python eval_pf_pascal.py --checkpoint trained_models/ncnet_pfpascal.pth.tar

Which resulted in this output:

NCNet evaluation script - PF Pascal dataset
Creating CNN model...
Loading checkpoint...
Using checkpoint parameters: 
  ncons_channels: [16, 16, 1]
  ncons_kernel_sizes: [5, 5, 5]
Downloading: "https://download.pytorch.org/models/resnet101-5d3b4d8f.pth" to /home/ubuntu/.torch/models/resnet101-5d3b4d8f.pth
100%|███████████████████████| 178728960/178728960 [00:16<00:00, 10820718.99it/s]
Copying weights...
Done!
/ebs1/ncnet/lib/pf_dataset.py:31: FutureWarning: Method .as_matrix will be removed in a future version. Use .values instead.
  self.category = self.pairs.iloc[:,2].as_matrix().astype('float')
Batch: [0/299 (0%)]
Batch: [1/299 (0%)]
Batch: [2/299 (1%)]
Batch: [3/299 (1%)]
Batch: [4/299 (1%)]
Batch: [5/299 (2%)]
Batch: [6/299 (2%)]
Batch: [7/299 (2%)]
Batch: [8/299 (3%)]
Batch: [9/299 (3%)]
Batch: [10/299 (3%)]
Batch: [11/299 (4%)]
Batch: [12/299 (4%)]
Batch: [13/299 (4%)]
Batch: [14/299 (5%)]
Batch: [15/299 (5%)]
Batch: [16/299 (5%)]
Batch: [17/299 (6%)]
Batch: [18/299 (6%)]
Batch: [19/299 (6%)]
Batch: [20/299 (7%)]
Batch: [21/299 (7%)]
Batch: [22/299 (7%)]
Batch: [23/299 (8%)]
Batch: [24/299 (8%)]
Batch: [25/299 (8%)]
Batch: [26/299 (9%)]
Batch: [27/299 (9%)]
Batch: [28/299 (9%)]
Batch: [29/299 (10%)]
Batch: [30/299 (10%)]
Batch: [31/299 (10%)]
Batch: [32/299 (11%)]
Batch: [33/299 (11%)]
Batch: [34/299 (11%)]
Batch: [35/299 (12%)]
Batch: [36/299 (12%)]
Batch: [37/299 (12%)]
Batch: [38/299 (13%)]
Batch: [39/299 (13%)]
Batch: [40/299 (13%)]
Batch: [41/299 (14%)]
Batch: [42/299 (14%)]
Batch: [43/299 (14%)]
Batch: [44/299 (15%)]
Batch: [45/299 (15%)]
Batch: [46/299 (15%)]
Batch: [47/299 (16%)]
Batch: [48/299 (16%)]
Batch: [49/299 (16%)]
Batch: [50/299 (17%)]
Batch: [51/299 (17%)]
Batch: [52/299 (17%)]
Batch: [53/299 (18%)]
Batch: [54/299 (18%)]
Batch: [55/299 (18%)]
Batch: [56/299 (19%)]
Batch: [57/299 (19%)]
Batch: [58/299 (19%)]
Batch: [59/299 (20%)]
Batch: [60/299 (20%)]
Batch: [61/299 (20%)]
Batch: [62/299 (21%)]
Batch: [63/299 (21%)]
Batch: [64/299 (21%)]
Batch: [65/299 (22%)]
Batch: [66/299 (22%)]
Batch: [67/299 (22%)]
Batch: [68/299 (23%)]
Batch: [69/299 (23%)]
Batch: [70/299 (23%)]
Batch: [71/299 (24%)]
Batch: [72/299 (24%)]
Batch: [73/299 (24%)]
Batch: [74/299 (25%)]
Batch: [75/299 (25%)]
Batch: [76/299 (25%)]
Batch: [77/299 (26%)]
Batch: [78/299 (26%)]
Batch: [79/299 (26%)]
Batch: [80/299 (27%)]
Batch: [81/299 (27%)]
Batch: [82/299 (27%)]
Batch: [83/299 (28%)]
Batch: [84/299 (28%)]
Batch: [85/299 (28%)]
Batch: [86/299 (29%)]
Batch: [87/299 (29%)]
Batch: [88/299 (29%)]
Batch: [89/299 (30%)]
Batch: [90/299 (30%)]
Batch: [91/299 (30%)]
Batch: [92/299 (31%)]
Batch: [93/299 (31%)]
Batch: [94/299 (31%)]
Batch: [95/299 (32%)]
Batch: [96/299 (32%)]
Batch: [97/299 (32%)]
Batch: [98/299 (33%)]
Batch: [99/299 (33%)]
Batch: [100/299 (33%)]
Batch: [101/299 (34%)]
Batch: [102/299 (34%)]
Batch: [103/299 (34%)]
Batch: [104/299 (35%)]
Batch: [105/299 (35%)]
Batch: [106/299 (35%)]
Batch: [107/299 (36%)]
Batch: [108/299 (36%)]
Batch: [109/299 (36%)]
Batch: [110/299 (37%)]
Batch: [111/299 (37%)]
Batch: [112/299 (37%)]
Batch: [113/299 (38%)]
Batch: [114/299 (38%)]
Batch: [115/299 (38%)]
Batch: [116/299 (39%)]
Batch: [117/299 (39%)]
Batch: [118/299 (39%)]
Batch: [119/299 (40%)]
Batch: [120/299 (40%)]
Batch: [121/299 (40%)]
Batch: [122/299 (41%)]
Batch: [123/299 (41%)]
Batch: [124/299 (41%)]
Batch: [125/299 (42%)]
Batch: [126/299 (42%)]
Batch: [127/299 (42%)]
Batch: [128/299 (43%)]
Batch: [129/299 (43%)]
Batch: [130/299 (43%)]
Batch: [131/299 (44%)]
Batch: [132/299 (44%)]
Batch: [133/299 (44%)]
Batch: [134/299 (45%)]
Batch: [135/299 (45%)]
Batch: [136/299 (45%)]
Batch: [137/299 (46%)]
Batch: [138/299 (46%)]
Batch: [139/299 (46%)]
Batch: [140/299 (47%)]
Batch: [141/299 (47%)]
Batch: [142/299 (47%)]
Batch: [143/299 (48%)]
Batch: [144/299 (48%)]
Batch: [145/299 (48%)]
Batch: [146/299 (49%)]
Batch: [147/299 (49%)]
Batch: [148/299 (49%)]
Batch: [149/299 (50%)]
Batch: [150/299 (50%)]
Batch: [151/299 (51%)]
Batch: [152/299 (51%)]
Batch: [153/299 (51%)]
Batch: [154/299 (52%)]
Batch: [155/299 (52%)]
Batch: [156/299 (52%)]
Batch: [157/299 (53%)]
Batch: [158/299 (53%)]
Batch: [159/299 (53%)]
Batch: [160/299 (54%)]
Batch: [161/299 (54%)]
Batch: [162/299 (54%)]
Batch: [163/299 (55%)]
Batch: [164/299 (55%)]
Batch: [165/299 (55%)]
Batch: [166/299 (56%)]
Batch: [167/299 (56%)]
Batch: [168/299 (56%)]
Batch: [169/299 (57%)]
Batch: [170/299 (57%)]
Batch: [171/299 (57%)]
Batch: [172/299 (58%)]
Batch: [173/299 (58%)]
Batch: [174/299 (58%)]
Batch: [175/299 (59%)]
Batch: [176/299 (59%)]
Batch: [177/299 (59%)]
Batch: [178/299 (60%)]
Batch: [179/299 (60%)]
Batch: [180/299 (60%)]
Batch: [181/299 (61%)]
Batch: [182/299 (61%)]
Batch: [183/299 (61%)]
Batch: [184/299 (62%)]
Batch: [185/299 (62%)]
Batch: [186/299 (62%)]
Batch: [187/299 (63%)]
Batch: [188/299 (63%)]
Batch: [189/299 (63%)]
Batch: [190/299 (64%)]
Batch: [191/299 (64%)]
Batch: [192/299 (64%)]
Batch: [193/299 (65%)]
Batch: [194/299 (65%)]
Batch: [195/299 (65%)]
Batch: [196/299 (66%)]
Batch: [197/299 (66%)]
Batch: [198/299 (66%)]
Batch: [199/299 (67%)]
Batch: [200/299 (67%)]
Batch: [201/299 (67%)]
Batch: [202/299 (68%)]
Batch: [203/299 (68%)]
Batch: [204/299 (68%)]
Batch: [205/299 (69%)]
Batch: [206/299 (69%)]
Batch: [207/299 (69%)]
Batch: [208/299 (70%)]
Batch: [209/299 (70%)]
Batch: [210/299 (70%)]
Batch: [211/299 (71%)]
Batch: [212/299 (71%)]
Batch: [213/299 (71%)]
Batch: [214/299 (72%)]
Batch: [215/299 (72%)]
Batch: [216/299 (72%)]
Batch: [217/299 (73%)]
Batch: [218/299 (73%)]
Batch: [219/299 (73%)]
Batch: [220/299 (74%)]
Batch: [221/299 (74%)]
Batch: [222/299 (74%)]
Batch: [223/299 (75%)]
Batch: [224/299 (75%)]
Batch: [225/299 (75%)]
Batch: [226/299 (76%)]
Batch: [227/299 (76%)]
Batch: [228/299 (76%)]
Batch: [229/299 (77%)]
Batch: [230/299 (77%)]
Batch: [231/299 (77%)]
Batch: [232/299 (78%)]
Batch: [233/299 (78%)]
Batch: [234/299 (78%)]
Batch: [235/299 (79%)]
Batch: [236/299 (79%)]
Batch: [237/299 (79%)]
Batch: [238/299 (80%)]
Batch: [239/299 (80%)]
Batch: [240/299 (80%)]
Batch: [241/299 (81%)]
Batch: [242/299 (81%)]
Batch: [243/299 (81%)]
Batch: [244/299 (82%)]
Batch: [245/299 (82%)]
Batch: [246/299 (82%)]
Batch: [247/299 (83%)]
Batch: [248/299 (83%)]
Batch: [249/299 (83%)]
Batch: [250/299 (84%)]
Batch: [251/299 (84%)]
Batch: [252/299 (84%)]
Batch: [253/299 (85%)]
Batch: [254/299 (85%)]
Batch: [255/299 (85%)]
Batch: [256/299 (86%)]
Batch: [257/299 (86%)]
Batch: [258/299 (86%)]
Batch: [259/299 (87%)]
Batch: [260/299 (87%)]
Batch: [261/299 (87%)]
Batch: [262/299 (88%)]
Batch: [263/299 (88%)]
Batch: [264/299 (88%)]
Batch: [265/299 (89%)]
Batch: [266/299 (89%)]
Batch: [267/299 (89%)]
Batch: [268/299 (90%)]
Batch: [269/299 (90%)]
Batch: [270/299 (90%)]
Batch: [271/299 (91%)]
Batch: [272/299 (91%)]
Batch: [273/299 (91%)]
Batch: [274/299 (92%)]
Batch: [275/299 (92%)]
Batch: [276/299 (92%)]
Batch: [277/299 (93%)]
Batch: [278/299 (93%)]
Batch: [279/299 (93%)]
Batch: [280/299 (94%)]
Batch: [281/299 (94%)]
Batch: [282/299 (94%)]
Batch: [283/299 (95%)]
Batch: [284/299 (95%)]
Batch: [285/299 (95%)]
Batch: [286/299 (96%)]
Batch: [287/299 (96%)]
Batch: [288/299 (96%)]
Batch: [289/299 (97%)]
Batch: [290/299 (97%)]
Batch: [291/299 (97%)]
Batch: [292/299 (98%)]
Batch: [293/299 (98%)]
Batch: [294/299 (98%)]
Batch: [295/299 (99%)]
Batch: [296/299 (99%)]
Batch: [297/299 (99%)]
Batch: [298/299 (100%)]
Total: 299
Valid: 299
PCK: 78.89%

But no folder of output matches was created, for subsequent processing by the Matlab InLoc program.

Cannot access trained models

https://www.di.ens.fr/willow/research/ncnet seems to be down.

Question about requirements

Hi Ignacio, thank you for your good work. I am also interested in your new paper Sparse-NCNet but I want to work on NCNet first.
I created an environment using anaconda, python=3.6 and torchvision=0.3.0 with CUDA 9.0

When I run
python eval_pf_pascal.py --checkpoint trained_models/ncnet_pfpascal.pth.tar

I got

NCNet evaluation script - PF Pascal dataset
Creating CNN model...
Loading checkpoint...
Using checkpoint parameters: 
  ncons_channels: [16, 16, 1]
  ncons_kernel_sizes: [5, 5, 5]
Traceback (most recent call last):
  File "eval_pf_pascal.py", line 37, in <module>
    checkpoint=args.checkpoint)
  File "/home/ufuk/Desktop/ncnet-master/lib/model.py", line 239, in __init__
    channels=ncons_channels)
  File "/home/ufuk/Desktop/ncnet-master/lib/model.py", line 137, in __init__
    nn_modules.append(Conv4d(in_channels=ch_in,out_channels=ch_out,kernel_size=k_size,bias=True))
  File "/home/ufuk/Desktop/ncnet-master/lib/conv4d.py", line 71, in __init__
    False, _quadruple(0), groups, bias)  
TypeError: __init__() missing 1 required positional argument: 'padding_mode'

What would be the reason for this, do you have any idea. Did you use the same libraries?

Thank you :)
Ufuk

The PCK on pf-pascal dataset is 75.35

Run python train.py --ncons_kernel_sizes 5 5 5 --ncons_channels 16 16 1 --dataset_image_path datasets/pf-pascal --dataset_csv_path datasets/pf-pascal/image_pairs/

The PCK on pf-pascal dataset is 75.35 (78.9 in paper).
Is there some other important hyperparameters? Thank you.

RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:4044: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
"Default grid_sample and affine_grid behavior has changed "
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.)
return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
Traceback (most recent call last):
File "eval_pf_pascal.py", line 73, in
corr4d = model(batch)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/content/ncnet/lib/model.py", line 275, in forward
corr4d = self.NeighConsensus(corr4d)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/content/ncnet/lib/model.py", line 147, in forward
x = self.conv(x)+self.conv(x.permute(0,1,4,5,2,3)).permute(0,1,4,5,2,3)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/container.py", line 139, in forward
input = module(input)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/content/ncnet/lib/conv4d.py", line 126, in forward
use_half=self.use_half,
File "/content/ncnet/lib/conv4d.py", line 52, in conv4d
padding=padding,
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.

Any idea on how to resolve this.

Can it is applicable for more then two objects in the image?

A question above negative loss calculation

Hi, Thanks for this awesome work, I'm trying to train it on my own dataset, so I am exploring how to organize my data. I notice that for unmatched images, you calculate their matching loss by rolling a batch of images(https://github.com/ignacio-rocco/ncnet/blob/master/train.py#L137):

    batch['source_image']=batch['source_image'][np.roll(np.arange(b),-1),:] # roll

I don't quite get why moving each sample one position forward creates unmatched pairs, how do you organize your data? Also, is the set information in the csv files useful during training? Thanks!

Is there known pretrained NCNet model on SPair-71k?

I wonder if there are pretrained NCNet on SPair-71k dataset

ERROR 403: Forbidden, when trying to download the INLOC dataset

I tried to download the inloc dataset by running download.sh but I get the error: permission denied.

Question regarding warping

Hi,

Thanks for providing this code. I have a question regarding calculation of warping. Is it any straight way to use matched points which are the outcome of c2m() to calculate some Homography matrix or warping/alignment between two images or patches?

I can see that in the demo code, you are using bilinearInterpPointTnf() to calculate the warped point, however, my super trivial question is that, how to expand this to the whole image transformation?

I also looked at your other paper, End-to-end weakly-supervised semantic alignment, which seems don't use consensus neighboring idea but addressing the issue of geometrical transformation between two images.

It would be great if you can point me to the right place to find a clue. Many thanks. -Shek