Could you please provide more guidance on reproducing Aachen for SIFT?

[GabbySuwichaya]

Could you please provide more guidance on reproducing Aachen for SIFT?

[vdvchen]

Hi,

1. Set self.root=True means using RootSIFT instead of SIFT. RootSIFT is l2-normalized by definition and SIFT extracted by opencv is also normalized by default (with a norm of 256 as I recall). So if you perform direct NN matching, the normalization should not be an issue. As a safe guard, you can l2-normalize these features manually.
2. For Aachen day-night I use the original img resolution.
3. Sorry but I think MNN doesn't have a thresholding. You can apply Lowe's ratio test(https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work) together with MNN to achieve better results. For RootSIFT, I think 0.8 is a good threshold for ratio test.
4. 512 means setting seed_top_k as [512,512], which is proportionally increased with keypoint number(8k). 0.2 means setting p_th as 0.2.
5. Yes, I think that's a correct setting for SG+RootSIFT.
6. One last thing, for Aachen, using upright RoofSIFT/SIFT generally improves the performance. I used the following code to produce upright features.

def run(self, img_path):  
    sift = cv2.xfeatures2d.SIFT_create(nfeatures=self.num_kpt, contrastThreshold=self.contrastThreshold)<br/>
    img = cv2.imread(img_path,cv2.IMREAD_GRAYSCALE)
    cv_kp= sift.detect(img, None)
    score=np.asarray([_kp.response for _kp in cv_kp])
    if self.upright:
        cv_kp=[cv2.KeyPoint(x=kp.pt[0],y=kp.pt[1],_size=kp.size,_angle=0) for kp in cv_kp]<br/>
    cv_kp,desc=sift.compute(img,cv_kp)
    desc=desc/np.linalg.norm(desc,axis=-1)[:,np.newaxis]
    if self.root:
        desc=np.sqrt(abs(desc/(np.linalg.norm(desc,axis=-1,ord=1)[:,np.newaxis]+1e-4)))
    kp = np.array([[_kp.pt[0], _kp.pt[1]] for _kp in cv_kp]) # N*2
    index=np.flip(np.argsort(score))
    kp,desc,score=kp[index],desc[index],score[index]
    return kp, desc,score

You may use codes in this reply.

[vdvchen]

Hi,

1. Set self.root=True means using RootSIFT instead of SIFT. RootSIFT is l2-normalized by definition and SIFT extracted by opencv is also normalized by default (with a norm of 256 as I recall). So if you perform direct NN matching, the normalization should not be an issue. As a safe guard, you can l2-normalize these features manually.

2. For Aachen day-night I use the original img resolution.

3. Sorry but I think MNN doesn't have a thresholding. You can apply Lowe's ratio test(https://stackoverflow.com/questions/51197091/how-does-the-lowes-ratio-test-work) together with MNN to achieve better results. For RootSIFT, I think 0.8 is a good threshold for ratio test.

4. 512 means setting seed_top_k as [512,512], which is proportionally increased with keypoint number(8k). 0.2 means setting p_th as 0.2.

5. Yes, I think that's a correct setting for SG+RootSIFT.

6. One last thing, for Aachen, using upright RoofSIFT/SIFT generally improves the performance. I used the following code to produce upright features.

def run(self, img_path):  
    sift = cv2.xfeatures2d.SIFT_create(nfeatures=self.num_kpt, contrastThreshold=self.contrastThreshold)<br/>
    img = cv2.imread(img_path,cv2.IMREAD_GRAYSCALE)
    cv_kp= sift.detect(img, None)
    score=np.asarray([_kp.response for _kp in cv_kp])
    if self.upright:
        cv_kp=[cv2.KeyPoint(x=kp.pt[0],y=kp.pt[1],_size=kp.size,_angle=0) for kp in cv_kp]<br/>
    cv_kp,desc=sift.compute(img,cv_kp)
    desc=desc/np.linalg.norm(desc,axis=-1)[:,np.newaxis]
    if self.root:
        desc=np.sqrt(abs(desc/(np.linalg.norm(desc,axis=-1,ord=1)[:,np.newaxis]+1e-4)))
    kp = np.array([[_kp.pt[0], _kp.pt[1]] for _kp in cv_kp]) # N*2
    index=np.flip(np.argsort(score))
    kp,desc,score=kp[index],desc[index],score[index]
    return kp, desc,score

[GabbySuwichaya]

Hi! Thanks so much. I have tried...
Yes, for RootSIFT, using the ratio test in combination with MNN (with 0.8 thresholding value) provides better performance.
I actually get a little lower performance: 41.8 / 56.1 / 61.2 .... If you have any further suggestions for better performance, please kindly let me know...

Also, I have another question... I am using your work with some dataset ....

What should I do if the minimum number of keypoints are lower than 256?
Sometimes when I used Superpoint, the number of keypoints is quite low...

It seems that I get this error related to the seeding.
This happens when the value of mask_survive.sum(dim=1)[0]+2 is equal to the number of keypoints.

Should I reduce the confidence value? or the value of seed ?

For example, if the number of keypoints = 233, which is less than 256.
And if it happens that mask_survive.sum(dim=1)[0]+2=233 then, I will get the error below

def seeding(nn_index1,nn_index2,x1,x2,topk,match_score,confbar,nms_radius,use_mc=True,test=False): 
   ....
 
    #confidence bar
    match_score[match_score<confbar]=-1
    mask_survive=match_score>0
    if test:
        topk=min(mask_survive.sum(dim=1)[0]+2,topk)  
    _,topindex = torch.topk(match_score,topk,dim=-1)#b*k
    seed_index1,seed_index2=topindex,nn_index1.gather(index=topindex,dim=-1)
 
    return seed_index1,seed_index2

The error looks like the following

> /mnt/HDD4TB1/BnchMrk3DRecon/SGMNet/sgmnet/match_model.py(51)seeding()
-> seed_index1,seed_index2=topindex,nn_index1.gather(index=topindex,dim=-1)
(Pdb) c
Traceback (most recent call last):
  File "local_feature_eval_scripts/matching_pipeline.py", line 163, in <module>
    matcher.matching_images()
  File "/mnt/HDD4TB1/local-feature-evaluation/scripts/custom_matching.py", line 324, in matching_images
    matches = self.SGM_wrapper_cuda.match(pred)  
  File "/mnt/HDD4TB1/BnchMrk3DRecon/Matcher_OG_wrap.py", line 70, in match
    corr1,corr2, predict_matches = self.SGMmatcher.run(test_data) 
  File "/mnt/HDD4TB1/BnchMrk3DRecon/SGMNet/SGMmatcher.py", line 46, in run
    res=self.model(feed_data,test_mode=True)
  File "/home/gabby-suwichaya/anaconda3/envs/SGAT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/HDD4TB1/BnchMrk3DRecon/SGMNet/sgmnet/match_model.py", line 216, in forward
    
  File "/mnt/HDD4TB1/BnchMrk3DRecon/SGMNet/sgmnet/match_model.py", line 51, in seeding
    seed_index1,seed_index2=topindex,nn_index1.gather(index=topindex,dim=-1)
RuntimeError: invalid argument 5: k not in range for dimension at /pytorch/aten/src/THC/generic/THCTensorTopK.cu:24

[vdvchen]

Hi,

For aachen, using upright features should further boost performance.

We assume the input kpt number should be larger than seed number. If you are using it with very low kpt number(eg ~200), we suggest to sample fewer seeds, while the performance is not guaranteed.

[GabbySuwichaya]

Hi! Thank you so much...
I am very sorry for this question.. I am very new to this field... Could you help explain how to extract the upright feature?

We assume the input kpt number should be larger than seed number. If you are using it with very low kpt number(eg ~200), we suggest to sample fewer seeds, while the performance is not guaranteed.

I see thank you very much for confirmation.. I will try accordingly.

[GabbySuwichaya]

Thank you very much ! I have missed it totally..