This is a user-friendly implementation of Area to Point Matching (A2PM) framework, powered by hydra.
It contains the implementation of SGAM (arXiv'23), a training-free version of MESA (CVPR'24) and DMESA (arXiv'24).
Due to the power of hydra, the implementation is highly configurable and easy to extend.
It supports the implementation of feature matching approaches adopting the A2PM framework, and also enables the combination of new point matching and area matching methods.
To begin with, you need to install the dependencies following the instructions below.
- We recommend using
conda
to create a new environment for this project.
conda create -n A2PM python==3.8
conda activate A2PM
-
Install torch, torchvision, and torchaudio by running the following command.
- Note the requirements of the torch version is soft, and the code is tested on
torch==2.0.0+cu118
.
pip install torch==2.0.0+cu118 torchvision==0.15.1+cu118 torchaudio==2.0.1 -index-url https://download.pytorch.org/whl/cu118
- Note the requirements of the torch version is soft, and the code is tested on
-
Install the basic dependencies by running the following command.
pip install -r requirements.txt
This code is based on hydra, which is a powerful configuration system for Python applications. The documentation of hydra can be found here.
In the following, we will introduce how to use the code by describing its components with hydra configurations.
We offer dataloaders for two widely-used datasets, including
ScanNet1500
andMegaDepth1500
.
-
You can follow the instructions in LoFTR to download the datasets and set the paths in the configuration files in
conf/dataset/
. -
As our methods rely on the segmentation results, whose paths are also needed to be set in the configuration files in
conf/dataset/
. For example,sem_mode
: the segmentation method used, includingSAM
,SEEM
,GT
sem_folder
: path to the folder containing the segmentation resultssem_post
: the file format of the segmentation results (For ScanNet, it is npy if usingSAM
, and png if usingSEEM
orGT
)
-
The segmentation process will be discussed in the following section.
-
More datasets can be easily added by adding new dataloaders in
dataloader/
and setting the corresponding configurations inconf/dataset/
.
The segmentation results are needed for the area matching methods.
- To use Segment Anything Model (SAM) for segmentation, we provide our inference code in
segmentor/
. To use it, you need to:- clone the SAM repository and put it into the
segmentor/..
folder, corresponding to the path set in thesegmentor/SAMSeger.py - L23
. - intall the dependencies in the SAM repository.
- set the pre-trained model path in the configuration dict in
segmentor/ImgSAMSeg.py - L34
.
- clone the SAM repository and put it into the
- See the
segmentor/sam_seg.sh
for the usage of the SAM segmentation code.
Area matching is to establish semantic area matches between two images for matching reduandancy reduction, which is the core of the A2PM framework.
- We provide three area matchers, including:
-
Semantic Area Matching
(our method in SGAM)- The implementation in
area_matchers/sem_am.py
. - The configuration in
conf/area_matcher/sem_area_matcher.yaml
. - See also the repository SGAM-code.
- The implementation in
-
MESA-free
- A training-free version of MESA.
- The implementation in
area_matchers/mesa.py
. - The configuration is in
conf/area_matcher/mesa-f.yaml
. - The
MESA-free
eliminates the need for training the area similarity calculation module in MESA, and directly uses the off-the-shelf patch matching in ASpanFormer like DMESA. - The
MESA-free
is easier to use, but its performance is slightly lower than the originalMESA
.
-
DMESA
- A dense counterpart of MESA proposed in paper, more einfficient and flexible.
- The implementation in
area_matchers/dmesa.py
. - The configuration is in
conf/area_matcher/dmesa.yaml
.
-
Point matching is to establish point matches between two (area) images.
-
Here, we provide four point matchers, including:
Sparse
: {SuperPoint+SuperGlue}Semi-Dense
: {ASpanFormer, LoFTR}Dense
: {DKM}.
-
Their configurations are put in
conf/point_matcher/
, with warppers inpoint_matchers/
. -
For some of them, the inside paths need to be modified, which we have fixed in the submodules in
point_matchers/
. -
Before running, you need download the pre-trained models and put them in the corresponding paths in the configuration
yaml
files. -
More point matchers can be easily added by adding simialr warppers.
We fuse matches from multiple inside-area point matching by the geometry area matching module.
-
The configuration is in
conf/geo_area_matcher/
. -
We provide two fusion methods, including:
- original GAM proposed in SGAM
- code in
geo_area_matcher/gam.py
- configuration in
conf/geo_area_matcher/gam.yaml
- code in
- a more effective GAM
- code in
geo_area_matcher/egam.py
- configuration in
conf/geo_area_matcher/egam.yaml
- code in
- original GAM proposed in SGAM
The A2PM framework will combine the above components to form a complete feature matching pipeline.
-
The implementation is in
scripts/test_a2pm.py
. You can run the shell scriptscripts/test_in_dev.sh
to test the A2PM framework on a pair of images. -
The pipeline configuration is set in
conf/experiment/*.yaml
. You can choose the one you want to use by setting the+experiment=xxx.yaml
in the shell script.
-
We provide the evaluation code in
metric/
. The evaluation metrics include:- Pose estimation AUC
- Mean Matching Accuracy
- Area Matching Accuracy
-
The
metric/instance_eval.py
is used to evaluate the instance-level matching results. It is used intest_a2pm.py
. -
The
metric/eval_ratios.py
is used to evaluate the batch-level matching results. Set the paths in the py file and run it to get the evaluation results.
You can run the benchmark test by running the shell script such as:
./scripts/dmesa-dkm-md.sh # DMESA+DKM on MegaDepth1500
You can change the configurations in the shell script to test different methods, i.e. +experiment=xxx
.
Take DKM as an example, the expected results are as follows:
SN1500( |
DKM | MESA-free+DKM | DMESA+DKM |
---|---|---|---|
Pose AUC@5 | 30.26 | 31.64 | 30.96 |
Pose AUC@10 | 51.51 | 52.80 | 52.41 |
Pose AUC@20 | 69.43 | 70.08 | 69.74 |
MD1500( |
DKM | MESA-free+DKM | DMESA+DKM |
---|---|---|---|
Pose AUC@5 | 63.61 | 63.85 | 65.65 |
Pose AUC@10 | 76.75 | 77.38 | 78.46 |
Pose AUC@20 | 85.72 | 86.47 | 86.97 |
-
In this evaluation code, we fix the random seed to '2' (see
scripts/test_a2pm.py
), which is different from the settings in our paper (without fixing the random seed). Thus, the results are slightly different from the results in the paper, but the effectiveness of our methods is consistent. -
Also, the parameters in the configuration files are set to the default values. Due to the complexity of the parameter settings, we have not tuned the parameters for the best performance.
- Better results can be achieved by tuning the parameters for specific datasets and tasks.
- However, the default parameters are enough to show the effectiveness of our methods.
If you find this work useful, please consider citing:
@article{SGAM,
title={Searching from Area to Point: A Hierarchical Framework for Semantic-Geometric Combined Feature Matching},
author={Zhang, Yesheng and Zhao, Xu and Qian, Dahong},
journal={arXiv preprint arXiv:2305.00194},
year={2023}
}
@InProceedings{MESA,
author = {Zhang, Yesheng and Zhao, Xu},
title = {MESA: Matching Everything by Segmenting Anything},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2024},
pages = {20217-20226}
}
@misc{DMESA,
title={DMESA: Densely Matching Everything by Segmenting Anything},
author={Yesheng Zhang and Xu Zhao},
year={2024},
eprint={2408.00279},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
We thank the authors of the following repositories for their great works: