jp-guo/gm-transformer

[ICASSP 2024] GMTR: Graph Matching Transformers

Graph Matching Transformers

Official implementation of ICASSP 2024 paper: "GMTR: Graph Matching Transformers".

Authors: Jinpei Guo, Shaofeng Zhang, Runzhong Wang, Chang Liu, Junchi Yan

Setup

Environment

We use the environments recommended in ThinkMatch. You can also see more details in ThinkMatch-runtime.

Data

1. PascalVOC-Keypoint

   1. Download VOC2011 dataset and make sure it looks like data/PascalVOC/TrainVal/VOCdevkit/VOC2011

   2. Download keypoint annotation for VOC2011 from Berkeley server or google drive and make sure it looks like data/PascalVOC/annotations

   3. The train/test split is available in data/PascalVOC/voc2011_pairs.npz. This file must be added manually.

2. SPair-71k

   1. Download SPair-71k dataset

   2. Unzip the dataset and make sure it looks like data/SPair-71k

Our code also supports Willow-Object-Class, CUB2011 and IMC-PT-SparseGM datasets, please refer to ThinkMatch for more deatails.

Pretrained Backbones

Please download the pretrained parameters for backbones from google drive and place them in experiments/pretrained.

Experiments

Please place all the yaml files in the experiments directory.

Backbones

If you have already built the singularity with docker image, use the following command:

singularity exec \
     --nv \
     --bind /data:/data \
     gm_env.sif \
     bash -c "CUDA_VISIBLE_DEVICES=0 python train_eval.py --cfg experiments/[backbone]_[framework]_[dataset].yaml"

Or if you have prepared your environments in other ways (e.g. conda), simply run

python train_eval.py --cfg experiments/[backbone]_[framework]_[dataset].yaml

We support the following choices:

[backbone]: vgg16, resnet34, gmt, vit, ceit, xcit
[framework]: ngmv2, bbgm
[dataset]: voc, spair71k

We also provide the yaml files including GMT and ViT backbones with NGMv2 and BBGM frameworks on Willow-Object-Class dataset.

For other frameworks and datasets, please download the corresponding yaml files from ThinkMatch.

GMTR

To evaluate the performance of attention module in GMTR, please use the yaml file with name gmt_ngmv2_[dataset] and turn on the switch in the file:

ATTN:
  SWITCH: True