hhc1997/l2rm

PyTorch implementation for "Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval (CVPR 2024)"

Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval

Requirements

• Python 3.8
• torch 1.12
• numpy
• scikit-learn
• pomegranate Install. Note that pomegranate requires Cython=0.29, NumPy, SciPy, NetworkX, and joblib. Then you can run python setup.py build and python setup.py install to install it.)
• Punkt Sentence Tokenizer:

import nltk
nltk.download()
> d punkt

Datasets

We follow NCR to obtain image features and vocabularies.

Download Dataset

Noise (Mismatching) Index

We use the same noise index settings as DECL and RCL, which could be found in noise_index. The mismatching ratio (noise ratio) is set as 0.2, 0.4, 0.6, and 0.8.

Training and Evaluation

Training new models

Modify some necessary parameters and run it.

For Flickr30K:

sh train_f30k.sh

For MS-COCO:

sh train_coco.sh

For CC152K:

sh train_cc152k.sh

Evaluation

Modify some necessary parameters and run it.

python main_testing.py

Pre-trained L2RM models

The pre-trained models are available here.

License

Apache License 2.0

Acknowledgements

The code is based on SCAN, SGRAF, NCR, DECL, and KPG-RL licensed under Apache 2.0.

l2rm's Issues

cannot import name 'TrueBetaDistribution' from 'pomegranate

Hello， thanks for your excellent work.

Now I try to run the code however, I encounter some errors.

I got the error as follows,
Traceback (most recent call last):
File "/home/bbb/shicaiwei/L2RM/grammar_test.py", line 2, in
from pomegranate import GeneralMixtureModel,TrueBetaDistribution
ImportError: cannot import name 'TrueBetaDistribution' from 'pomegranate' (/home/bbb/anaconda3/envs/pytorch1.12/lib/python3.8/site-packages/pomegranate-0.14.8-py3.8-linux-x86_64.egg/pomegranate/init.py)

The version of the pomegranate I used is 0.14.8. I guess that may be caused by the version. Could you tell me what's version you used?

About the warmup process

Hello~
Why loss is decreasing while the accuracy is decreasing at the same time ? After 1 epoch, the accuracy is about 70%, then it drops as the epoch increases. Maybe this doesn't matter?

Results on MS-COCO 5K with noise ratio 0

nice work! Could you please provide the results on well-annotated MS-COCO 5K, i.e., MRate is 0.?
Or could you please provide the pre-trained models on MS-COCO?

About the OT Loss

您好
OT Loss只计算了随机选择的reconstructed pairs的损失，那这个loss的数值有什么参考价值吗？我跑出来的loss一直在随机变化，应该是正常的吧？
请问您有没有训练全过程的training log，想参考一下各阶段loss变化和指标提升的情况