witdsl/krt-mlcil

Knowledge Restore and Transfer for Multi-Label Class-Incremental Learning

PyTorch code for the ICCV 2023 paper:
Knowledge Restore and Transfer for Multi-Label Class-Incremental Learning
Songlin Dong, Haoyu Luo, Yuhang He, Xing Wei, Jie Cheng and Yihong Gong
IEEE/CVF International Conference on Computer Vision 2023
https://openaccess.thecvf.com/content/ICCV2023/html/Dong_Knowledge_Restore_and_Transfer_for_Multi-Label_Class-Incremental_Learning_ICCV_2023_paper.html

Abstract

Current class-incremental learning research mainly focuses on single-label classification tasks while multi-label class-incremental learning (MLCIL) with more practical application scenarios is rarely studied. Although there have been many anti-forgetting methods to solve the problem of catastrophic forgetting in single-label class-incremental learning, these methods have difficulty in solving the MLCIL problem due to label absence and information dilution problems. To solve these problems, we propose a Knowledge Restore and Transfer (KRT) framework including a dynamic pseudo-label (DPL) module to solve the label absence problem by restoring the knowledge of old classes to the new data and an incremental cross-attention (ICA) module with session-specific knowledge retention tokens storing knowledge and a unified knowledge transfer token transferring knowledge to solve the information dilution problem. Comprehensive experimental results on MS-COCO and PASCAL VOC datasets demonstrate the effectiveness of our method for improving recognition performance and mitigating forgetting on multi-label class-incremental learning tasks.

Task protocol

Data partitioning is found in src/helper_functions/IncrementalDataset.py

Setup

Install anaconda: https://www.anaconda.com/distribution/
set up conda environment w/ python 3.7, ex: conda create --name coda python=3.7
conda activate coda
pip install -r requirements.txt
NOTE: this framework was tested using torch == 1.11.0 but should work for previous versions
timm ==0.5.4 inplace_abn=1.1.0

Datasets and pretrained model

Download MS-COCO 2014
Put the dataset as ./datasets/coco2014
Donwload PASCAL VOC 2007
Put the dataset as ./datasets/voc2007
TResNetM pretrained on ImageNet 21k is available at TResNetM_pretrained_model, download it to ./pretrained_models and rename it as tresnet_m_224_21k.pth

Training

All commands should be run under the project root directory. The scripts are set up for 2 GPUs but can be modified for your hardware.

sh train_mlcil_coco.sh
sh train_mlcil_voc.sh

Results

Results will be saved in a folder named logs/. To get the exprimental detail, retrieve the number in the file logs/**/log/log.txt The model for the incremental stage is stored under saved_models/

Citation

If you found our work useful for your research, please cite our work:

       @InProceedings{dong2023knowledge,
         title={Knowledge Restore and Transfer for Multi-Label Class-Incremental Learning}, 
         author={Dong, Songlin and Luo, Haoyu and He, Yuhang and Wei, Xing and Cheng, Jie and Gong, Yihong},
         booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
         pages={18711--18720},
         year={2023}
       }

krt-mlcil's Issues

About the construction of the datasets

Thank you for your contribution to multi-label continual learning, excellent work, I started following your preprint version when it was just coming out. I would like to ask the authors if it is possible to show the details of the datasets partition and the strategy used to partition the datasets.

inplace abn=1.1.0 is very difficult to install

Very good work. inplace_abn=1.1.0 is very difficult to install in conda environment. Can the author provide any advice

Critical DPL and ICA modules

We've taken a close look at your code. Are your critical DPL and ICA modules yet to be released？？？