liubin06/attentionce

AttentioNCE: Contrastive Learning with Instance Attention

AttentioNCE: Contrastive Learning with Instance Attention

AttentioNCE integrates the attention mechanism into contrastive learning to guide the model's attention towards high-quality samples while disregarding noisy ones. Consequently, AttentioNCE constructs a variational lower bound for an ideal contrastive loss, offering a worst-case guarantee for maximum likelihood estimation under noisy conditions.

AttentioNCE Framework:

Flags:

--d_pos: scaling factor for positive samples.

--d_neg: scaling factor for negative samples.

Model Pretraining

For instance, run the following command to train an embedding on different datasets.

python main.py --dataset_name 'cifar10'  --d_pos 1 --d_neg 10

python main.py --dataset_name 'stl10'  --d_pos 2 --d_neg 0.5

python main.py --dataset_name 'cifar100'  --d_pos 1 --d_neg 10

python main.py --dataset_name 'tinyImageNet'  --d_pos 4 --d_neg 1

Linear evaluation

The model is evaluated by training a linear classifier after fixing the learned embedding.

path flags: --model_path: choose the model for evaluation.

python linear.py --dataset_name 'stl10' --model_path '../results/stl10/stl10_SimCLR_4model_256_400_2.0_0.5.pth'

Pretrained Models on CIFAR10 Dataset

Method	$d_\text{pos}$	$d_\text{neg}$	Arch	Epoch	Batch Size	Accuracy(%)	Download
SimCLR	-	-	ResNet50	400	256	91.12	model
AttentioNCE	1	10	ResNet50	200	256	92.42	model
AttentioNCE	1	10	ResNet50	400	256	93.08	model

Pretrained Models on STL10 Dataset

Method	$d_\text{pos}$	$d_\text{neg}$	Arch	Epoch	Batch Size	Accuracy(%)	Download
SimCLR	-	-	ResNet50	400	256	80.15	model
AttentioNCE	2	0.5	ResNet50	200	256	87.12	model
AttentioNCE	2	0.5	ResNet50	400	256	89.45	model

Pretrained Models on CIFAR100 Dataset

Method	$d_\text{pos}$	$d_\text{neg}$	Arch	Epoch	Batch Size	Accuracy(%)	Download
SimCLR	-	-	ResNet50	400	256	66.55	model
AttentioNCE	1	10	ResNet50	200	256	69.78	model
AttentioNCE	1	10	ResNet50	400	256	70.23	model

Pretrained Models on TinyImageNet

Method	$d_\text{pos}$	$d_\text{neg}$	Arch	Latent Dim	Batch Size	Accuracy(%)	Download
SimCLR	-	-	ResNet50	400	256	53.40	model
BCL	4	1	ResNet50	200	256	56.58	model
BCL	4	1	ResNet50	400	256	58.61	model

Acknowledgements

Part of this code is credited to SimCLR(ICML 2020), DCL(NeurIPS 2020) and HCL(ICLR 2021).