ivanpanshin/supcon-framework

Implementation of Supervised Contrastive Learning with AMP, EMA, SWA, and many other tricks

supcon-framework's Issues

Suggest to loosen the dependency on albumentations

Hi, your project SupCon-Framework requires "albumentations==0.5.2" in its dependency. After analyzing the source code, we found that the following versions of albumentations can also be suitable without affecting your project, i.e., albumentations 0.5.1. Therefore, we suggest to loosen the dependency on albumentations from "albumentations==0.5.2" to "albumentations>=0.5.1,<=0.5.2" to avoid any possible conflict for importing more packages or for downstream projects that may use SupCon-Framework.

May I pull a request to further loosen the dependency on albumentations?

By the way, could you please tell us whether such dependency analysis may be potentially helpful for maintaining dependencies easier during your development?

---

We also give our detailed analysis as follows for your reference:

Your project SupCon-Framework directly uses 8 APIs from package albumentations.

albumentations.augmentations.transforms.Resize.__init__, albumentations.core.composition.Compose.__init__, albumentations.augmentations.transforms.RandomResizedCrop.__init__, albumentations.pytorch.transforms.ToTensorV2.__init__, albumentations.augmentations.transforms.ToGray.__init__, albumentations.augmentations.transforms.Rotate.__init__, albumentations.augmentations.transforms.ColorJitter.__init__, albumentations.augmentations.transforms.Normalize.__init__

Beginning from the 8 APIs above, 14 functions are then indirectly called, including 14 albumentations's internal APIs and 0 outsider APIs. The specific call graph is listed as follows (neglecting some repeated function occurrences).

[/ivanpanshin/SupCon-Framework]
+--albumentations.augmentations.transforms.Resize.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
+--albumentations.core.composition.Compose.__init__
|      +--albumentations.core.composition.BaseCompose.__init__
|      |      +--albumentations.core.composition.Transforms.__init__
|      |      |      +--albumentations.core.composition.Transforms._find_dual_start_end
|      |      |      |      +--albumentations.core.composition.Transforms._find_dual_start_end
|      +--albumentations.augmentations.bbox_utils.BboxProcessor.__init__
|      |      +--albumentations.core.utils.DataProcessor.__init__
|      +--albumentations.core.composition.BboxParams.__init__
|      |      +--albumentations.core.utils.Params.__init__
|      +--albumentations.augmentations.keypoints_utils.KeypointsProcessor.__init__
|      |      +--albumentations.core.utils.DataProcessor.__init__
|      +--albumentations.core.composition.KeypointParams.__init__
|      |      +--albumentations.core.utils.Params.__init__
|      +--albumentations.core.composition.BaseCompose.add_targets
+--albumentations.augmentations.transforms.RandomResizedCrop.__init__
|      +--albumentations.augmentations.transforms._BaseRandomSizedCrop.__init__
|      |      +--albumentations.core.transforms_interface.BasicTransform.__init__
+--albumentations.pytorch.transforms.ToTensorV2.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
+--albumentations.augmentations.transforms.ToGray.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
+--albumentations.augmentations.transforms.Rotate.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
|      +--albumentations.core.transforms_interface.to_tuple
+--albumentations.augmentations.transforms.ColorJitter.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__
|      +--albumentations.augmentations.transforms.ColorJitter.__check_values
+--albumentations.augmentations.transforms.Normalize.__init__
|      +--albumentations.core.transforms_interface.BasicTransform.__init__

We scan albumentations's versions and observe that during its evolution between any version from [0.5.1] and 0.5.2, the changing functions (diffs being listed below) have none intersection with any function or API we mentioned above (either directly or indirectly called by this project).

diff: 0.5.2(original) 0.5.1
['albumentations.augmentations.transforms.MedianBlur', 'albumentations.augmentations.transforms.CropNonEmptyMaskIfExists.targets_as_params', 'albumentations.augmentations.transforms.GaussianBlur', 'albumentations.augmentations.transforms.CropNonEmptyMaskIfExists.update_params', 'albumentations.pytorch.transforms.ToTensorV2', 'albumentations.pytorch.transforms.ToTensorV2.apply', 'albumentations.augmentations.transforms.CropNonEmptyMaskIfExists.get_params_dependent_on_targets', 'albumentations.augmentations.transforms.CropNonEmptyMaskIfExists', 'albumentations.augmentations.transforms.CropNonEmptyMaskIfExists._preprocess_mask']

Therefore, we believe that it is quite safe to loose your dependency on albumentations from "albumentations==0.5.2" to "albumentations>=0.5.1,<=0.5.2". This will improve the applicability of SupCon-Framework and reduce the possibility of any further dependency conflict with other projects.

这个loss可以和推荐算法相结合吗？

RT

License

Would you consider adding a license to this repo? Otherwise it is not clear what other people are allowed to do with your code.

Thanks!

can supCon loss used in multi-label classification?

I have a text multi-label classification task，can i use supCon loss ?
supCon loss is accumulated by every label view，for example:
batch data label = [[1, 0, 1], [0, 1, 1], [1, 1, 0], [0, 1, 1] ]
from view label 0, positive examples = {0, 2}，negative samples = {1, 3}
from view label 1, positive examples = {1, 2,3}, negative samples = {0}
from view label 2, positive examples = {0, 1, 2}, negative samples = {2}

is this setting here reasonable ？

SupCon loss

@ivanpanshin Thanks for the wonderful implementation. Does the fact that the SupCon loss can be seen as a generalization of the triplet mean that we should expect the embeddings to lie either in a euclidean space or low dimensional manifold?

Run custom dataset

Hello everyone, I am trying to run my own dataset, but I face the same error all the time.

Traceback (most recent call last):
File "train.py", line 64, in
loaders = utils.build_loaders(data_dir, transforms, batch_sizes, num_workers, second_stage=(stage == 'second'))
File "/home/karantai/SupCon-Framework/tools/utils.py", line 93, in build_loaders
transform=transforms['valid_transforms'], second_stage=True)
File "/home/karantai/SupCon-Framework/tools/datasets.py", line 79, in create_supcon_dataset
return DATASETS[dataset_name](data_dir, train, transform, second_stage)
File "/home/karantai/SupCon-Framework/tools/datasets.py", line 50, in init
super().init(root=data_dir, train=train, download=False, transform=transform)
File "/home/karantai/anaconda3/envs/myenv/lib/python3.6/site-packages/torchvision/datasets/cifar.py", line 69, in init
' You can use download=True to download it')
RuntimeError: Dataset not found or corrupted. You can use download=True to download it

I have set Download = False. And I have created a pickled dict with my data as the cifar 10 is. Maybe my problem is trivial, but I would appreciate any help in utils.dataset.py modification or the dataset's layout.

Thank you in advance guys!

Output dimensions of model(data) in utils.py

Dear Ivan, than you very much for your work!

The output of the model in utils.py in line 250:

            output = model(data) is a tensor of [20,2] (the batch size = 20).

My question is: why does the second column mean? It would make more sense if it was [20,1] resembling the predictions of the 20 images.
Thank you very much in advance !

John

ValueError in Stage2

Hi, I'm trying to run your code on CIFAR-10. The training and SWA in stage1 were fine, but I got the following error when training stage2:

root@864d7f9c24b4:/SupCon-Framework# python train.py --config_name configs/train/train_supcon_resnet18_cifar10_stage2.yml 
{'model': {'backbone': 'resnet18', 'ckpt_pretrained': 'weights/supcon_first_stage_cifar10/swa', 'num_classes': 10}, 'train': {'n_epochs': 20, 'amp': True, 'ema': True, 'ema_decay_per_epoch': 0.3, 'logging_name': 'supcon_second_stage_cifar10', 'target_metric': 'accuracy', 'stage': 'second'}, 'dataset': 'data/cifar10', 'dataloaders': {'train_batch_size': 20, 'valid_batch_size': 20, 'num_workers': 12}, 'optimizer': {'name': 'SGD', 'params': {'lr': 0.01}}, 'scheduler': {'name': 'CosineAnnealingLR', 'params': {'T_max': 20, 'eta_min': 0.001}}, 'criterion': {'name': 'LabelSmoothing', 'params': {'classes': 10, 'smoothing': 0.01}}}
Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "train.py", line 111, in <module>
    train_metrics = utils.train_epoch_ce(loaders['train_features_loader'], model, criterion, optimizer, scaler, ema)
  File "/SupCon-Framework/tools/utils.py", line 250, in train_epoch_ce
    ema.update(model.parameters())
  File "/usr/local/lib/python3.8/dist-packages/torch_ema/ema.py", line 88, in update
    parameters = self._get_parameters(parameters)
  File "/usr/local/lib/python3.8/dist-packages/torch_ema/ema.py", line 65, in _get_parameters
    raise ValueError(
ValueError: Number of parameters passed as argument is different from number of shadow parameters maintained by this ExponentialMovingAverage

Another minor problem is GPU usage. I used to run another implementation of SupContrast. It requires 8x GPU memory (and higher utilization of each GPU) to train stage1 of the same backbone and batch size. Did your know what cause that difference?

Train in 2 steps instead of 4?

Hi thanks for your neat work here. It looks like there are currently 4 proposed steps to train:
for stage 1: train.py, swa.py and for stage 2: train.py, swa.py.

Is there a way to pre-train using supcon loss once, and classification training once, for a total of two steps?