This is an open source repository for our paper SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding based on the pytorch framework. In this paper, we propose a Supervised Cross-modal Contrastive Learning Framework for Audio-Visual Coding (SCLAV).
Different flows for self-supervised contrastive learning and our proposed supervised cross-modal contrastive learning. The self-supervised contrastive learning (as shown in subfigure (a)) contrasts each anchor and its augmentation against the remaining negatives of the entire batch, which can be viewed as a clustering problem. Our supervised cross-modal contrastive learning (as shown in subfigure (b)) produces a classification hypersphere guided by weak labels. All samples of the same category are compared as positives with the negatives from the rest samples of the batch, and the SCL loss can narrow the distance between different modal representations of the same sample.
@inproceedings{sun2023sclav,
title={SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding},
author={Sun, Chao and Chen, Min and Cheng, Jialiang and Liang, Han and Zhu, Chuanbo and Chen, Jincai},
booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
pages={261--270},
year={2023}
}