supersunn/sclav

Supervised Cross-modal Contrastive Learning Framework for Audio-Visual Coding (SCLAV)

This is an open source repository for our paper SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding based on the pytorch framework. In this paper, we propose a Supervised Cross-modal Contrastive Learning Framework for Audio-Visual Coding (SCLAV).

Supervised Cross-modal Contrastive Loss (SCL Loss)

Different flows for self-supervised contrastive learning and our proposed supervised cross-modal contrastive learning. The self-supervised contrastive learning (as shown in subfigure (a)) contrasts each anchor and its augmentation against the remaining negatives of the entire batch, which can be viewed as a clustering problem. Our supervised cross-modal contrastive learning (as shown in subfigure (b)) produces a classification hypersphere guided by weak labels. All samples of the same category are compared as positives with the negatives from the rest samples of the batch, and the SCL loss can narrow the distance between different modal representations of the same sample.

(1) Self-supervised Contrastive Learning

(2) Supervised Cross-modal Contrastive Learning

Citation

If you find this work useful, please consider citing it.

@inproceedings{sun2023sclav,
  title={SCLAV: Supervised Cross-modal Contrastive Learning for Audio-Visual Coding},
  author={Sun, Chao and Chen, Min and Cheng, Jialiang and Liang, Han and Zhu, Chuanbo and Chen, Jincai},
  booktitle={Proceedings of the 31st ACM International Conference on Multimedia},
  pages={261--270},
  year={2023}
}

Acknowledgement

Part of our code is borrowed from the following repositories. We thank to the authors for releasing their codes.

XDVioDet

AVE-ECCV18