ishine/leveraging-self-supervised-learning-for-avsr

Official PyTorch implementation of paper Leveraging Uni-Modal Self Supervised Learning for Multimodal Audio-visual Speech Recognition

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition

Official PyTorch implementation of paper Leveraging Uni-Modal Self Supervised Learning for Multimodal Audio-visual Speech Recognition

How to install environment

1. Clone the repo into a directory.

git clone https://github.com/LUMIA-Group/Leveraging-Self-Supervised-Learning-for-AVSR.git

2. Install all required packages.

pip install -r requirements.txt

Noted that the Pytorch-Lightning lib do not support a wrapped ReduceLROnPlateau scheduler, we need to modify the lib manually by:

python -c "exec(\"import pytorch_lightning\nprint(pytorch_lightning.__file__)\")"
vi /path/to/pytorch_lightning/trainer/optimizers.py

and comments the 154-156 lines

# scheduler["reduce_on_plateau"] = isinstance(
#     scheduler["scheduler"], optim.lr_scheduler.ReduceLROnPlateau
# )

How to prepare dataset

1. Download LRW dataset and LRS2 dataset
2. Download pretrained MoCo v2 model and wav2vec 2.0 model
3. Change the directory in config.py to "relative directory" relative to the project root directory
4. Preprocessing the LRW dataset.

cd trainFrontend
python saveh5.py

5. Preprocessing the LRS2 dataset.

python saveh5.py

How to train

1. Train the visual front-end on LRW.

python trainfrontend.py

2. Change the args["MOCO_FRONTEND_FILE"] in config.py to the trained front-end file, and config args["MODAL"] to choose modality.
3. Train the AO and VO model first.

python train.py

4. Then train the AV model. Before that, change the args["TRAINED_AO_FILE"] and args["TRAINED_VO_FILE"] to the trained AO and VO model.

python train.py

How to test

1. Choose test configuration and model.
2. Evaluate the visual word classification performance.

python evalfrontend.py

3. Evaluate the AO/VO/AV model.

python eval.py