declare-lab/msa-robustness

NAACL 2022 paper on Analyzing Modality Robustness in Multimodal Sentiment Analysis

MSA-Robustness

NAACL 2022 paper on Analyzing Modality Robustness in Multimodal Sentiment Analysis

Setup the environment

Configure the environment of different models respectively, configure the corresponding environment according to the requirements.txt in the model directory.

Data Download

• Install CMU Multimodal SDK. Ensure, you can perform from mmsdk import mmdatasdk.

Running the code

Take MISA as an example

1. cd MISA
2. cd src
3. Set word_emb_path in config.py to glove file.
4. Set sdk_dir to the path of CMU-MultimodalSDK.
5. bash run.sh When doing robustness training, run the "TRAIN" section of run.sh, and when doing diagnostic tests, run the "TEST" section of run.sh.

    --train_method means the robustness training method, one of {missing, g_noise, hybird}, missing means set to zero noise, g_noise means set to Gaussian Noise, hybird means the data of train_changed_pct is set to zero_noise, and the data of train_changed_pct is set to Gaussian_Noise.

    --train_changed_modal means the modality of change during training, one of {language, video, audio}.

    --train_changed_pct means the percentage of change during training, can set between 0~1.

    --test_method means the diagnostic tests method, one of {missing, g_noise, hybird}, missing means set to zero noise, g_noise means set to Gaussian Noise, hybird means the data of test_changed_pct is set to zero_noise, and the data of test_changed_pct is set to Gaussian_Noise.

    --test_changed_modal means the modality of change during testing, one of {language, video, audio}.

    --train_changed_pct means the percentage of change during testing, can set between 0~1.

Citation

@article{hazarika2022analyzing,
  title={Analyzing Modality Robustness in Multimodal Sentiment Analysis},
  author={Hazarika, Devamanyu and Li, Yingting and Cheng, Bo and Zhao, Shuai and Zimmermann, Roger and Poria, Soujanya},
  publisher={NAACL},
  year={2022}
}

msa-robustness's Issues

Obvious error

There are a lot of missing files in the code base

Hyperparameter Values for Experiments

It would be amazing if the hyperparameters (and other settings) were shared to enable easy reproduction of all the results. Take for example, BBFN, there's 47 different arguments to the program, a lot of which will have some impact on the resulting performance of the model.

While some hyperparameters are given in the paper there's definitely some missing, e.g. Batch Size, lambda_d for when using the discriminator, etc.

I am trying to reproduce some results(BBFN at the moment), and so far, I can get within a few per cent of the original paper and this paper, but there's surely some random toggle or parameter I am missing.

Please, just whatever the settings were which produced the results presented in the paper.

Multimodal infomax does not have modules.encoders file

MSA-Robustness/Multimodal-Infomax/src/model.py

Line 8 in 2466d33

from modules.encoders import LanguageEmbeddingLayer, CPC, MMILB, RNNEncoder, SubNet

The Multimodal-InfomaxMultimodal-Infomax directory lacks the utils.py file