viethoang1512/icdar-emoreccom

Multimodal Emotion Recognition on Comics scenes (ICDAR 2021)

ICDAR 2021

Multimodal Emotion Recognition on Comics scenes

Overview

• This is the code for EmoRecCom #1 solution (Tensorflow 2.0 version)
• For usage of this code, please follow here
• The ensemble models (TF + Pytorch) achieved 0.685 in the private leaderboard [paper]

Data preparation

Competition data

• The data folder is organized as presented in here, you can also edit this file to adapt to your working directory (not recommended). Instead, this could be directly downloaded from drive by running setup.sh

• The data directory by default is as follow:

├── private_test
│   ├── images
│   ├── readme.md
│   ├── results.csv
│   └── transcriptions.json
├── public_test
│   ├── images
│   ├── results.csv
│   └── transcriptions.json
└── public_train
    ├── additional_infor:train_emotion_polarity.csv
    ├── images
    ├── readme.md
    ├── train_5_folds.csv
    ├── train_emotion_labels.csv
    └── train_transcriptions.json

Additional data (optional)

• In case you want to train a model with static word embeddings (word2vec, glove, fasttext, etc.). Download them by uncommenting the desired pretrained models in setup.sh. By default, static word embedding is not used in our approach
• The provided static embedding models are in pickle file for easy loading, refer prepare_data.sh for more detail

Prerequisites

• tensorflow
• numpy
• pandas
• sklearn
• transformers
• efficientnet

Running setup.sh also installs the dependencies

Train & inference

• Example bash scripts for training and inference are train.sh and infer.sh

Train example

python src/main.py \
    --train_dir data/public_train \
    --target_cols angry disgust fear happy sad surprise neutral other \
    --gpus 0 1 2 \
    --image_model efn-b2 \
    --bert_model roberta-base \
    --word_embedding embeddings/glove.840B.300d.pkl \
    --max_vocab 30000 \
    --image_size 256 \
    --max_word 36 \
    --max_len 48 \
    --text_separator " " \
    --n_hiddens -1 \
    --lr 0.00003 \
    --n_epochs 5 \
    --seed 1710 \
    --do_train \
    --lower \

Inference example

python src/main.py \
    --test_dir data/private_test \
    --target_cols angry disgust fear happy sad surprise neutral other \
    --gpus 1 \
    --ckpt_dir outputs/efn-b2_256_roberta-base_48_-1_0.1/ \
    --do_infer \

• In addition, we perform stacking by Logistic Regression, requires out-of-fold along with test prediction

Outputs

• Folder containing all TF experiments

Reproducing:

• Technical report: English / Vietnamese

• Best single model (0.676 ROC-AUC) configuration

icdar-emoreccom's Issues

Google Drive link not available

I'm sorry to bother you.
When I try to run the code for this repository, I see that the download of the dataset times out.
Could you please check if the link is valid.
Thanks.

Separate train-infer

Restore models from checkpoints directory without passing arguments, these hyper-parameters should be explicitly inferred from experiment name

pseudo labelling

Make pseudo labels (refer here for an example)

Static word embedding

Combining BERT with Static Word Embeddings (refer this paper for more detail)
Change needed:

• build_model()
• ICDARGenerator

Private test inference

Add private test inference pipeline

Duplicated variable names

Resolve duplication in variable name, especially in case of jupyter enviroment experiment
Examples:

# Adapt working directory
TRAIN_IMAGE_DIR = os.path.join(args.data_dir, TRAIN_IMAGE_DIR)
TEST_IMAGE_DIR = os.path.join(args.data_dir, TEST_IMAGE_DIR)
TRAIN_LABELS = os.path.join(args.data_dir, TRAIN_LABELS)
TRAIN_POLARITY = os.path.join(args.data_dir, TRAIN_POLARITY)
TRAIN_SCRIPT = os.path.join(args.data_dir, TRAIN_SCRIPT)
TEST_SCRIPT = os.path.join(args.data_dir, TEST_SCRIPT)
SAMPLE_SUBMISSION = os.path.join(args.data_dir, SAMPLE_SUBMISSION)

tqdm

Import tqdm from tqdm.auto instead of tqdm for both jupyter and terminal envs
Change needed:

• tqdm

Private test pipeline

Add private test pipeline

Minor bug in inference

Save prediction into checkpoint directory instead of provided output arguments
Eg : OUTPUT_DIR -> args.ckpt_dir

pred = np.mean(preds, axis=0)
test_processed[TARGET_COLS] = pred
result_path = os.path.join(OUTPUT_DIR, "results.csv")
test_processed[["image_id"] + TARGET_COLS].to_csv(result_path, header=False)
test_path = os.path.join(OUTPUT_DIR, "test_pred.npy")
with open(test_path, "wb") as f:
        np.save(f, test_processed[TARGET_COLS].values)
oof_df = pd.DataFrame.sort_index(pd.concat(oof_preds, axis=0))
oof_path = os.path.join(OUTPUT_DIR, "oof_pred.npy")
with open(oof_path, "wb") as f:
        np.save(f, oof_df[TARGET_COLS].values)