genbing99/mean_spot-then-recognize

A Micro-Expression Analysis Network (MEAN) to spot-then-recognize micro-expressions

MEAN architecture

Characteristics:

• Shallow network
• Multi-stream
• Multi-output with two task-specific networks
• Capable of both micro-expression spotting and recognition

Results

Take note that analysis is equivalent to spot-then-recognize and STRS is our proposed evaluation metric.
Please refer to the paper for more experiment results.
Here is the detailed result for micro-expression spotting (S) and analysis (A):

How to run the code

Step 1) Download the micro-expression datasets for experiment, we suggest the files to be structured as follows:

├─MEAN_Weights
├─Utils
├─define_model.py
├─face_crop.py
├─feature_extraction.py
├─load_excel.py
├─load_images.py
├─main.py
├─prepare_training.py
├─train_evaluate.py
├─training_utils.py
├─requirements.txt
├─CASME_sq

├─code_final.xlsx
├─rawpic

├─CASME2

├─CASME2

├─CASME2-RAW
└─CASME2_label_Ver_2.xls

├─SAMM

└─SAMM_20181215_Micro_FACS_Codes_v2.xlsx

├─SAMMLV

├─SAMM_longvideos
└─SAMM_LongVideos_V2_Release.xlsx

└─SMIC

├─SMIC-E_raw image
├─HS_long

├─SMIC-HS-E

├─HS
└─SMIC-HS-E_annotation.xlsx

├─NIR_long

└─SMIC-NIR-E

├─NIR
└─SMIC-NIR-E_annotation.xlsx

└─VIS_long

└─SMIC-VIS-E

├─VIS
└─SMIC-VIS-E_annotation.xlsx

Step 2) Installation of packages using pip

pip install -r requirements.txt

Step 3) MEAN Training and Evaluation

python main.py

  Note for parameter settings

   --dataset_name (CASME_sq/SAMMLV/CASME2/SMIC_HS/SMIC_VIS/SMIC_NIR)
   --train (True/False)

Additional Notes

If you find this work useful, please cite the paper:

@article{liong2023spot,
title={Spot-then-Recognize: A Micro-Expression Analysis Network for Seamless Evaluation of Long Videos},
author={Liong, Gen-Bing and See, John and Chan, Chee-Seng},
journal={Signal Processing: Image Communication},
volume={110},
pages={116875},
year={2023},
publisher={Elsevier}
}

Please email me at [email protected] if you have any inquiries or issues.

mean_spot-then-recognize's Issues

How do I recognize microexpressions use my own test examples?

Dear author, it is my great honor to star your project. I would like to ask you how to use your project to recognize micro-expressions in videos or pictures

Inconsistent results

Dear author, it is an honor to have read your code. I would like to ask why my dataset and code are the same as yours, and why there is so much difference in accuracy after complete training. I trained myself twice using the dataset SMIC-NIR and SMIC-VIS, with SMIC-NIR having an Accuracy Score of 0.3684 and 0.5294, and SMIC-VIS having an Accuracy Score of 0.6667 and 0.4828, respectively, which is somewhat different from the accuracy in the author's paper, May I ask why this is

Recognition without Spotting

can you help me to run special training on micro expression recognition without spotting (pseudo labeling). I found an error like this

"ValueError: Unexpected result of train_function (Empty logs). Please use Model.compile(..., run_eagerly=True), or tf.config.run_functions_eagerly(True) for more information of where went wrong, or file a issue/bug to tf.keras."