aimerykong/dimensional-emotion-analysis-of-facial-expression

Dimensional-Emotion-Analysis-of-Facial-Expression

[project page] [paper] [demos] [video] [models] [data] [result]

Shu Kong, last update: 05/23/2018, aimerykong At g-m-a-i-l dot com

Automated facial expression analysis has a variety of applications in human-computer interaction. Traditional methods mainly analyze prototypical facial expressions of no more than eight discrete emotions as a classification task. However, in practice, spontaneous facial expressions in naturalistic environment can represent not only a wide range of emotions, but also different intensities within an emotion family. In such situation, these methods are not reliable or adequate. In this paper, we propose to train deep convolutional neural networks (CNNs) to analyze facial expressions explainable in a dimensional emotion model. The proposed method accommodates not only a set of basic emotion expressions, but also a full range of other emotions and subtle emotion intensities that we both feel in ourselves and perceive in others in our daily life. Specifically, we first mapped facial expressions into dimensional measures so that we transformed facial expression analysis from a classification problem to a regression one. We then tested our CNN-based methods for facial expression regression and these methods demonstrated promising performance. Moreover, we improved our method by a bilinear pooling which encodes second-order statistics of features. We showed such bilinear-CNN models significantly outperformed their respective baselines.

Keywords Dimensional Emotion Model, Fine-Grained Analysis, Facial Expression, High-Order Correlation, Psychology, Affective-Cognitive Computing, Physiological Computing.

For visualization and training, please download all files from google drive, and put them under directory './transcript/exp/', so that the scripts can fetch models to train, test, or fine-tune. For training details, please go to './trainscrip/'.

The dataset in the dimensional emotion space can be downloaded in this google drive. Please download it and put it under folder './trainscript/' in order to evaluate or train models.

All the results reported in the paper can be found in this google drive.

MatConvNet is used in our project; please compile a modified version under folder ``libs'' accordingly --

LD_LIBRARY_PATH=/usr/local/cuda/lib64:local matlab 

path_to_matconvnet = './libs/matconvnet-1.0-beta23_modifiedDagnn/';
run(fullfile(path_to_matconvnet, 'matlab', 'vl_setupnn'));
addpath(fullfile(path_to_matconvnet, 'matlab'));
vl_compilenn('enableGpu', true, ...
               'cudaRoot', '/usr/local/cuda', ...
               'cudaMethod', 'nvcc', ...
               'enableCudnn', true, ...
               'cudnnRoot', '/usr/local/cuda/cudnn/lib64') ;

If you find our model/method/dataset useful, please cite our work:

@inproceedings{Kong2018dimensionalEmotion,
  title={Fine-Grained Facial Expression AnalysisUsing Dimensional Emotion Model},
  author={Zhou, Feng and Kong, Shu and Fowlkes, Charless and Chen, Tao, and Lei, Baiying},
  booktitle={arxiv 1805.01024},
  year={2018}
}

dimensional-emotion-analysis-of-facial-expression's Issues

Some problems when compiling matconvnet

Hello,
I am trying to use your model to do an experiment, but some problems occured to me. My version is Win10, Matlab2018b and VS2017 community. I was compiling MatConvNet. Here is the description:
错误使用 mex
ucrt.lib(api-ms-win-crt-math-l1-1-0.dll) : error LNK2005: round 已经在 roipooling_cpu.obj 中定义
正在创建库
D:\Dimensional-Emotion-Analysis-of-Facial-Expression-master\Dimensional-Emotion-Analysis-of-Facial-Expression-master\libs\matconvnet-1.0-beta23_modifiedDagnn\matlab\mex\vl_imreadjpeg.lib
和对象
D:\Dimensional-Emotion-Analysis-of-Facial-Expression-master\Dimensional-Emotion-Analysis-of-Facial-Expression-master\libs\matconvnet-1.0-beta23_modifiedDagnn\matlab\mex\vl_imreadjpeg.exp
D:\Dimensional-Emotion-Analysis-of-Facial-Expression-master\Dimensional-Emotion-Analysis-of-Facial-Expression-master\libs\matconvnet-1.0-beta23_modifiedDagnn\matlab\mex\vl_imreadjpeg.mexw64
: fatal error LNK1169: 找到一个或多个多重定义的符号

出错 vl_compilenn>mex_link (line 547)
mex(mopts{:}) ;

出错 vl_compilenn (line 498)
mex_link(opts, objs, mex_dir, flags.mexlink) ;

What can I do to solve the problem?

a question about dimensional labels

Thanks for sharing your work!

Could you elaborate more on how are the dimensional labels for each image derived? The labels provided in your imdb_DimEmotion.mat seem to be normalized between 0 and 1 but from the equation in your paper it's not normalized.