zengqunzhao/former-dfer

[MM'21] Former-DFER: Dynamic Facial Expression Recognition Transformer

Former-DFER

Zengqun Zhao, Qingshan Liu. "Former-DFER: Dynamic Facial Expression Recognition Transformer". ACM International Conference on Multimedia.

Setup

conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=10.2 -c pytorch

Training on DFEW

• Step 1: download DFEW dataset.
• Step 2: fill in all the your_dataset_path in script.py, then run script.py.
• Step 3: run sh main_DFEW_trainer.sh

Recent Updates

Pretrain Models on DFEW

The trained models on DFER (fd1, fd2, fd3, fd4, fd5) can be downloaded here (Google Driver).

Performance on FERV39k

Recently, a new dynamic FER dataset named FERV39k is proposed, the results of the Former-DFER on FERV39k are as follows:

Happiness	Sadness	Neutral	Anger	Surprise	Disgust	Fear	UAR	WAR
67.57	44.16	51.81	48.93	25.09	10.80	9.80	36.88	45.72

Citation

If you find our work useful, please consider citing our paper:

@inproceedings{zhao2021former,
  title={Former-DFER: Dynamic Facial Expression Recognition Transformer},
  author={Zhao, Zengqun and Liu, Qingshan},
  booktitle={Proceedings of the 29th ACM International Conference on Multimedia},
  pages={1553--1561},
  year={2021}
}

former-dfer's Issues

我在运行main_DFEW.py文件时，显示“页面文件太小，无法完成操作”，请问这是我电脑配置的原因吗？是因为内存不足吗？望回复，谢谢

Generate heatmaps

Hello, is it convenient to upload the code for generating the heatmaps? Thank you.

Unexpected keys in pretrained weight

Thanks for your impressive works!
However, I noticed that there are some unexcepted keys in the pretrained weight you provide. Setting strict=False will do the trick and load_status shows no missing keys. Still, I am wondering if it matters, or you just upload a weight mixing with old parameters.

Unexpected key List:

unexpected_keys: ['s_former.spatial_transformer.layers.1.0.fn.norm.weight', 's_former.spatial_transformer.layers.1.0.fn.norm.bias', 's_former.spatial_transformer.layers.1.0.fn.fn.to_qkv.weight', 's_former.spatial_transformer.layers.1.0.fn.fn.to_out.0.weight', 's_former.spatial_transformer.layers.1.0.fn.fn.to_out.0.bias', 's_former.spatial_transformer.layers.1.1.fn.norm.weight', 's_former.spatial_transformer.layers.1.1.fn.norm.bias', 's_former.spatial_transformer.layers.1.1.fn.fn.net.0.weight', 's_former.spatial_transformer.layers.1.1.fn.fn.net.0.bias', 's_former.spatial_transformer.layers.1.1.fn.fn.net.3.weight', 's_former.spatial_transformer.layers.1.1.fn.fn.net.3.bias', 's_former.spatial_transformer.layers.2.0.fn.norm.weight', 's_former.spatial_transformer.layers.2.0.fn.norm.bias', 's_former.spatial_transformer.layers.2.0.fn.fn.to_qkv.weight', 's_former.spatial_transformer.layers.2.0.fn.fn.to_out.0.weight', 's_former.spatial_transformer.layers.2.0.fn.fn.to_out.0.bias', 's_former.spatial_transformer.layers.2.1.fn.norm.weight', 's_former.spatial_transformer.layers.2.1.fn.norm.bias', 's_former.spatial_transformer.layers.2.1.fn.fn.net.0.weight', 's_former.spatial_transformer.layers.2.1.fn.fn.net.0.bias', 's_former.spatial_transformer.layers.2.1.fn.fn.net.3.weight', 's_former.spatial_transformer.layers.2.1.fn.fn.net.3.bias']

FERV39k数据集的训练模型

请问可以提供在FERV39k上训练后的模型吗，望回复

可以简单解释一下什么是“Dynamic Sampling”吗？文章中这一策略明显提升了性能，是分段采样视频帧？望回复，谢谢。

Inquiry about Pre-training of ResNet18 Used in Your Paper

Hello,

I recently read your paper and am interested in the implementation details of the ResNet18 model you used. Could you please clarify if the ResNet18 model employed in your research was pre-trained on the MS-Celeb-1M dataset? Understanding this detail is crucial for replicating your results and furthering my research.

Thank you for your time and contribution to the field.