rakutentech/fau_cvpr2021

Facial Action Unit Detection with Transformers

This repository contains Tensorflow training code for the CVPR 2021 paper:

• Facial Action Unit Detection with Transformers

For details see Facial Action Unit Detection with Transformers by Geethu Miriam Jacob, Bjorn Stenger If you use this code for a paper please cite:

@inproceedings{jacob2021facial,
  title={Facial Action Unit Detection With Transformers},
  author={Jacob, Geethu Miriam and Stenger, Bjorn},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={7680--7689},
  year={2021}
}

Inference

Download pretrained models (BP4D dataset) from the following links:

Pre-trained model fold1

Pre-trained model fold2

Pre-trained model fold3

python inference.py

Usage

First, clone the repository locally:

git clone https://github.com/RIT/FAU_transformers.git

Then, install the packages in requirements file:

pip install -r requirements.txt

Data preparation

python Prepare_data.py

Train models

python main.py

Demo colab link

License

This repository is released under the Apache 2.0 license as found in the LICENSE file.

Contributing

We welcome your pull requests! Please see CONTRIBUTING.md and CODE_OF_CONDUCT.md for more info.

fau_cvpr2021's Issues

Testing with webcam

I tried inference model with webcam BGR input frame, use SCRFD to detect face then crop and convert to GRAY image before feed to the FAU model but the prediction is not consistent with my movement. Could you please share the correct way to preprocess image?

action units maps extraction

hi,
can you please confirm that gh2 is the attention maps?

FAU_CVPR2021/inference.py

Line 53 in 0bfb778

attention = gh2

it is of size (batch_size, 12, 12, number_action_units). here, number_action_units=12.
attention maps are output of the arrow in this fig.3 from https://openaccess.thecvf.com/content/CVPR2021/papers/Jacob_Facial_Action_Unit_Detection_With_Transformers_CVPR_2021_paper.pdf

when plotting one of the attention maps, i am supposed to see something similar to fig1, right?

i run python inference.py and extracted gh2. i plotted all gh2[0, :, :, i] for i in range(12), right to the image. but i am seeing something strange.
below are the plots from 0 to 11.
here are the unique values per map:

map: 0:  [1.]
map: 1:  [0.        0.9982668 1.       ]
map: 2:  [0. 1.]
map: 3:  [1.]
map: 4:  [0.]
map: 5:  [0. 1.]
map: 6:  [1.]
map: 7:  [0.]
map: 8:  [0.]
map: 9:  [0.0000000e+00 2.0861626e-07 9.4047523e-01 1.0000000e+00]
map: 10:  [0.]
map: 11:  [0.]

also strange. the sigmoid could be doing this. but with or without sigmoid, i am supposed to get attentions that point to rois.
can you help? i may be missing something.
can you show how did you plot attentions in fig1?
very much appreciated.
thanks

Recurring problems

Hello, thank you for your code contribution. Could you please send me the final code? I cannot reproduce this code, because there are many undeclared variables in the code.