bcv-uniandes/aunets

Pytorch implementation of Multi-View Dynamic Facial Action Unit Detection, Image and Vision Computing (2018)

aunets's Issues

Demo wrong result

AU12 - OF None | Forward | ./Demo\000.jpg : 0.5
AU12 - OF None | Forward | ./Demo\001.jpg : 0.5
AU12 - OF None | Forward | ./Demo\002.jpg : 0.5
AU12 - OF None | Forward | ./Demo\003.jpg : 0.5
AU12 - OF None | Forward | ./Demo\004.jpg : 0.5
AU12 - OF None | Forward | ./Demo\005.jpg : 0.5
AU12 - OF None | Forward | ./Demo\006.jpg : 0.5
AU12 - OF None | Forward | ./Demo\007.jpg : 0.5
AU12 - OF None | Forward | ./Demo\008.jpg : 0.5
AU12 - OF None | Forward | ./Demo\009.jpg : 0.5

Process finished with exit code 0

I run the Demo but got 0.5 for all images. Can you release your pretrained model ? I want test it in other dataset

Weights link expired

Thanks for the work, could you please update the link to your weights, that's expired?

Model weights

Hi,

Thanks for all the great work! I'd love to try it out. Unfortunately, I see that link in #1 is expired. Could you please reupload?

main.py run: error: argument {DeepFaceLive}: invalid choice: 'deepfacelive' (choose from 'DeepFaceLive')

main.py run: error: argument {DeepFaceLive}: invalid choice: 'deepfacelive' (choose from 'DeepFaceLive')

Pre-trained model please.

Hello, thanks very much for your sharing of your codes. Can you please share your pre-trained models pls.

Actually, I have a question. As you said in the paper there are two kinds of networks: Hydra-Net and AUNets, alright? So for Hydra-Net there are 12 models, but how abouts AUNets? I means there are 36G model, are they only for Hydra-Net? if they are all two large, can you please share the convolutional
encoder?

Questions about the preprocessing

Are you performing image alignment prior to calculating the opticalflow? I found some code on the repository related to alignment and did not quite understand what technique you used for the alignment. I am trying to see how your models perform on uncontrolled environment data i have, so i want to know, if i should align before running the models?
Why are you calculating the mean face?
How are you performing the view point classification?

Regarding Optical Flow

Hello, thank you for sharing the code. I wanted to know when you calculate optical flow you calculate using two consecutive frames as input. My doubt is if you have 300 frames of RGB images in total, there would be 299 optical flow images. So when you pass the input to the model do you ignore the first RGB frame pr the last RGB frame since we give a concatenated input of RGB and optical flow image(as the number of OF images would be one number less than RGB images). I would be so grateful if you could clarify this. Thank you once again.

emotionnet trained model

Thanks for your shared pretrained models. would you please share the models trained on emotionnet too? I find using ENet fintune can improve around10 percentage than INet in Table 1 in your paper so these models are also important.

GPU memory error

Hello, I've been able to set up the entire BP4D dataset for training using the correct txt files. However, a memory error always comes up after preprocessing. My GPU memory is capped at 8.1 GB, and even when I try decreasing the batch size down to 4, it yields the same error.
The code output is (batch size set to 4)
"RuntimeError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 7.93 GiB total capacity; 7.22 GiB already allocated; 14.06 MiB free; 7.42 GiB reserved in total by PyTorch)".

I receive the same error even when I set the batch size down to 1, and monitoring my GPU RAM I see that it reaches close to it's capacity before the error.
I also tried running it without GPU on a 126 GB RAM server, and it steadily increased RAM usage until the training was "killed".

Do you know if there is something in the training scheme that I should change to prevent this? Many thanks in advance, and for taking the time to read this.

BroxDir missing

I tried to download the BroxDir file by clicking the link you provided, but It doesn't seem to work.
The link I am talking about is into the ./generate_date folder.
Could you please fix it?

Thank you in advance.

Input data format

Hello, thank you very much for sharing your code.
My question will be very basic. While training how the structure of input data should be organized? So I've got full BP4D dataset and how I can get train.txt and val.txt files. I'll be very grateful for answer.

main.py: error: unrecognized arguments

Hi again, I finally have al the dependencies down, but the demo doesn't seem to work for me.

~/Desktop/AUNets-master$ ./main.sh -AU 12 -gpu 0 -fold 0 -OF None -DEMO Demo
./main.py -- --AU=12 --fold=0 --GPU=0 --OF None --DEMO Demo --batch_size=117 --finetuning=emotionnet --mode_data=normal
usage: main.py [-h] [--image_size IMAGE_SIZE] [--lr LR]
               [--batch_size BATCH_SIZE] [--dataset {BP4D}]
               [--num_epochs NUM_EPOCHS] [--num_epochs_decay NUM_EPOCHS_DECAY]
               [--stop_training STOP_TRAINING] [--beta1 BETA1] [--beta2 BETA2]
               [--num_workers NUM_WORKERS] [--HYDRA] [--DELETE] [--TEST_TXT]
               [--TEST_PTH]
               [--OF {None,Alone,Horizontal,Vertical,Channels,Conv,FC6,FC7}]
               [--test_model TEST_MODEL] [--mode {train,val,test,sample}]
               [--use_tensorboard] [--SHOW_MODEL] [--GPU GPU]
               [--metadata_path METADATA_PATH] [--log_path LOG_PATH]
               [--model_save_path MODEL_SAVE_PATH]
               [--results_path RESULTS_PATH] [--fold FOLD]
               [--mode_data {normal,aligned}] [--AU AU]
               [--finetuning {emotionnet,imagenet,random}]
               [--pretrained_model PRETRAINED_MODEL] [--DEMO DEMO]
               [--log_step LOG_STEP]
main.py: error: unrecognized arguments: -- --AU=12 --fold=0 --GPU=0 --OF None --DEMO Demo --batch_size=117 --finetuning=emotionnet --mode_data=normal

Simply copied your command into the terminal but with no luck.

Model Weights

Hello,

I am a master student in the field of Face expression recognition at Ecole de Technologie Superieure (ETS) of Montreal. I would be interesting in obtaining model weights for AUNets. Could you provide them ? Thanks a lot for sharing.

Best Regards,

Thomas TEIXEIRA
Master Student at Ecole de Technologie Superieure (Montreal)

Issues while running demo

Hi,

I'd like to run a demo of AUNets. I've downloaded all folds and put them in my home directory. I've made some changes to source files, as it seems they were needed:

diff --git a/main.py b/main.py
index 5b83276..40b2d6d 100755
--- a/main.py
+++ b/main.py
@@ -1,4 +1,4 @@
-#!/usr/local/bin/ipython
+#!/usr/bin/ipython
 import os
 import argparse
 from data_loader import get_loader
diff --git a/models/vgg_pytorch.py b/models/vgg_pytorch.py
index 3c0c135..910eb60 100644
--- a/models/vgg_pytorch.py
+++ b/models/vgg_pytorch.py
@@ -384,7 +384,7 @@ def vgg16(pretrained='', OF_option='None', model_save_path='', **kwargs):
     sheet={k.encode("utf-8"): v for k,v in model_zoo_.iteritems()}
 
   elif pretrained=='emotionnet' and OF_option=='None':
-    emo_file = sorted(glob.glob('/home/afromero/datos2/EmoNet/snapshot/models/EmotionNet/normal/fold_all/Imagenet/*.pth'))[-1]
+    emo_file = sorted(glob.glob('/home/lsowa/fold_0/OF_Horizontal/*.pth'))[-1]
     model_zoo_ = torch.load(emo_file)
     # print("Finetuning from: "+emo_file)
     model_zoo_={k.replace('model.',''): v for k,v in model_zoo_.iteritems()}
@@ -484,4 +484,4 @@ def vgg16(pretrained='', OF_option='None', model_save_path='', **kwargs):
       model.load_state_dict(model_zoo_2) 
       # ipdb.set_trace()    
 
-  return model
\ No newline at end of file
+  return model
diff --git a/requirements.txt b/requirements.txt
index ccce7b5..16e2f75 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -11,12 +11,11 @@ tqdm==4.11.2
 scikit_image==0.10.1
 torchvision==0.2.0
 ipdb==0.10.1
-pytorchviz==0.0.1
+torchviz==0.0.1
 Wand==0.4.4
 matplotlib==1.5.1
 Pillow==5.1.0
 matlab==0.1
 ops==0.4.7
-skimage==0.0
 scikit_learn==0.19.1
 xlsxwriter==1.0.4

it's some package fixing and substituting your local path, by my local path pointing to the fold_0 weights. While running demo like this ./main.sh -AU 12 -gpu 0 -fold 0 -OF None -DEMO /home/lsowa/demo/ (images are raw screens from my webcam) I encounter error:

./main.py -- --AU=12 --fold=0 --GPU=0 --OF None --DEMO /home/lsowa/demo/ --batch_size=117 --finetuning=emotionnet --mode_data=normal
Namespace(AU='12', DELETE=False, DEMO='/home/lsowa/demo/', GPU='0', HYDRA=False, OF=False, OF_option='None', SHOW_MODEL=False, TEST_PTH=False, TEST_TXT=False, batch_size=117, beta1=0.5, beta2=0.999, dataset='BP4D', finetuning='emotionnet', fold='0', image_size=224, log_path='./snapshot/logs/BP4D/normal/fold_0/AU12/OF_None/emotionnet', log_step=2000, lr=0.0001, metadata_path='./data/BP4D/normal/fold_0/AU12', mode='train', mode_data='normal', model_save_path='./snapshot/models/BP4D/normal/fold_0/AU12/OF_None/emotionnet', num_epochs=12, num_epochs_decay=13, num_workers=4, pretrained_model='', results_path='./snapshot/results', stop_training=2, test_model='', use_tensorboard=False, xlsfile='./snapshot/results/normal/emotionnet.xlsx')
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/home/lsowa/AUNets/main.py in <module>()
    119 
    120   print(config)
--> 121   main(config)

/home/lsowa/AUNets/main.py in main(config)
     39   # Solver
     40   from solver import Solver
---> 41   solver = Solver(rgb_loader, config, of_loader=of_loader)
     42 
     43   if config.SHOW_MODEL:

/home/lsowa/AUNets/solver.pyc in __init__(self, rgb_loader, config, of_loader)
     93     # Build tensorboard if use
     94     if config.mode!='sample':
---> 95       self.build_model()
     96       if self.SHOW_MODEL: return
     97       if self.use_tensorboard:

/home/lsowa/AUNets/solver.pyc in build_model(self)
    151     if self.TEST_TXT: return
    152     from models.vgg16 import Classifier
--> 153     self.C = Classifier(pretrained=self.finetuning, OF_option=self.OF_option, model_save_path=self.model_save_path)
    154 
    155     trainable_params, name_params = self.get_trainable_params()

/home/lsowa/AUNets/models/vgg16.pyc in __init__(self, pretrained, OF_option, model_save_path)
     30     self.model_save_path = model_save_path
     31 
---> 32     self._initialize_weights()
     33 
     34   def _initialize_weights(self):

/home/lsowa/AUNets/models/vgg16.pyc in _initialize_weights(self)
     36     if 'emotionnet' in self.finetuning:
     37       mode='emotionnet'
---> 38       self.model = model_vgg16(pretrained=mode, OF_option=self.OF_option, model_save_path=self.model_save_path, num_classes=22)
     39       modules = self.model.modules()
     40       for m in modules:

/home/lsowa/AUNets/models/vgg_pytorch.pyc in vgg16(pretrained, OF_option, model_save_path, **kwargs)
    403     if pretrained:
    404       # ipdb.set_trace()
--> 405       model.load_state_dict(model_zoo_)
    406 
    407   #====================================================================================================#

/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.pyc in load_state_dict(self, state_dict, strict)
    517                                        'whose dimensions in the model are {} and '
    518                                        'whose dimensions in the checkpoint are {}.'
--> 519                                        .format(name, own_state[name].size(), param.size()))
    520             elif strict:
    521                 raise KeyError('unexpected key "{}" in state_dict'

RuntimeError: While copying the parameter named classifier.0.weight, whose dimensions in the model are torch.Size([4096, 25088]) and whose dimensions in the checkpoint are torch.Size([4096, 50176]).

Do you know why's that? Is it something to do with the fact that I'm using weights for OF=Horizontal and I'm passing in images without OF? How to fix that?

Error while running DEMO

Hello,
Thank you for sharing your code!
I've got issue while running DEMO. When I run command:
python main.py --AU=1 --fold=0 --GPU=0 --OF Horizontal --DEMO=Demo --mode_data=normal --pretrained_model /home/patryk/git/AU01.pth

I've got this output messages:
Namespace(AU='1', DELETE=False, DEMO='Demo', GPU='0', HYDRA=False, OF=True, OF_option='Horizontal', SHOW_MODEL=False, TEST_PTH=False, TEST_TXT=False, batch_size=118, beta1=0.5, beta2=0.999, dataset='BP4D', finetuning='emotionnet', fold='0', image_size=224, log_path='./snapshot/logs/BP4D/normal/fold_0/AU01/OF_Horizontal/emotionnet', log_step=2000, lr=0.0001, metadata_path='./data/BP4D/normal/fold_0/AU01', mode='train', mode_data='normal', model_save_path='./snapshot/models/BP4D/normal/fold_0/AU01/OF_Horizontal/emotionnet', num_epochs=12, num_epochs_decay=13, num_workers=4, pretrained_model='/home/patryk/git/AU01.pth', results_path='./snapshot/results', stop_training=2, test_model='', use_tensorboard=False, xlsfile='./snapshot/results/normal/emotionnet.xlsx')
[!!] loaded trained model: /home/patryk/git/AU01.pth!
Killed

It happend multiple times but once my pycharm was closed and there was error:
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5d47cda000, 16384, 0) failed; error='Not enough space' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 16384 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/patryk/java_error_in_PYCHARM_2427.log
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00007f5d08d63000, 16384, 0) failed; error='Not enough space' (errno=12)
[thread 8672 also had an error]

[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f5c95fde994]

Is this related to operating system? What OS did you use?
I'm using C python 2.7 and Ubuntu 16.04.

Thanks!

API

Hi !
I have 2 doubts:-

1. Is there an API available that allows me to run this as a part of an existing python script?
2. Can this work on videos?

Thanks a lot for your great work!

Sharing Pre-Trained Model wo OF

If possible, could you share pre-trained model without OF images?
I understood that the pre-trained model you shared was trained with RGB + OF images.
I want to compare the detection performance between w OF and wo OF on other database.
Thanks,

issue with dropout for the inference

Hi,

it's a nice and interesting work. But I have a question.

It seems the dropout probability stays at 0.5 for different layers (default value) for the inference with DEMO option (and not only for the training) . Is is normal or did I make a mistake?

Thanks in advance for the answer.

Model weights

I am asking for model weights. Thank you very much for sharing your work.

咨询下BP4D数据集怎么获取

Model weights

hello. Thank you for uploading.
I want to try AU-Nets. Please give me pre-trained model?

Thank you in advance.

weight model and info

Hello, thank you for your contribution.
Can you please share the weights with me?

Also are these the main AU usually used for emotion recognition? I know there are other but I rarely see them in AU model prediction.

main.py run: error: argument {DeepFaceLive}: invalid choice: 'deepfacelive' (choose from 'DeepFaceLive')

main.py run: error: argument {DeepFaceLive}: invalid choice: 'deepfacelive' (choose from 'DeepFaceLive')

Processing images

Hi,

I am really interested in your project. I was wondering if there is a way to test the model on single images or even the webcam.

The first thing I need is to load the model from disk. I downloaded the .pth files you provided, but I am not sure how to load them and reconstruct the network.
My folder structure is:
models:
-file.pth
-fold_0
--OF_Horizontal
---AU01.pth
---AU02.pth
---...
-fold_1
--OF_Horizontal
---AU01.pth

After that it should be quite straight forward to send images into the network and visualize the output.

Thanks a lot,

Julian

KeyError::

Hello, when i am trying to processes this git, i get the following error
Keyerror: '\x1f'
i am sharing the screen shot.

RealTime using computer-vision

Is it possible to classify different classes using AU's like happiness, uneasy and all other emotions.

Bad Interpreter: No such file or directory

$ ./main.sh -AU 12 -gpu 0 -fold 0 -OF Horizontal -DEMO Demo
./main.py -- --AU=12 --fold=0 --GPU=0 --OF Horizontal --DEMO Demo --batch_size=24 --finetuning=emotionnet --mode_data=normal
./main.sh: ./main.py: /usr/local/bin/ipython: bad interpreter: No such file or directory

I'm running Python 2.7 because I'm using a pre-installed VM. Installed IPython 5.8.0 which should be compatible with Python 2.7, but the error persists. Any help is appreciated.

Weights demo error

`/AUNets/venv/lib/python3.6/site-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
828 if len(error_msgs) > 0:
829 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
--> 830 self.class.name, "\n\t".join(error_msgs)))
831 return _IncompatibleKeys(missing_keys, unexpected_keys)
832

RuntimeError: Error(s) in loading state_dict for Classifier:
size mismatch for model.classifier.0.weight: copying a param with shape torch.Size([4096, 50176]) from checkpoint, the shape in current model is torch.Size([4096, 25088]).
`

./main.sh -AU 04 -gpu 0 -fold 0 -OF None -DEMO Demo
./main.py -- --AU=04 --fold=0 --GPU=0 --OF None --DEMO Demo --batch_size=117 --finetuning=emotionnet --mode_data=normal

changed '--pretrained_model', '--test_model' to './fold_0/fold_0/OF_Horizontal/AU04.pth'

Data formatting issues

Hi, first of all, great work and thank you for providing the full implementation of your project! My question is regarding the data organization. I've seen your answer on how to prepare the text file for RGB and optical flow images from issue #22,

"It is quite simple, it must have the following form:

Faces/file_name0.jpg 0
Faces/file_name1.jpg 1
Faces/file_name2.jpg 1
Faces/file_name3.jpg 0
...

And a different folder with the pre-calculated Optical Flow

Faces_Flow/file_name0.jpg 0
Faces_Flow/file_name1.jpg 1
Faces_Flow/file_name2.jpg 1
Faces_Flow/file_name3.jpg 0
...

Please take a look at data_loader script and change variables accordingly."

I only have a few uncertainties:
a) Although I have all the image and optical flow names in two separate text files, will I need to place all of the BP4D images in one directory as well, or should I keep them as they are in their respective subject folders and just make sure that my root path is correctly defined?
b) What do the numbers after the file names represent?
c) I was able to retrieve the three-fold information that you use. To do training on the same three folds, will I need to make three separate text files for each fold, or sort the images in different directories for each time?

Thank you very much for your help, any help would be appreciated!