Feature extraction process?

[genbing99]

Feature extraction process?

[genbing99]

We use exactly the same opt flow algorithm as the SOFTNet work, but the ROI cropping is not applied in this work. So the process is first extracting opt flow features, then resizing to 28x28.

[xjtupanda]

We use exactly the same opt flow algorithm as the SOFTNet work, but the ROI cropping is not applied in this work. So the process is first extracting opt flow features, then resizing to 28x28.

Now I understand it. Thank you so much, and I wish you success in your research!

[xjtupanda]

Sorry to bother you again. When reproducing the result, I'm not sure if I'm in the right way, and I want to ask a few questions.

1. Did you subtract opt. flow of nose region from the whole opt. flow features and do the eye masking? Because I believe these two are key to reducing noise induced by head shaking and eye blinking respectively. If so, I think the preprocessing pipeline is: Face detection and cropping --> optical flow extraction --> subtract global movement and do eye masking --> resize to 28 x 28 feature maps.
2. I noticed that by training ME & MaE frames together, the total number of positive samples get huge gains. But according to my experiments in SoftNet the data augmentation on ME samples is very crucial to the model performance. Though in MTSN's setting the amount of positive sample gets larger, i.e., the imbalance between positive & negative samples is alleviated to some extent, the imbalance between ME & MaE samples arises. (ME overwhelmed by MaE). And I guess this is part of the reason why the detection performance on ME is not so good as SoftNet. Is my understanding correct? Please correct me if I'm getting anything wrong.

[genbing99]

No problem.

1. Nose subtraction is used in the pre-processing, but eye masking is not applied.
2. Some parts are not so accurate.

But according to my experiments in SoftNet the data augmentation on ME samples is very crucial to the model performance.

Through my post-experiments of SOFTNet, I realized that data augmentation is not an important factor in performance. The major factor is the dataset imbalance between expression vs non-expression frames, hence I set a ratio of 1:1 in the MTSN paper.

And I guess this is part of the reason why the detection performance on ME is not so good as SoftNet. Is my understanding correct?

Yes. In fact, spotting ME is more difficult than spotting MaE. Because of the challenge, we are slightly biased on the MaE spotting, so the overall performance can be higher.

[xjtupanda]

No problem.

1. Nose subtraction is used in the pre-processing, but eye masking is not applied.
2. Some parts are not so accurate.

But according to my experiments in SoftNet the data augmentation on ME samples is very crucial to the model performance.

Through my post-experiments of SOFTNet, I realized that data augmentation is not an important factor in performance. The major factor is the dataset imbalance between expression vs non-expression frames, hence I set a ratio of 1:1 in the MTSN paper.

And I guess this is part of the reason why the detection performance on ME is not so good as SoftNet. Is my understanding correct?

Yes. In fact, spotting ME is more difficult than spotting MaE. Because of the challenge, we are slightly biased on the MaE spotting, so the overall performance can be higher.

Thanks a lot! I think I've got more inspiration and insight into the task.