Problems on result reproduction on MEGC2021 Benchmark

[genbing99]

Problems on result reproduction on MEGC2021 Benchmark

[xjtupanda]

I've also uploaded my notebook of reproduction.https://drive.google.com/file/d/1L-8EHmPC3HIj5Swd_aArrcCjTGZ99Wc6/view?usp=share_link

The only modification is on some evaluation detail, if len(preds_micro) == 0: preds_micro.append([0, 0, 0, 0, 0, -1])#, 0]) # -1 to bypass the count of additional fp
Because the evaluation function requires each pred item of length 6 and GT item of length 7, so I remove the last element in each item. The original notebook has length 7 and 8 respectively though.

[xjtupanda]

After a closer inspection, I think the main reason for the bad result is the low F1-score of micro-expression detection, (I still don't understand why I can't reproduce the results though.) I've found many of my results producing few TPs and large FPs for micro-expression, and thus dragging the result of macro-expression. And there are so many hyper-parameters for spotting to tune, which I believe is crucial to the final result. Is there any search strategy or just tuning by viewing the score plot?

[genbing99]

Thanks for your investigation, especially the typo in the original codes.

Please refer to the Jupyter Notebook as I might wrongly copy some parts when converting to py files.

For the modification that you did:
preds_micro.append([0, 0, 0, 0, 0, -1])#, 0]) # -1 to bypass the count of additional fp
You should use the line in the original code:
preds_micro.append([0, 0, 0, 0, 0, -1, 0]) # -1 to bypass the count of additional fp
Otherwise, you will encounter an issue of adding an FP if no prediction is made on the particular video.
See that the FP:7 for the evaluation of CASME_sq Subject 1. In fact, it should be FP:0.

[genbing99]

The hyperparameters were set by using a loop, which will take some time. I think this is normal to do so in the spotting task, and other methods also used a lot of hyperparameters while processing the "signal" graph.

[xjtupanda]

The hyperparameters were set by using a loop, which will take some time. I think this is normal to do so in the spotting task, and other methods also used a lot of hyperparameters while processing the "signal" graph.

I see. Thank you again for your patience.