Wrong train/test split for the ShanghaiTech dataset

[c0510gy]

Wrong train/test split for the ShanghaiTech dataset

[luca-zanella-dvl]

Hi Sanggeon,

In order to ensure a fair comparison between AnomalyCLIP and other video anomaly detection (VAD) baseline methods, we adopt ShanghaiTech in the configuration of Zhong et al. [1], which adapts it to the weakly-supervised setting, as mentioned in section 4.1 of the paper. As AnomalyCLIP performs joint VAD and video anomaly recognition (VAR), we complement this configuration with class label information.

[1] Zhong, Jia-Xing, et al. "Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[c0510gy]

Thank you for answering, we already checked the paper you mentioned, and we found the same problem on their split.
Specifically, for the "Fall" class, there are only four videos which are all located in the Anomaly_Test.txt file in your pre-extracted feature data (https://drive.google.com/file/d/1jbpK1jQkYgZMXF_DyIGy__0ZKYrSoUsj/view?usp=drive_link) and non in the Anomaly_Train_Abnormal.txt:

fall/03_0032
fall/03_0033
fall/04_0004
fall/07_0006

We found similar problems in the following classes: Circuit, Fall, Jumping, Monocycle, Push, Stoop where Circuit, Jumping, Push, Stoop has a different problem that exists in the videos where multiple anomalies appear. On training, since we need to use a weakly supervised learning method, in which we only use video-labeled data, I'm curious how you use the videos with the multiple anomalies appearing during training.
Thanks,

[luca-zanella-dvl]

Yes, we also observed that some classes in the test set do not appear in the training set or vice versa. This is probably because ShanghaiTech was not originally designed for VAR. Therefore, future research, particularly that focused on VAR, could benefit from developing a more appropriate data split for the ShanghaiTech dataset.

Regarding multiple classes of anomalies in the same video, you are right. We treated this special case by dividing videos with multiple anomalies into multiple videos with a single anomaly. Specifically, we calculate the difference in frames between the last occurrence of one type of anomaly and the first occurrence of the next, and divide the video evenly according to this number.

You can see this in both Anomaly_Train_Abnormal.txt and Anomaly_Test.txt, located in the same Google Drive folder containing the pre-extracted features, where videos that contain multiple anomalies, such as 03_0035, are divided into

chase/03_0035_0 0 165 1
jump/03_0035_1 166 384 6

[c0510gy]

I see, thank you for your detailed answer, we will take that into account in our research.
Great work! I enjoyed reading your paper.

[luca-zanella-dvl]

Thank you for your interest in our work, I highly appreciate it :) Also, thanks for raising this issue, in the next few days I will either upload the script to create the ShanghaiTech annotation files or the videos already divided into categories for better reproducibility.