Accuracy recurrence issues

[Liu-arch]

Accuracy recurrence issues

[Anirudh257]

Hi @Liu-arch you can look into the preprocessing of data as mentioned in the paper and create the zip file accordingly. Follow the train/test split given in my repo.

[Liu-arch]

Dear author,
I processed the video images via ffmpeg and used your txt file of the train set, test set and the validation set. In addition, place them in folders and name them after the original video name. I'm not sure if the ssv2 dataset should use 10x more videos per class in the training set. But for the other three datasets the pre-processing should be the same as yours without 10x more videos per class in the training set. Could you please help me to see if I am missing some processing steps so that I load the model you uploaded but it does not reproduce the accuracy of the paper exactly?
Best regards

[Anirudh257]

Hi @Liu-arch, what is your current accuracy for the other datasets? What was your procedure for processing datasets?

[Liu-arch]

Dear author, Thank you very much for your reply. Here is my procedure. 10 accuracy tests on each of the three datasets and averaged over 1,500 random test tasks. UCF101(96.22%+-0.2%), HMDB51(75.69%+-0.5%), Kinetics(85.52%+-0.5%) import os import subprocess import json from glob import glob from joblib import Parallel, delayed import sys out_h = 256 out_w = 256 in_folder = r'E:\hmdb51_UCF_videos\dataset_hmdb51' out_folder = r'F:\hmdb_images_ff' split_dir = "splits/hmdb_ARN" wc = os.path.join(split_dir, "*.txt") def run_cmd(cmd): try: os.mkdir(cmd[1]) subprocess.call(cmd[0]) except: pass try: os.mkdir(out_folder) except: pass for fn in glob(wc): classes = [] vids = [] print(fn) if "train" in fn: cur_split = "train" elif "val" in fn: cur_split = "val" elif "test" in fn: cur_split = "test" with open(fn, "r") as f: data = f.readlines() #print(data) for x in data: #print(x) c = x.split('/')[-2].strip() #print(c) v = x.split('/')[-1].strip() #print(v) vids.append(v) classes.append(c) for c in list(set(classes)): try: os.mkdir(os.path.join(out_folder, c)) except: pass cmds = [] print(vids) print(classes) for v, c in zip(vids, classes): print(c) source_vid = os.path.join(in_folder, c,"{}.avi".format(v)) extract_dir = os.path.join(out_folder, c, v) #extract_dir = os.path.join(out_folder, cur_split, c, v) if os.path.exists(extract_dir): continue out_wc = os.path.join(extract_dir, '%08d.jpg') print(source_vid, out_wc) scale_string = 'scale={}:{}'.format(out_w, out_h) os.mkdir(extract_dir) try: cmd = ['ffmpeg', '-i', source_vid, '-vf', scale_string, '-q:v', '5', out_wc] cmds.append((cmd, extract_dir)) subprocess.call(cmd) print("完成") except: # print("fail") # sys.exit() pass Best regards.  ------------------ 原始邮件 ------------------ 发件人: "Anirudh257/strm" ***@***.***>; 发送时间: 2023年4月5日(星期三) 凌晨4:23 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Anirudh257/strm] Accuracy recurrence issues (Issue #27) Hi @Liu-arch, what is your current accuracy for the other datasets? What was your procedure for processing datasets? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Anirudh257]

Hi @Liu-arch, check:

1. #9 (comment) for data processing steps.

2. Train for more iterations and also look into ViT backbone.

[Liu-arch]

Dear author, Thank you very much for your reply. 1.I know that your processing is consistent with TRX, so I have referenced TRX. And the parameters are the same as his. But the UCF is 95.89%，HMDB is 73.93%， Kinetics is 84.78 over 10,000 random test tasks When loading your best model to test accuracy with resnet 50. Only SSv2 achieved the accuracy rate in the text. I don't understand. Please help me! 2. I'm not using VIT at the moment because I haven't reached the targets in the paper with resnet50 as the backbone. Best regards.

…

------------------ 原始邮件 ------------------ 发件人: "Anirudh257/strm" ***@***.***>; 发送时间: 2023年4月6日(星期四) 凌晨0:55 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Anirudh257/strm] Accuracy recurrence issues (Issue #27) Hi @Liu-arch, check: #9 (comment) for data processing steps. Train for more iterations and also look into ViT backbone. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Liu-arch]

For vit, which vit do you use? vit_b_16, vit_b_32, vit_l_16, vit_l_32?

[Anirudh257]

Hi @Liu-arch You can try with VIT_B_16. For HMDB, Kinetics, just train for more episodes(20K/30K, etc) and then perform tests.

[Liu-arch]

Dear author, Thank you very much for your reply. After several conversations with you, I still haven't reproduced the accuracy of network architecture with resnet50 as the backbone with the optimal model you uploaded to git. So you think it is easier to reproduce the accuracy with vit as backbone? But there is no optimal model in git. Best regards.

…

------------------ 原始邮件 ------------------ 发件人: "Anirudh257/strm" ***@***.***>; 发送时间: 2023年4月7日(星期五) 上午6:03 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Anirudh257/strm] Accuracy recurrence issues (Issue #27) Hi @Liu-arch You can try with VIT_B_16. For HMDB, Kinetics, just train for more episodes(20K/30K, etc) and then perform tests. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Liu-arch]

Dear author, Thank you very much for your reply. After several conversations with you, I still haven't reproduced the accuracy of network architecture with resnet50 as the backbone with the optimal model you uploaded to git. So you think it is easier to reproduce the accuracy with vit as backbone? But there is no optimal model in git.  In addition, I try the VIT_B_16. just add: elif args.method == "vit":                 args.trans_linear_in_dim = 768 But without any other changes, is it possible？ Best regards.

…

------------------ 原始邮件 ------------------ 发件人: "Anirudh257/strm" ***@***.***>; 发送时间: 2023年4月7日(星期五) 上午6:03 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Anirudh257/strm] Accuracy recurrence issues (Issue #27) Hi @Liu-arch You can try with VIT_B_16. For HMDB, Kinetics, just train for more episodes(20K/30K, etc) and then perform tests. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Anirudh257]

Yes, for VIT, that's the required step.

[Liu-arch]

I have trained the UCF by VIT_B_16 for 100k. But the accuracy rate only went from 49.7% at the beginning to 73.4%.

…

------------------ 原始邮件 ------------------ 发件人: "Anirudh257/strm" ***@***.***>; 发送时间: 2023年4月12日(星期三) 上午9:29 ***@***.***>; ***@***.******@***.***>; 主题: Re: [Anirudh257/strm] Accuracy recurrence issues (Issue #27) Yes, for VIT, that's the required step. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Liu-arch]

Dear Author
Does the paper include the value of confidence for the accuracy of the four datasets?

[M77827]

Dear Author Does the paper include the value of confidence for the accuracy of the four datasets?

你好，我想问一下，你的正确率的问题解决了没有

[Anirudh257]

Hi @Liu-arch and @M77827 there is no confidence value.