stonemo/slavc

Official Codebase of "A Closer Look at Weakly-Supervised Audio-Visual Source Localization" (NeurIPS 2022)

slavc's Issues

How to get VGG-SS & Flicker 144k

Hello !
First of all, thank you for your nice work. It would be helpful for future research of VSL :)

I'd like to ask you the methods: how to get the VGG-SS.
Is it right, from vggss.csv, get frame of the timestamp (t) & get the .wav file of 10 seconds (t ~ t+10 s) ?
Also, for the flickr, I can't find even any csv files.

Would you mind if I ask you to share, how can I get the VGG-SS & Flickr with code or command, or even website ?

Confidence value for relative prediction?

Based on the provided test code, the confidence value is calculated as the average of the top 25% values of the model output before performing relative prediction. However, if you enable the relative prediction option, you would need to recalculate the prediction values. This is because the range of pixel values in the heatmap may change significantly before and after applying the Min-Max normalization. Could you please clarify if this understanding is correct?

test.py

conf_av = np.sort(scores_av.flatten())[-n//4:].mean()
conf_obj = np.sort(scores_obj.flatten())[-n//4:].mean()
conf_av_obj = np.sort(scores_av_obj.flatten())[-n//4:].mean()

if args.relative_prediction:
    pred_av = utils.normalize_img(scores_av)
    pred_obj = utils.normalize_img(scores_obj)
    pred_av_obj = utils.normalize_img(scores_av_obj)

    thr_av = np.sort(pred_av.flatten())[int(n * args.pred_size)]
    thr_obj = np.sort(pred_obj.flatten())[int(n * args.pred_size)]
    thr_av_obj = np.sort(pred_av_obj.flatten())[int(n * args.pred_size)]
else:
    pred_av = scores_av
    pred_obj = scores_obj
    pred_av_obj = scores_av_obj

    thr_av = thr_obj = thr_av_obj = args.pred_thr

evaluator_av.update(bb, gt_map, conf_av, pred_av, thr_av, name[i])
evaluator_obj.update(bb, gt_map, conf_obj, pred_obj, thr_obj, name[i])
evaluator_av_obj.update(bb, gt_map, conf_av_obj, pred_av_obj, thr_av_obj, name[i])

TRAINING ON TESTING SET?

Why there are multiple testing files (i.e., UKgZCUgOSfo_000062 from VGGSound) appear in your training set?

Help with reproducing VGGSS-EXTEND

Hi Mo,

Thanks for sharing the code and dataset. I currently working on reproducing the results for the "Extended VGG-Sound Source" dataset to verify the video data pre-processing. I failed to achieve the results listed on the README page and what I got is about 40% lower. For example:
==================== AV+OGL ====================

• AP-all@50: 0.2351479097289724
• AUC-all@50: 0.3747188832880961
• F1-all@50: 27.4 29.7 31.7 32.7 32.9 32.1 29.3 24.1 18.5 10.5 0.1
• Precision-all@50: 0.1590732842186894

What I have done is simply download the dataset from "https://drive.google.com/file/d/1dcaSyJ8xyfCPKmESDLyKucglm-1ROEOt/view" and use the following script to run the testing file:

python test.py --test_data_path "./VGGSound-test-plus-silent/" --model_dir ./ckpt/checkpoints --experiment_name vggss144k_slavc --testset vggss_plus_silent --alpha 0.9

I am not sure which parts went wrong on my side, hopefully, you could help me with that.
Looking forward to hearing from you

YH

the audio and visual processing script

I would appreciate it if u can provide the audio and visual processing script

there is a bug in train.py line 208

best_precision, best_ap, best_f1 = precision, f1, ap
the position of f1 and ap should be switched @stoneMo