stonemo/ez-vsl

Official Codebase of "Localizing Visual Sounds the Easy Way" (ECCV 2022)

ez-vsl's Issues

How to use this on our custom videos ? Please guide

How to use this on our custom videos ? Please guide

which frame of the video is extracted?

Hi, thanks for sharing the codes.

I read your paper enjoyably. It was very well-written. I have one question and appreciate your reply.

In the "Experimental Setup" section of the paper, you mentioned that a single image frame is extracted (with the 20s of audio) from each video clip. Did you extract the frame randomly? or you chose the middle frame?

Thanks :)

dataset about Flickr SoundNet

In ur setting, I wonder that how do u use the Flickr dataset, train from raw data or soundnet feature ?

ValueError: need at least one array to concatenate

File "/data2/scripts/EZ-VSL/audio_io.py", line 66, in load_audio_av
    audio = np.concatenate([af[1] for af in chunks], 1)
  File "<__array_function__ internals>", line 180, in concatenate
ValueError: need at least one array to concatenate

I realized the error comes from audio_io.py at line 59.
chunks.append((chunk_start_time, resampler.resample(frame).to_ndarray()))
should be changed to
chunks.append((chunk_start_time, resampler.resample(frame)[0].to_ndarray()))

How were the 10k and 144k data filtered from original dataset? Was it selected at random?

Questions about the dataset

Hi, I have tried many ways to download the two datasets you mentioned in the article but they all failed. Could you kindly provide a way to download both datasets?