roysheffer/im2wav

Official implementation of the pipeline presented in I hear your true colors: Image Guided Audio Generation

im2wav's Issues

Evaluation code

Hi Roy! Are you planning to release the code of how to compute those metrics reported in your paper (FAD, KL, CS, ACC)? Thanks!

Video Data Preprocessing

Hi Roy,

I am collecting the CLIP representation of VGGSound videos by running the collect_video_CLIP.py file. I had to change some codes for some reason, and now it takes a long time to calculate all the videos. I'm wondering how long will it take to collect all the CLIP features for videos in the VGGSound dataset through the original collect_video_CLIP.py ?

BTW, I just changed the way loading clip model for my server is offline.

GPU configuration

Hi Roy,

Thanks for your interesting work! I was wondering how many GPUs are required at least to perform the training (bs=2)?

train/test split

Hi,

I was wondering in your experiments did you use the exact same train/test split as specified in VGGSound's csv file or did you simply just use a train_test_split ratio to divide the whole dataset. Because when I checked the code, I saw you did the split here using the ratio. But in the paper, you mentioned that "We follow the original VGGSound train/test splits." Please correct me if I was wrong. Thank you!

Issues when sampling from pre-trained model

Hi, thx for your great work! But I have some issues when I try to sample from the pre-trained model
Errors are the following:
No such file or directory: 'logs/im2wav_vq/checkpoint_latest.pth.tar'
and
UnboundLocalError: local variable 'upsampler' referenced before assignment.

I try to sample with images and already gdown the pretrained model and collect CLIP representations

generated sound is noise

The sound obtained by following the steps in readme and using the pre trained model is noise. May I ask what is the reason for this?

The loss does't decrease

I use your model to train VQVAE, but it seems the loss does't decrease. Are there something that I need to change?

Unable to get ACC result reported in the paper

Hi Roy!
For the pretrained im2wav model, I can reproduce most of the metrics reported in your paper. However, when I use pretrained PaSST to compute the accuracy over 30 ImageHear classes for 15,446 VGGSound test set, I only get 23.6% for the reference audio and 13.4% for the generated audio by the pretrained im2wav model you provide. I've only got 2800 samples from VGGSound-test that belong to ImageHear class following the mapping provided here.

The pretrained PaSST is specified as get_model_passt(arch="passt_s_swa_p16_128_ap476", n_classes=527, s_patchout_t=40, s_patchout_f=4) following their repo.

Is it possible for you to share your code of computing ACC? Thanks!

"skvideo.io.vread" reports error

Hi Roy,

Thank you for your prompt reply! I really appreciate that you always respond quickly.

I'm collecting CLIP representation of VGGSound videos by running the collect_video_CLIP.py following your instruction. However, for video l6q7shFb8zs_000041 and EktPRpYX9KI_000038, it reports this error:

Traceback (most recent call last):
  File "/home/anaconda3/envs/myenv/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 271, in _read_frame_data
    assert len(arr) == framesize
AssertionError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "debug.py", line 194, in <module>
    check_corrupted_video()
  File "debug.py", line 173, in check_corrupted_video
    videodata = skvideo.io.vread(video)
  File "/home/anaconda3/envs/myenv/lib/python3.8/site-packages/skvideo/io/io.py", line 148, in vread
    for idx, frame in enumerate(reader.nextFrame()):
  File "/home/anaconda3/envs/myenv/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 297, in nextFrame
    yield self._readFrame()
  File "/home/anaconda3/envs/myenv/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 281, in _readFrame
    s = self._read_frame_data()
  File "/home/anaconda3/envs/myenv/lib/python3.8/site-packages/skvideo/io/ffmpeg.py", line 275, in _read_frame_data
    raise RuntimeError("%s" % (err1,))
RuntimeError

I was wondering if this happens to you as well? I obtain 199,176 videos from the VGGSound dataset and only these two cannot be processed.

Can sample length be changed for a pretrained model?

im2wav/models/sample.py

Line 215 in aff5a80

parser.add_argument("-sample_length", dest='sample_length', action='store', type=int, default=65536)

I found the sample length argument in models/sample.py, but change it does not change the generated audio length.
Is there any way to change the generated audio length for a pretrained model?
It seems that sample_length depends on self.sample_length = self.n_ctx*self.raw_to_tokens and cannot be changed.

From AudioSet to ImageHear and VGGSound classes

Hi,
Could you share a list that you matched to evaluate ACC for ImageHear and VGGSound?