This repo contains the official PyTorch implementation of the pipeline presented in I Hear Your True Colors: Image Guided Audio Generation: Paper, Project page.
We propose Im2Wav, an image guided open-domain audio generation system. Given an input image or a sequence of images, Im2Wav generates a semantically relevant sound. Im2Wav is based on two Transformer language models, that operate over a hierarchical discrete audio representation obtained from a VQ-VAE based model. We first produce a low-level audio representation using a language model. Then, we upsample the audio tokens using an additional language model to generate a high-fidelity audio sample. We use the rich semantics of a pre-trained CLIP embedding as a visual representation to condition the language model. In addition, to steer the generation process towards the conditioning image, we apply the classifier-free guidance method. Results suggest that Im2Wav significantly outperforms the evaluated baselines in both fidelity and relevance evaluation metrics. Additionally, we provide an ablation study to better assess the impact of each of the method components on overall performance. Lastly, to better evaluate image-to-audio models, we propose an out-of-domain image dataset, denoted as ImageHear. ImageHear can be used as a benchmark for evaluating future image-to-audio models.
git clone [email protected]:RoySheffer/im2wav.git
cd im2wav
pip install -r requirements.txt
Note: torch installation may depend on your cuda version. see Install torch
We provide a toy example using two videos and two single images. The same scripts can be used for the full VGGSound or any other custom dataset.
We additionally include the ImageHear dataset under Data/ImageHear/ folder.
We start by setting the directory where all scripts should be run from:
mkdir run && cd run
Next, we collect the CLIP image representations:
python ../Data/preprocess/collect_video_CLIP.py \
-videos_dir ../Data/examples/video
python ../Data/preprocess/collect_image_CLIP.py \
-path_list ../Data/ImageHear/bongo1.jpg ../Data/ImageHear/dog1.jpg
Train the models:
- Set a batch size (bs) according to your GPU size.
python ../models/train.py \
--hps=small_multi_level_vqvae \
--name=im2wav_vq \
--sample_length=65536 \
--bs=2 \
--audio_files_dir=../Data/examples/wav \
--labels=False \
--train \
--aug_shift \
--aug_blend \
python ../models/train.py \
--hps=small_multi_level_vqvae,small_labelled_prior,all_fp16,cpu_ema \
--name=im2wav_low \
--sample_length=65536 \
--n_ctx=2048 \
--bs=2 \
--aug_shift \
--aug_blend \
--audio_files_dir=../Data/examples/wav \
--labels=True \
--train \
--test \
--prior \
--restore_vqvae=logs/im2wav_vq/checkpoint_latest.pth.tar \
--levels=2 \
--level=1 \
--weight_decay=0.01 \
--save_iters=2 \
--file2CLIP=video_CLIP \
--clip_emb \
--video_clip_emb \
--class_free_guidance_prob=0.5
python ../models/train.py \
--hps=small_multi_level_vqvae,small_upsampler,all_fp16,cpu_ema \
--name=im2wav_up \
--sample_length=65536 \
--n_ctx=8192 \
--bs=2 \
--audio_files_dir=../Data/examples/wav \
--labels=True \
--train \
--test \
--aug_shift \
--aug_blend \
--save_iters=2 \
--prior \
--restore_vqvae=logs/im2wav_vq/checkpoint_latest.pth.tar \
--file2CLIP=video_CLIP \
--levels=2 \
--level=0 \
--clip_emb
After the models converge, we can use the trained models for an audio generation as follows:
python ../models/sample.py \
-bs 2 \
-experiment_name video_CLIP \
-CLIP_dir video_CLIP \
-models my_model
python ../models/sample.py \
-bs 2 \
-wav_per_object 2 \
-experiment_name image_CLIP \
-CLIP_dict image_CLIP/CLIP.pickle \
-models my_model
We start by setting the directory where the pre-trained model weights should be downloaded to:
mkdir ../pre_trained
pip install gdown
gdown 1lCrGsMXqmeKBk-3B3J2jzxNur9olWseb -O ../pre_trained/
gdown 1v9dmCwrEwkwJhbe2YF3ScM2gjVplSLzt -O ../pre_trained/
gdown 1UyNBjoxgqBYqA_aYhOu6BHYlkT4CD_M_ -O ../pre_trained/
Repeat the Video/Image condition sampling steps replacing my_model with im2wav.
If you find this implementation useful please consider citing our work:
@misc{sheffer2022i,
title={I Hear Your True Colors: Image Guided Audio Generation},
author={Roy Sheffer and Yossi Adi},
year={2022},
eprint={2211.03089},
archivePrefix={arXiv},
primaryClass={cs.SD}
}
This repository is released under the MIT license as found in the LICENSE file. Some of the code in models dir was adapted from the JukeBox repository.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent