varuy322/awesome-diffusionmodels-paper

A curasted list of papers with the topic of Diffusion Models for Multi-Modal

Awesome papers for Diffusion Models for Multi-Modal Generation

A curated list of Diffusion Models for Mutli-Modal Generation with awesome resources (paper, code, application, review, survey, etc.), which aims to comprehensively and systematically summarize the recent advances to the best of our knowledge.

Table of contents

• 1. Text-to-Image Generation
• 2. Scene Graph-to-Image Generation
• 3. Text-to-3D Generation
• 4. Text-to-Motion Generation
• 5. Text-to-Video Generation
• 6. Text-to-Audio Generation

1. Text-to-Image Generation

• A survey of vision-language pre-trained models. (2022). Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. 2022. paper
• Imagic: Text-Based Real Image Editing with Diffusion Models. (2022) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2022. paper
• Least squares estimation without priors or supervision. Neural computation 23, 2 (2011), 374–420 Martin Raphan and Eero P Simoncelli. 2011. paper
• UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model on a Single Image. SIGGRAPH 2023 (2022) Dani Valevski, Matan Kalman, Yossi Matias, and Yaniv Leviathan. 2022. paper (2022).
• Hierarchical text-conditional image generation with clip latents. NiPs (2022). Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. paper
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. ICML. 16784–16804. Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. 2022. paper
• Vector quantized diffusion model for text-to-image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition. 10696–10706. Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. 2022. paper
• MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation. Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Björn Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, Andres Felipe Cruz-Salinas, Patrick Schramowski, Kristian Kersting, Samuel Weinbach. NeurIPS 2023. paper blog
• Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model. Shin-I Cheng*, Yu-Jie Chen*, Wei-Chen Chiu, Hung-Yu Tseng, Hsin-Ying Lee. WACV 2023. paper blog code
• SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model. CVPR 2023. paper
• Versatile Diffusion: Text, Images and Variations All in One Diffusion Model. Xingqian Xu, Zhangyang Wang, Eric Zhang, Kai Wang, Humphrey Shi. ICCV 2023. paper code
• ObjectStitch: Object Compositing With Diffusion Model.Yizhi Song, Zhifei Zhang, Zhe Lin, Scott Cohen, Brian Price, Jianming Zhang, Soo Ye Kim, Daniel Aliaga. CVPR 2023. paper
• One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale. Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, Jun Zhu. CVPR 2023. paper code
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability. Runhui Huang, Jianhua Han, Guansong Lu, Xiaodan Liang, Yihan Zeng, Wei Zhang, Hang Xu. ICCV 2023. paper
• An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions. Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, Sang-Woo Lee. NIPs 2022. paper

2. Scene Graph-to-Image Generation

• Roei Herzig, Amir Bar, Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. 2020. Learning canonical representations for scene graph to image generation. ECCV. 210–227. paper
• Justin Johnson, Agrim Gupta, and Li Fei-Fei. 2018. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1219–1228 papercode
• Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. 2019. Pastegan: A semi-parametric method to generate image from scene graph. Advances in NeurIPS 32 (2019). paper code
• Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In CVPR. 10684–10695 paper code
• Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion　models. In IEEE Conference on Computer Vision and Pattern Recognition. 10684–10695 paper code

3. Text-to-3D Generation

• Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. 2022. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988(2022) paper review demo
• Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2022. Magic3D: High-Resolution Text-to-3D Content Creation. in CVPR (2023). paper blog
• Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao. 2022. Dream3D: Zero-Shot Text-to-3D Synthesis Using 3D Shape Prior and Text-to-Image Diffusion Models. CVPR (2023). paper code

4. Text-to-Motion Generation

• Jihoon Kim, Jiseob Kim, and Sungjoon Choi. 2022. Flame: Free-form language-based motion synthesis & editing. in AAAI (2023). paper code demo
• Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. 2022. Human motion diffusion model. ICLR (2023). paper code demo
• Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022. Motiondiffuse: Text-driven human motion generation with diffusion model. in arXiv (2022). paper code demo

5. Text-to-Video Generation

• Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. 2022. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022). paper
• Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022. Make-a-video: Text-to-video generation without text-video data. ICLR Poster (2022) paper demo
• Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. 2023. FateZero: Fusing Attentions for Zero-shot Text-based Video Editing. in ICCV Oral (2023). paper code
• Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. 2022. Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation. in ICCV (2023). paper code
• Video diffusion models. Ho, Jonathan and Salimans, Tim and Gritsenko, Alexey and Chan, William and Norouzi, Mohammad and Fleet, David J. in arXiv (2022). demo
• MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation. Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, Baining Guo; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. paper code

6. Text-to-Audio Generation.

• Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. 2021. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML. 8599–8608. paper code paper
• Lawrence R Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2 (1989), 257–286. paper
• Sungwon Kim, Heeseung Kim, and Sungroh Yoon. 2022. Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data. PMLR (2023). paper demo hf
• Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. 2022. Diffsound: Discrete Diffusion Model for Text-to-sound Generation. arXiv preprint arXiv:2207.09983 (2022). paper code
• Jaesung Tae, Hyeongju Kim, and Taesu Kim. 2021. EdiTTS: Score-based Editing for Controllable Text-to-Speech. arXiv preprint arXiv:2110.02584 (2021) paper code demo
• Rongjie Huang, Zhou Zhao, Huadai Liu, Jinglin Liu, Chenye Cui, and Yi Ren. 2022. ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech. in MM (2022). paper code