AI-Generated Images as Data Source: The Dawn of Synthetic Era [Paper]
This project is associated with our survey paper which comprehensively contextualizes the advance of the recent AI-Generated Images as Data Source (AIGS) and visual AIGC by formulating taxonomies according to methodologies and applications.
You are welcome to promote papers via pull request.
The process to submit a pull request:
- a. Fork the project into your own repository.
- b. Add the Title, Author, Conference, Paper link, Project link, and Code link in
README.md
with below format:
**Title**<br>
*Author*<br>
Conference
[[Paper](Paper link)]
[[Project](Project link)]
[[Code](Code link)]
[[Video](Video link)]
- c. Submit the pull request to this branch.
Machine Learning for Synthetic Data Generation: A Review
Yingzhou Lu, Minjie Shen, Huazheng Wang, Wenqi Wei
arXiv 2023 [Paper]
Synthetic Data in Human Analysis: A Survey
Indu Joshi, Marcel Grimmer, Christian Rathgeb, Christoph Busch, Francois Bremond, Antitza Dantcheva
arXiv 2022 [Paper]
A Review of Synthetic Image Data and Its Use in Computer Vision
Keith Man, Javaan Chahl
J. Imaging 2022 [Paper]
Survey on Synthetic Data Generation, Evaluation Methods and GANs
Alvaro Figueira, Bruno Vaz
Mathematics 2022 [Paper]
Methods:
Applications:
- 2D Visual Perception
- Visual Generation
- Self-supervised Learning
- 3D Visual Perception
- Other Applications
Datasets:
[BigGAN] Large Scale GAN Training for High Fidelity Natural Image Synthesis
Andrew Brock, Jeff Donahue, Karen Simonyan
ICLR 2019 [Paper]
[VQ-Diffusion] Vector Quantized Diffusion Model for Text-to-Image Synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo
CVPR 2022 [Paper][Code]
[LDM] High-Resolution Image Synthesis with Latent Diffusion Models
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer
CVPR 2022 [Paper][Code]
[Imagen] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, Mohammad Norouzi
NeurIPS 2022 [Paper][Project]
[DALL-E 2] Hierarchical Text-Conditional Image Generation with CLIP Latents
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen
arXiv 2022 [Paper]
GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, Mark Chen
ICML 2022 [Paper][Code]
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort
Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler
CVPR 2021 [Paper][Project][Code]
DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models
Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, Chunhua Shen
NeurIPS 2023 [Paper][Project][Code]
DALL-E for Detection: Language-driven Compositional Image Synthesis for Object Detection
Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Neel Joshi, Laurent Itti, Vibhav Vineet
arXiv 2022 [Paper][Code]
[StyleGAN 2]Analyzing and Improving the Image Quality of StyleGAN
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila
CVPR 2020 [Paper][Code][Video]
Data augmentation generative adversarial networks
GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
Christopher Bowles, Liang Chen, Ricardo Guerrero, Paul Bentley, Roger Gunn, Alexander Hammers, David Alexander Dickie, Maria Valdés Hernández, Joanna Wardlaw, Daniel Rueckert
arXiv 2018 [Paper]
Enhancement of Image Classification Using Transfer Learning and GAN-Based Synthetic Data Augmentation
Subhajit Chatterjee, Debapriya Hazra, Yung-Cheol Byun, Yong-Woon Kim
Mathmatics 2022 [Paper]
A data augmentation perspective on diffusion models and retrieval
Max F. Burg, Florian Wenzel, Dominik Zietlow, Max Horn, Osama Makansi, Francesco Locatello, Chris Russell
arXiv 2023 [Paper]
Effective Data Augmentation With Diffusion Models
Brandon Trabucco, Kyle Doherty, Max Gurinas, Ruslan Salakhutdinov
arXiv 2023 [Paper][Project]
Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E. Gonzalez, Trevor Darrell
arXiv 2023 [Paper][Code]
VMRF: View Matching Neural Radiance Fields
Jiahui Zhang, Fangneng Zhan, Rongliang Wu, Yingchen Yu, Wenqing Zhang, Bai Song, Xiaoqin Zhang, Shijian Lu
ACM MM 2022 [Paper]
NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields
Thomas Lips, Victor-Louis De Gusseme, Francis wyffels
ICRA 2022 [Paper][Project][Code][Video]
Neural-Sim: Learning to Generate Training Data with NeRF
Yunhao Ge, Harkirat Behl, Jiashu Xu, Suriya Gunasekar, Neel Joshi, Yale Song, Xin Wang, Laurent Itti, Vibhav Vineet
ECCV 2022 [Paper][Code]
3D Data Augmentation for Driving Scenes on Camera
Wenwen Tong, Jiangwei Xie, Tianyu Li, Hanming Deng, Xiangwei Geng, Ruoyi Zhou, Dingchen Yang, Bo Dai, Lewei Lu, Hongyang Li
arXiv 2023 [Paper]
This Dataset Does Not Exist: Training Models from Generated Images
OpenGAN: Open-Set Recognition via Open Data Generation
Shu Kong, Deva Ramanan
ICCV 2021 [Paper][Project][Code][Video]
Is synthetic data from generative models ready for image recognition?
Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, Xiaojuan Qi
ICLR 2023 [Paper][Code]
Fake it till you make it: Learning transferable representations from synthetic ImageNet clones
Training on Thin Air: Improve Image Classification with Generated Data
Not just pretty pictures: Text-to-image generators enable interpretable interventions for robust representations
Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion
Leaving reality to imagination: Robust classification via generated datasets
Image Captions are Natural Prompts for Text-to-Image Models
Diversify your vision datasets with automatic diffusion-based augmentation
Synthetic Data from Diffusion Models Improves ImageNet Classification
Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, David J. Fleet
arXiv 2023 [Paper]
Imagic: Text-Based Real Image Editing with Diffusion Models
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Fill-Up: Balancing Long-Tailed Data with Generative Models
Diffusion Models and Semi-Supervised Learners Benefit Mutually with Few Labels
Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach
Yuhua Chen, Wen Li, Xiaoran Chen, Luc Van Gool
CVPR 2019 [Paper]
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, Sanja Fidler
CVPR 2021 [Paper][Project][Code]
Repurposing GANs for One-shot Semantic Part Segmentation
Nontawat Tritrong, Pitchaporn Rewatbowornwong, Supasorn Suwajanakorn
CVPR 2021 [Paper][Project][Code]
Few-shot 3d multi-modal medical image segmentation using generative adversarial learning
Semi and weakly supervised semantic segmentation using generative adversarial network
Using GANs to augment data for cloud image segmentation task
Diffusion Models for Zero-Shot Open-Vocabulary Segmentation
Laurynas Karazija, Iro Laina, Andrea Vedaldi, Christian Rupprecht
arXiv 2023 [Paper]
Open-vocabulary Object Segmentation with Diffusion Models
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
Few-shot 3D Multi-modal Medical Image Segmentation using Generative Adversarial Learning
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations
Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, Antonio Torralba
CVPR 2022 [Paper][Project][Code]
HandsOff: Labeled Dataset Generation With No Additional Human Annotations
Austin Xu, Mariya I. Vasileva, Achal Dave, Arjun Seshadri
CVPR 2023 [Paper][Project][Code]
DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models
Dataset Diffusion: Diffusion-based Synthetic Dataset Generation for Pixel-Level Semantic Segmentation
Quang Nguyen, Truong Vu, Anh Tran, Khoi Nguyen
NeurIPS 2023 [Paper[Code]
MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation
Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, Chen Change Loy
arXiv 2023 [Paper][Code]
Self-ensembling with gan-based data augmentation for domain adaptation in semantic segmentation
Intra-Source Style Augmentation for Improved Domain Generalization
DifFSS: Diffusion Model for Few-Shot Semantic Segmentation
Weimin Tan, Siyuan Chen, Bo Yan
arXiv 2023 [Paper]
Pixel level data augmentation for semantic image segmentation using generative adversarial networks
Can segmentation models be trained with fully synthetically generated data?
Label-Efficient Semantic Segmentation with Diffusion Models
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, Artem Babenko
ICLR 2022 [Paper][Project][Code]
[ODISE] Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello
CVPR 2023 [Paper][Project][Code]
[GeoDiffusion] Integrating Geometric Control into Text-to-Image Diffusion Models for High-Quality Detection Data Generation via Text Prompt
Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, Dit-Yan Yeung
arXiv 2023 [Paper][Project]
Explore the Power of Synthetic Data on Few-shot Object Detection
Shaobo Lin, Kun Wang, Xingyu Zeng, Rui Zhao
CVPRW 2023 [Paper]
[SODGAN] Synthetic Data Supervised Salient Object Detection
Zhenyu Wu, Lin Wang, Wei Wang, Tengfei Shi, Chenglizhao Chen, Aimin Hao, Shuo Li
ACM MM 2022 [Paper]
ImaginaryNet: Learning Object Detectors without Real Images and Annotations
Minheng Ni, Zitong Huang, Kailai Feng, Wangmeng Zuo
ICLR 2023 [Paper][Code]
The Big Data Myth: Using Diffusion Models for Dataset Generation to Train Deep Detection Models
Roy Voetman, Maya Aghaei, Klaas Dijkstra
arXiv 2023 [Paper]
Re-Aging GAN: Toward Personalized Face Age Transformation
Farkhod Makhmudkhujaev, Sungeun Hong, and In Kyu Park
ICCV 2021 [Paper][Video]
Only a Matter of Style: Age Transformation Using a Style-Based Regression Model
Yuval Alaluf, Or Patashnik, Daniel Cohen-Or
SIGGRAPH 2021 [Paper][Project][Code][Video]
Production-Ready Face Re-Aging for Visual Effects
Gaspard Zoss, Prashanth Chandran, Eftychios Sifakis, Markus Gross, Paulo Gotardo, Derek Bradley
TOG 2021 [Paper][Project][Video]
Zero-1-to-3: Zero-shot One Image to 3D Object
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, Carl Vondrick
ICCV 2023 [Paper][Project][Code]
DreamBooth3D: Subject-Driven Text-to-3D Generation
Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, Varun Jampani
arXiv 2023 [Paper][Project][Video]
StyleAvatar3D: Leveraging Image-Text Diffusion Models for High-Fidelity 3D Avatar Generation
Chi Zhang, Yiwen Chen, Yijun Fu, Zhenglin Zhou, Gang YU, Billzb Wang, Bin Fu, Tao Chen, Guosheng Lin, Chunhua Shen
arXiv 2023 [Paper]
Generative Models as a Data Source for Multiview Representation Learning
Ali Jahanian, Xavier Puig, Yonglong Tian, Phillip Isola
ICLR 2022 [Paper][Project][Code][Video]
StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners
Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, Dilip Krishnan
arXiv 2023 [Paper]
Ensembling with Deep Generative Views
Lucy Chai, Jun-Yan Zhu, Eli Shechtman, Phillip Isola, Richard Zhang
CVPR 2021 [Paper][Project][Code]
DreamTeacher: Pretraining Image Backbones with Deep Generative Models
Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler
ICCV 2023 [Paper][Project]
INeRF: Inverting Neural Radiance Fields for Pose Estimation
Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Alberto Rodriguez, Phillip Isola, Tsung-Yi Lin
IROS 2021 [Paper][Project][Code][Video]
LENS: Localization enhanced by NeRF synthesis
Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, Arnaud de La Fortelle
CoRL 2021 [Paper][Video]
NeRF-Pose: A First-Reconstruct-Then-Regress Approach for Weakly-supervised 6D Object Pose Estimation
Fu Li, Hao Yu, Ivan Shugurov, Benjamin Busam, Shaowu Yang, Slobodan Ilic
arXiv 2022 [Paper]
Vision-only robot navigation in a neural radiance world
Michal Adamkiewicz, Timothy Chen, Adam Caccavale, Rachel Gardner, Preston Culbertson, Jeannette Bohg, Mac Schwager
RAL 2022 [Paper][Project][Code][Video]
Event-based Camera Tracker by ∇t NeRF
Mana Masuda, Yusuke Sekikawa, Hideo Saito
WACV 2023 [Paper]
Lift3D: Synthesize 3D Training Data by Lifting 2D GAN to 3D Generative Radiance Field
Leheng Li, Qing Lian, Luozhou Wang, Ningning Ma, Ying-Cong Chen
CVPR 2023 [Paper][Project][Code]
UniSim: A Neural Closed-Loop Sensor Simulator
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, Raquel Urtasun
CVPR 2023 [Paper][Project][Video]
MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving
Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, Hao Zhao
CICAI 2023 [Paper][Project][Code][Video]
[MedGAN] Generating Multi-label Discrete Patient Records using Generative Adversarial Networks
Edward Choi, Siddharth Biswal, Bradley Malin, Jon Duke, Walter F. Stewart, Jimeng Sun
MLHC 2017 [Paper]
CorGAN: Correlation-Capturing Convolutional Generative Adversarial Networks for Generating Synthetic Healthcare Records
Amirsina Torfi, Edward A. Fox
FLAIRS 2020 [Paper][Code]
Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, Akshay Chaudhari
NeurIPS 2022 [Paper]
Skin Lesion Classification Using GAN based Data Augmentation
Rashid Haroon, Tanveer M. Asjid, Aqeel Khan Hassan
EMBC 2019 [Paper]
GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification
Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, Hayit Greenspan
Neurocomputing 2018 [Paper]
Synthetic data augmentation using GAN for improved liver lesion classification
Maayan Frid-Adar, Eyal Klang, Michal Amitai, Jacob Goldberger, Hayit Greenspan
ISBI 2018 [Paper]
Leveraging GANs for data scarcity of COVID-19: Beyond the hype
Hazrat Ali, Christer Gronlund, Zubair Shah
CVPRW 2023 [Paper]
Benchmarking Deepart Detection
Yabin Wang, Zhiwu Huang, Xiaopeng Hong
arXiv 2023 [Paper]
Benchmarking Robustness to Text-Guided Corruptions
Mohammadreza Mofayezi, Yasamin Medghalchi
CVPRW 2023 [Paper][Code]
DiffusionDB (https://github.com/poloclub/diffusiondb)
JourneyDB (https://github.com/JourneyDB/JourneyDB)
Pick-a-Pic v2 (https://huggingface.co/datasets/yuvalkirstain/pickapic_v2)
ImageReward (https://huggingface.co/datasets/THUDM/ImageRewardDB)
HPD v2 (https://huggingface.co/datasets/xswu/human_preference_dataset)
DFFD (http://cvlab.cse.msu.edu/dffd-dataset.html)
ForgeryNet (https://yinanhe.github.io/projects/forgerynet.html)
CNNSpot (https://github.com/peterwang512/CNNDetection)
CIFAKE (https://www.kaggle.com/datasets/birdy654/cifake-real-and-ai-generated-synthetic-images)
GenImage (https://github.com/GenImage-Dataset/GenImage)
If you find our work useful for your research, please consider citing the paper:
@article{yang2023aigs,
title={AI-Generated Images as Data Source: The Dawn of Synthetic Era},
author={Zuhao Yang and Fangneng Zhan and Kunhao Liu and Muyu Xu and Shijian Lu},
journal={arXiv preprint arXiv:2310.01830},
year={2023}
}