taldatech/soft-intro-vae-pytorch

[CVPR 2021 Oral] Official PyTorch implementation of Soft-IntroVAE from the paper "Soft-IntroVAE: Analyzing and Improving Introspective Variational Autoencoders"

soft-intro-vae-pytorch

[CVPR 2021 Oral] Soft-IntroVAE: Analyzing and Improving Introspective Variational Autoencoders

Tal Daniel • Aviv Tamar

Official repository of the paper

CVPR 2021 Oral

Project Website • Video

Soft-IntroVAE

Soft-IntroVAE: Analyzing and Improving Introspective Variational Autoencoders
Tal Daniel, Aviv Tamar

Abstract: The recently introduced introspective variational autoencoder (IntroVAE) exhibits outstanding image generations, and allows for amortized inference using an image encoder. The main idea in IntroVAE is to train a VAE adversarially, using the VAE encoder to discriminate between generated and real data samples. However, the original IntroVAE loss function relied on a particular hinge-loss formulation that is very hard to stabilize in practice, and its theoretical convergence analysis ignored important terms in the loss. In this work, we take a step towards better understanding of the IntroVAE model, its practical implementation, and its applications. We propose the Soft-IntroVAE, a modified IntroVAE that replaces the hinge-loss terms with a smooth exponential loss on generated samples. This change significantly improves training stability, and also enables theoretical analysis of the complete algorithm. Interestingly, we show that the IntroVAE converges to a distribution that minimizes a sum of KL distance from the data distribution and an entropy term. We discuss the implications of this result, and demonstrate that it induces competitive image generation and reconstruction. Finally, we describe two applications of Soft-IntroVAE to unsupervised image translation and out-of-distribution detection, and demonstrate compelling results.

Citation

Daniel, Tal, and Aviv Tamar. "Soft-IntroVAE: Analyzing and Improving the Introspective Variational Autoencoder." arXiv preprint arXiv:2012.13253 (2020).

@InProceedings{Daniel_2021_CVPR,
author    = {Daniel, Tal and Tamar, Aviv},
title     = {Soft-IntroVAE: Analyzing and Improving the Introspective Variational Autoencoder},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month     = {June},
year      = {2021},
pages     = {4391-4400}

}

Preprint on ArXiv: 2012.13253

• soft-intro-vae-pytorch
• Soft-IntroVAE
  • Citation
  • Prerequisites
  • Repository Organization
  • Credits

Prerequisites

• For your convenience, we provide an environemnt.yml file which installs the required packages in a conda environment name torch.
  • Use the terminal or an Anaconda Prompt and run the following command conda env create -f environment.yml.
• For Style-SoftIntroVAE, more packages are required, and we provide them in the style_soft_intro_vae directory.

Library	Version
Python	3.6 (Anaconda)
torch	>= 1.2 (tested on 1.7)
torchvision	>= 0.4
matplotlib	>= 2.2.2
numpy	>= 1.17
opencv	>= 3.4.2
tqdm	>= 4.36.1
scipy	>= 1.3.1

Repository Organization

File name	Content
/soft_intro_vae	directory containing implementation for image data
/soft_intro_vae_2d	directory containing implementations for 2D datasets
/soft_intro_vae_3d	directory containing implementations for 3D point clouds data
/soft_intro_vae_bootstrap	directory containing implementation for image data using bootstrapping (using a target decoder)
/style_soft_intro_vae	directory containing implementation for image data using ALAE's style-based architecture
/soft_intro_vae_tutorials	directory containing Jupyter Noteboook tutorials for the various types of Soft-IntroVAE

Related Projects

• March 2022: augmentation-enhanced-soft-intro-vae - GitHub - using differentiable augmentations to improve image generation FID score.

Credits

• Adversarial Latent Autoencoders, Pidhorskyi et al., CVPR 2020 - Code, Paper.
• FID is calculated natively in PyTorch using Seitzer implementation - Code