luis-armando-perez-rey/lsbd-vae

LSBD-VAE

This is the code for the paper "Quantifying and Learning Symmetry-Based Disentanglement"[1] presented in ICML2022. Please find a tutorial for running a basic experiment within this repository in this notebook.

Requirements

This code has been tested with the following packages:

• tensorflow== 2.4.1
• tensorflow-datasets==1.2.0
• tensorflow-probability=0.12.2

Data

The data used in this paper can be downloaded from the following link. Please unzip the files within the main folder of the repository to obtain the same folder structure as detailed.

The ModelNet40 dataset we have used to produce the renders can be found here. If you use the dataset we provided please also cite their work [2].

Note: The COIL-100 dataset is downloaded using tensorflow-datasets and should download the data into the data folder.

Reproduce Results

In order to reproduce the results obtained by training the LSBD-VAE using semi-supervised training type from the main repository the following:

bash ./experiments/run_paths.py

Or to reproduce the results obtained by training with data obtained through random walks over the generative factors type:

bash ./experiments/run_semisup.py

The Python file run.py can also be used to train the LSBD-VAE with a specific dataset. Available datasets with toroidal underlying structure are arrow, pixel16, modelnet_colors. And datasets with cylindrical structure are coil100, *modelnet40_airplanes".

Type from the main repository folder:

python ./experiments/run.py --dataset NAMEDATASET --epochs 1000

Folder Structure

lsbd-vae
│   README.md
│   LICENSE   
└───lsbd_vae
│   └───data_utils
│   |   │   data_loader.py
│   |   │   factor_dataset.py
│   |   │   file_dataset.py
│   |   └───transform_image.py
│   └───metrics
│   |   └───dlsbd_metric.py
|   |
│   └───models
│   |   │   architectures.py
│   |   │   latentspace.py
│   |   │   lsbd_vae.py
│   |   └───reconstruction_losses.py
│   └───utils
│       │   architectures.py
│       │   latentspace.py
│       │   lsbd_vae.py
│       └───reconstruction_losses.py
└───experiments
|   |   run.py
|   |   run_paths.sh
|   |   run_semisup.sh
│   └───configs
│       └───paper_dataset_parameters.py
└───notebooks
|   └───basic_experiment.ipynb
└───data
    └───arrow_images
    |   └───arrow_64.png
    └───modelnet40
        │   airplane_train.h5
        └───modelnet_color_single_64_64.h5

Contact

For any questions regarding the code refer to Loek Tonnaer and Luis Armando Pérez Rey

Citation

[1] Tonnaer, L., Perez Rey, L.A., Menkovski, V., Holenderski, M., Portegies, J.W. (2022). Quantifying and Learning Linear Symmetry-Based Disentanglement. In The Thirty-ninth International Conference on Machine Learning (ICML). BibTeX

@article{tonnaer2020quantifying,
  title={Quantifying and Learning Linear Symmetry-Based Disentanglement},
  author={Tonnaer, Loek and Rey, Luis A P{\'e}rez and Menkovski, Vlado and Holenderski, Mike and Portegies, Jacobus W},
  journal={arXiv preprint arXiv:2011.06070},
  year={2020}
}

[2] Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015). 3d shapenets: A deep representation for volumetric shapes In Proceedings of the IEEE conference on computer vision and pattern recognition.

lsbd-vae's Issues

Fix bug with RandomWalkIdentities. Batch size needs to be a multiple of the fixed factors

If I have 8 3D models with 6 possible colors I should get 48 identities times 6 random walk steps I should get 288 images. When using batch_size of 5 I am getting 270 images. I don't have problem if the batch size is 6.

EDIT: Turned out it was not a bug. It was behaving as expected. I will leave this issue because it is important to know that the data loader is fine.

Add class to load dataset with images saved as factor1_factor2_factor3..._factorN.png

Add class to load a tf.data.Dataset from a dataset of multiple factors. Similar to RandomWalkFactor dataset but each datapoint provided as output by dataset is of shape (num_identities, nfactors1, nfactors2, ... nfactorsN, *image_shape) where num_identities represents a certain object identity which can be defined by the user based on certain factors e.g. object shape and color.

Such dataset is important to evaluate DLSBD metric since this metric requires embeddings to be organized as (n_identity, nfactors1, ..., nfactorsN, latent_dim).

Update README file

• Dependencies
• Explanation on how to run code
• Images
• Citation
• Add contacts

Create jupyter notebooks for training and evaluation of methods

• Create jupyter notebook that trains semi-supervised
• Create jupyter notebook that trains supervised with paths
• Create jupyter notebook that shows how to calculate metric
  Important! Don't forget to push notebooks to the repository without the outputs!!!!

Add code for creating batches of data from images and factor values

Add code that can take images with shape (n, h, w, d) and factors with shape (n, n_factors) and produce batches of images that can be used by an LSBDVAE

Add code to reproduce semi-supervised training

• Loading of data
• Training of models
• Reproduce arrow dataset experiments
• Reproduce pixel dataset experiments
• Reproduce airplane dataset experiments

Add RandomWalkIdentities subclass

RandomWalkIdentities subclass that can create a tf.data.Dataset that outputs a random walk for each element in a list of combinations of, e.g. colors and 3D models corresponding to the identities of the objects.

Add LSBD-VAE with triplet loss

Add subclass of SupervisedLSBDVAE where embeddings of an Euclidean latent space get pushed together based on certain object identity.

Add code to reproduce path experiments

• Loading of data
• Training of models
• Reproduce arrow dataset experiments
• Reproduce pixel dataset experiments
• Reproduce airplane dataset experiments
• Reproduce modelnet dataset experiments
• Reproduce coil dataset experiments

Reconstruction loss for Bernoulli distributed data doesn't use logits for more stable training

To increase training stability we should:

• Remove the use of sigmoid in the decoding neural networks
• Change the Bernoulli loss to use logits

Add RandomFactor data class

Add dataset class that takes all possible combinations of the changing factors and creates randomly grouped data with the changing factors for each identity.

Remove architectures from other project Luis

There are some calls to architectures that we don't use in the LSBDVAE project

Merge and remove branch Loek

Update reconstruction loss

Add LSBD metric to code and document code.

• Add LSBD-VAE metric module torus
• Add LSBD-VAE metric module cylinder
• Add documentation to LSBD-VAE metric functions
• Add testing scripts for LSBD-VAE metric torus
• Add testing scripts for LSBD-VAE metric cylinder