monniert/dti-clustering

(NeurIPS 2020 oral) Code for "Deep Transformation-Invariant Clustering" paper

DTI-Clustering

Pytorch implementation of "Deep Transformation-Invariant Clustering" paper (accepted at NeurIPS 2020 as an oral)

Check out our paper and webpage for details!

If you find this code useful, don't forget to star the repo ⭐ and cite the paper:

@inproceedings{monnier2020dticlustering,
  title={{Deep Transformation-Invariant Clustering}},
  author={Monnier, Tom and Groueix, Thibault and Aubry, Mathieu},
  booktitle={NeurIPS},
  year={2020},
}

Installation 👷

1. Create conda environment

conda env create -f environment.yml
conda activate dti-clustering

Optional: some monitoring routines are implemented, you can use them by specifying the visdom port in the config file. You will need to install visdom from source beforehand

git clone https://github.com/facebookresearch/visdom
cd visdom && pip install -e .

2. Download non-torchvision datasets

Following script will download affNIST-test and FRGC datasets, as well as our unfiltered Instagram collections associated to #santaphoto and #weddingkiss:

./download_data.sh

NB: it may happen that gdown hangs, if so you can download them by hand with following gdrive links, then unzip and move them to the datasets folder:

• affNist source link
• FRGC source link
• IG santaphoto gdrive link
• IG weddingkiss gdrive link

How to use 🚀

1. Launch a training

cuda=gpu_id config=filename.yml tag=run_tag ./pipeline.sh

where:

• gpu_id is a target cuda device id,
• filename.yml is a YAML config located in configs folder,
• run_tag is a tag for the experiment.

Results are saved at runs/${DATASET}/${DATE}_${run_tag} where DATASET is the dataset name specified in filename.yml and DATE is the current date in mmdd format. Some training visual results like prototype evolutions and transformation prediction examples will be saved. Here is an example of learned MNIST prototypes and transformation predictions for a given query image:

Prototypes

Transformation predictions

2. Reproduce our quantitative results on MNIST-test (10 runs)

cuda=gpu_id config=mnist_test.yml tag=dtikmeans ./multi_pipeline.sh

Switch the model name to dtigmm in the config file to reproduce results for DTI GMM. Available configs are:

• affnist_test.yml
• fashion_mnist.yml
• frgc.yml
• mnist.yml
• mnist_1k.yml
• mnist_color.yml
• mnist_test.yml
• svhn.yml
• usps.yml

3. Reproduce our qualitative results on Instagram collections

1. (skip if you already downloaded data using script above) Create a santaphoto dataset by running process_insta_santa.sh script. It can take a while to scrape the 10k posts from Instagram.
2. Launch training with cuda=gpu_id config=instagram.yml tag=santaphoto ./pipeline.sh

That's it! You can apply the process to other IG hashtags like #trevifountain or #weddingkiss and discover prototypes similar to:

4. Reproduce our qualitative results on MegaDepth

1. You need to download desired landmarks from the original MegaDepth project webpage, e.g. Florence Cathedral
2. Move images to a datasets/megadepth/firenze/train folder
3. Launch training with cuda=gpu_id config=megadepth.yml tag=firenze ./pipeline.sh

You should end up with 20 learned prototypes and random sample examples in each cluster. To assess the quality of clustering, you can visualized for each cluster, the prototype, random samples and transformed prototypes like:

Further information

If you like this project, please check out related works from our group:

Follow-ups

• Monnier et al. - Unsupervised Layered Image Decomposition into Object Prototypes (ICCV 2021)

Previous works on deep transformations

• Deprelle et al. - Learning elementary structures for 3D shape generation and matching (NeurIPS 2019)
• Groueix et al. - 3D-CODED: 3D Correspondences by Deep Deformation (ECCV 2018)
• Groueix et al. - AtlasNet: A Papier-Mache Approach to Learning 3D Surface Generation (CVPR 2018)

dti-clustering's Issues

HTTP Error 403: Forbidden - When trying to run the training part (on mnist.yml file)

I am running the entire experiment on Google Colab (after setting up all the required environments).
On running the line:

! source activate dtic && cuda=gpu_id config=mnist.yml tag=run_tag ./pipeline.sh

I get the following error:

[2021-03-08 16:43:28] Trainer initialisation: run directory is /content/dti-clustering/runs/mnist/0308_run_tag
[2021-03-08 16:43:28] Config /content/dti-clustering/configs/mnist.yml copied to run directory
[2021-03-08 16:43:28] Using cpu device, nb_device is None
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /content/dti-clustering/datasets/MNIST/raw/train-images-idx3-ubyte.gz
0it [00:00, ?it/s]Traceback (most recent call last):
File "src/trainer.py", line 617, in
trainer = Trainer(config, run_dir, seed=seed)
File "/content/dti-clustering/src/utils/init.py", line 102, in wrapper
return f(*args, **kw)
File "src/trainer.py", line 72, in init
train_dataset = get_dataset(self.dataset_name)("train", **self.dataset_kwargs)
File "/content/dti-clustering/src/dataset/torchvision.py", line 40, in init
dataset = self.dataset_class(root=self.root, transform=self.transform, download=True, **kwargs)
File "/usr/local/envs/dtic/lib/python3.7/site-packages/torchvision/datasets/mnist.py", line 70, in init
self.download()
File "/usr/local/envs/dtic/lib/python3.7/site-packages/torchvision/datasets/mnist.py", line 137, in download
download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
File "/usr/local/envs/dtic/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 249, in download_and_extract_archive
download_url(url, download_root, filename, md5)
File "/usr/local/envs/dtic/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 83, in download_url
raise e
File "/usr/local/envs/dtic/lib/python3.7/site-packages/torchvision/datasets/utils.py", line 71, in download_url
reporthook=gen_bar_updater()
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 247, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 222, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 531, in open
response = meth(req, response)
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 641, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 569, in error
return self._call_chain(*args)
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 503, in _call_chain
result = func(*args)
File "/usr/local/envs/dtic/lib/python3.7/urllib/request.py", line 649, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
0it [00:00, ?it/s]

Performance on other datasets

Thank you for sharing codes, could you show the performance on other datasets such as CIFAR10 and STL?

MegaDepth train images for Florence Cathedral

Hi! I am wondering if you can provide the directory name (number) for Florence Cathedral training images.
After downloading and running "create_MD_train_val.py" to create trainset, I got error,

FileNotFoundError: [Errno 2] No such file or directory: '/phoenix/S6/zl548/MegaDpeth_code//final_list/train_val_list/landscape/imgs_MD.p'

So, please let me know the name of the directory for Florence Cathedral folders in MegaDepth files
or give some recommendations for dealing this issue. ^^