jxzhangjhu/poem

Out-of-distribution Detection with Posterior Sampling

This project is for the paper: POEM: Out-of-distribution Detection with Posterior Sampling published at ICML 2022 (long talk). Some parts of the codebase are adapted from ODIN, Outlier Exposure, Deep Mahalanobis Detector and NTOM.

Required Packages

The following packages are required to be installed:

• PyTorch
• Scipy
• Numpy
• Sklearn

Our experiments are conducted on Ubuntu Linux 20.04 with Python 3.8.

Method Overview

In-distribution and Auxiliary Outlier Datasets

• In-distribution training set:
  • CIFAR: included in PyTorch.

• Auxiliary outlier training set:

  • 80 Million Tiny Images: to download 80 Million Tiny Images dataset. After downloading it, place it in this directory: datasets/unlabeled_datasets/80M_Tiny_Images

  • ImageNet-RC (Downsampled ImageNet Datasets](https://patrykchrabaszcz.github.io/Imagenet32/): we use the ImageNet64x64, which can be downloaded from ImageNet Website. After downloading it, place it in this directory: datasets/unlabeled_datasets/ImageNet.

Out-of-distribution Test Datasets

We provide links and instructions to download each dataset:

• SVHN: download it and place it in the folder of datasets/ood_datasets/svhn. Then run python select_svhn_data.py to generate test subset.
• Textures: download it and place it in the folder of datasets/ood_datasets/dtd.
• Places365: download it and place it in the folder of datasets/ood_datasets/places365/test_subset. We randomly sample 10,000 images from the original test dataset.
• LSUN-C: download it and place it in the folder of datasets/ood_datasets/LSUN.
• LSUN-R: download it and place it in the folder of datasets/ood_datasets/LSUN_resize.
• iSUN: download it and place it in the folder of datasets/ood_datasets/iSUN.

For example, run the following commands in the root directory to download LSUN-C:

cd datasets/ood_datasets
wget https://www.dropbox.com/s/fhtsw1m3qxlwj6h/LSUN.tar.gz
tar -xvzf LSUN.tar.gz

Quick Start and Model Checkpoints

Checkpoints: Checkpoints for models trained with poem can be found here.

For example, to test the model trained with POEM at 100 epochs on CIFAR-100, please download the corresponding checkpoint to ./checkpoints/CIFAR-100/POEM_Sample_1/checkpoint_100.pth.tar. You can also train from scratch with the following command:

# train an POEM model on CIFAR-10 with DenseNet
python train_poem.py --name POEM --in-dataset CIFAR-10 --auxiliary-dataset imagenet --epochs 100 --model-arch densenet

Here we provide the commands to evaluate the OOD detection performance of POEM:

# Generate evaluation results for POEM:
python get_score.py --name POEM --in-dataset CIFAR-10 --model-arch densenet --test_epochs 100

# Present OOD detection metrics based on results:
python get_results.py --name POEM  --in-dataset CIFAR-10 --test_epochs 100

For bibtex citations

If you find our work helpful for your research, please consider citing our paper:

@inproceedings{ming2022poem,
  author={Ming, Yifei  and Fan, Ying  and Li, Yixuan},
  title={POEM: Out-of-distribution Detection with Posterior Sampling},
  booktitle={International Conference on Machine Learning},
  year={2022},
  organization={PMLR}
}

Issues and further discussions

Please create an issue for code-related questions. For further discussions, free feel to drop an email at [email protected].