jingzhengli/openood

Benchmarking Generalized Out-of-Distribution Detection

OpenOOD: Benchmarking Generalized OOD Detection

This repository reproduces representative methods within the Generalized Out-of-Distribution Detection Framework, aiming to make a fair comparison across methods that initially developed for anomaly detection, novelty detection, open set recognition, and out-of-distribution detection. This codebase is still under construction. Comments, issues, contributions, and collaborations are all welcomed!

Image from Fig.3 in our survey - Timeline for representative methodologies.

Updates

• 12 April, 2022: Primary release to support Full-Spectrum OOD Detection.

Get Started

To setup the environment, we use conda to manage our dependencies, and CUDA 10.1 to run our experiments.

You can specify the appropriate cudatoolkit version to install on your machine in the environment.yml file, and then run the following to create the conda environment:

conda env create -f environment.yml
conda activate openood

Datasets are provided here. Our codebase accesses the datasets from ./data/ by default.

├── ...
├── data
│   ├── images
│   ├── covid_images
│   └── imglist
├── openood
├── scripts
├── main.py
├── ...

The easiest hands-on script is to train LeNet on MNIST and evaluate its OOD or FS-OOD performance with MSP baseline.

sh scripts/0_basics/mnist_train.sh
sh scripts/c_ood/0_mnist_test_ood_msp.sh
sh scripts/c_ood/0_mnist_test_fsood_msp.sh

More tutorials are provided in our wiki pages.

Supported Methods

This part lists all the methods we include in this codebase.

Anomaly Detection

• DeepSVDD (ICML'18)
• KDAD (arXiv'20)
• CutPaste (CVPR'2021)
• PatchCore (arXiv'2021)
• DRÆM (ICCV'21)

Open Set Recognition

• OpenMax (CVPR'16)
• CROSR (CVPR'19) (@OmegaDING in progress)
• ARPL (TPAMI'21)
• OpenGAN (ICCV'21)

Out-of-Distribution Detection

No Extra Data:

• MSP (ICLR'17)
• ODIN (ICLR'18)
• MDS (NeurIPS'18)
• CONF (arXiv'18) (@JediWarriorZou in progress)
• G-ODIN (CVPR'20) (@Prophet-C in progress)
• Gram (ICML'20) (@Zzitang in progress)
• DUQ (ICML'20) (@Zzitang in progress)
• CSI (NeurIPS'20) (@Prophet-C in progress)
• EBO (NeurIPS'20)
• MOS (CVPR'21)
• MOOD (CVPR'21)
• GradNorm (NeurIPS'21) (@haoqiwang in progress)
• ReAct (NeurIPS'21)
• VOS (ICLR'22)
• VIM (CVPR'22) (@haoqiwang in progress)
• SEM (arXiv'22)

With Extra Data:

• OE (ICLR'19)
• MCD (ICCV'19)
• UDG (ICCV'21)

Citation

If you find our repository useful for your research, please consider citing our paper:

@article{yang2022openood,
    author = {Yang {\textit{et al.}}, Jingkang},
    title = {OpenOOD: Benchmarking Generalized Out-of-Distribution Detection},
    year = {2022}
}

@article{yang2022fsood,
    author = {Yang, Jingkang and Zhou, Kaiyang and Liu, Ziwei},
    title = {Full-Spectrum Out-of-Distribution Detection},
    journal={arXiv preprint arXiv:2204.05306},
    year = {2022}
}

@article{yang2021oodsurvey,
  title={Generalized Out-of-Distribution Detection: A Survey},
  author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
  journal={arXiv preprint arXiv:2110.11334},
  year={2021}
}