jingkang50/oodsurvey

The Official Repository for "Generalized OOD Detection: A Survey"

Generalized Out-of-Distribution Detection: A Survey

   

1. Overview

This repository is with our survey paper:

Title: Generalized Out-of-Distribution Detection: A Survey
Authors: Jingkang Yang¹, Kaiyang Zhou¹, Yixuan Li², Ziwei Liu¹
Institutions: ¹MMLab@NTU, ²University of Wisconsin-Madison.

This survey comprehensively reviews the similar topics of outlier detection (OD), anomaly detection (AD), novelty detection (ND), open set recognition (OSR), and out-of-distribution (OOD) detection, extensively compares their commomality and differences, and eventually unifies them under a big umbrella of "generalized OOD detection" framework.

We hope that this survey can help readers and participants better understand the open-world field centered on OOD detection. At the same time, it urges future work to learn, compare, and develop ideas and methods from the broader scope of generalized OOD detection, with clear problem definition and proper benchmarking.

We prepare this repository for the following two reasons:

1. We consider it an awesome list to easily access the references mentioned in the paper Table 1. We also believe this list will continue to include more promising works as new works appear. Please feel free to nominate good related works with Pull Requests.
2. We hope this repository becomes a discussion panel for readers to ask questions, raise concerns, and make constructive comments for the broad generalized OOD detection field. Please feel free to post your ideas in the Issues.

We are also planning to build an evaluation benchmark to compare representative generalized OOD detection methods from every sub-task to further unify the field. The work will be collaborated with SenseTime EIG Research, which recently have many full-time researcher openings for this benchmarking project and other OOD-related research. Check their Recruitment Info for more information.

Fig.1.1: Two kinds of distribution shift to assist better understanding of our framework.	Fig.1.2: Taxonomy diagram of generalized OOD detection framework.

2. Taxonomy

• 2.1 Anomaly Detection
  • 2.1.1 Sensory Anomaly Detection
  • 2.1.2 Semantic Anomaly Detection
• 2.2 Novelty Detection
  • 2.2.1 One-Class Novelty Detection
  • 2.2.2 Multi-Class Novelty Detection
• 2.3 Open Set Recognition
• 2.4 Out-of-Distribution Detection
• 2.5 Outlier Detection
• 2.6 Discussion

3. Anomaly Detection & One-Class Novelty Detection

• 3.1 Density-based Methods
  • 3.1.1 Classic Density Estimation
  • 3.1.2 NN-based Density Estimation
  • 3.1.3 Energy-based Model
  • 3.1.4 Frequency-based Model
• 3.2 Reconstruction-based Methods
  • 3.2.1 Sparse Representation Methods
  • 3.2.2 Reconstruction-Error Methods
• 3.3 Classification-based Methods
  • 3.3.1 One-Class Classification
  • 3.3.2 Positive-Unlabeled Learning
  • 3.3.3 Self-Supervised Learning
• 3.4 Distance-based Methods
• 3.5 Gradient-based Methods
• 3.6 Discussion and Theoretical Analysis

4. Multi-Class Novelty Detection & Open Set Recognition

• 4.1 Classfication-based Methods
  • 4.1.1 EVT-based Calibration
  • 4.1.2 EVT-free Calibration
  • 4.1.3 Unknown Class Generation
  • 4.1.4 Label Space Redesign
• 4.2 Distance-based Methods
• 4.3 Reconstruction-based Methods
  • 4.3.1 Sparse Representation Methods
  • 4.3.2 Reconstruction-Error Methods

5. Out-of-Distribution Detection

• 5.1 Classfication-based Methods
  • 5.1.0 Baseline
  • 5.1.1 Output-based Methods
    • 5.1.1.1 Post-hoc Detection
    • 5.1.1.2 Confidence Enhancement
    • 5.1.1.3 Outlier Exposure
  • 5.1.2 OOD Data Generation
  • 5.1.3 Gradient-based Methods
  • 5.1.4 Bayesian Models
  • 5.1.5 Large-scale OOD Detection
• 5.2 Density-based Methods
• 5.3 Distance-based Methods

6. Outlier Detection

• 6.1 Density-based Methods
• 6.2 Distance-based Methods
  • 6.2.1 Cluster-based Methods
  • 6.2.2 Graph-based Methods
• 6.3 Classification-based Methods

7. Challenges and Future Direction

• 7.1 Challenges
  • 7.1.1 Proper Evaluation and Benchmarking
  • 7.1.2 Outlier-free OOD Detection
  • 7.1.3 Tradeoff Between Classification and OOD Detection
  • 7.1.4 Real-world Benchmarks and Evaluations
• 7.2 Future Directions
  • 7.2.1 Methodologies across Sub-tasks
  • 7.2.2 OOD Detection & OOD Generalization
  • 7.2.3 OOD Detection & Open-Set Noisy Labels
  • 7.2.4 Theoretical Analysis

8. Conclusion

In this survey, we comprehensively review five topics: AD, ND, OSR, OOD detection, and OD, and unify them as a framework of generalized OOD detection. By articulating the motivations and definitions of each sub-task, we encourage follow-up works to accurately locate their target problems and find the most suitable benchmarks. By sorting out the methodologies for each sub-task, we hope that readers can easily grasp the mainstream methods, identify suitable baselines, and contribute future solutions in light of existing ones. By providing insights, challenges, and future directions, we hope that future works will pay more attention to the existing problems and explore more interactions across other tasks within or even outside the scope of generalized OOD detection.

Citation

If you find our survey and repository useful for your research, please consider citing our paper:

@article{yang2021oodsurvey,
  title={Generalized Out-of-Distribution Detection: A Survey},
  author={Yang, Jingkang and Zhou, Kaiyang and Li, Yixuan and Liu, Ziwei},
  journal={arXiv preprint arXiv:2110.11334},
  year={2021}
}

Acknowledgements

This repository is created and maintained by Jingkang Yang and Peng Wenxuan from NTU; Kunyuan Ding, Zixu Song, Pengyun Wang, Zitang Zhou, and Dejian Zou from BUPT.

oodsurvey's Issues

OOD/OSR of mlti-label recognition

Hi.Amazing survey,but i notice for recognition task mlti-label recognition is missed,wonder is there any progress or amazing work for OOD/OSR of mlti-label recognition?

Question: I wonder why the "Representation Enhancement" could be categorized as one of the models under the "Density Estimation with Deep Generative Models"?

Hi,

Really thanks for this super comprehensive survey, which really helped me get to the basic level of this area with a complete framework, and saved a lot of my time. I understood most of the paper but still got a few questions. I am curious about this question mostly: I wonder why the "Representation Enhancement" could be categorized as one of the models under the "Density Estimation with Deep Generative Models", what is the reason that leads you to put it as d part under 3.1.2?

the topic "novety class discovery" is missing.

nice work. I notice that the topic about "novety class discovery" is missing. That is, sometimes it is not enough for the ood samples
are just recognized as class "ood", they also need to be specifically classified. I recommend that the authors could consider to add this part.