jfc43/robust-ood-detection

Robust Out-of-distribution Detection in Neural Networks

Robust Out-of-distribution Detection in Neural Networks

This project is for the paper: Robust Out-of-distribution Detection in Neural Networks. Some codes are from ODIN, Outlier Exposure and deep Mahalanobis detector.

Preliminaries

It is tested under Ubuntu Linux 16.04.1 and Python 3.6 environment, and requires some packages to be installed:

• PyTorch
• scipy
• numpy

Downloading in-distribution Dataset

• CIFAR: included in PyTorch.
• GTSRB: we provide scripts to download it.

Downloading out-of-distribution Datasets

• 80 Million Tiny Images
• SVHN
• Textures
• Places365
• LSUN-C
• LSUN-R
• iSUN

Overview of the Code

Running Experiments

• For SVHN dataset, you can run select_svhn_data.py to select test data.
• For GTSRB dataset, you can run prepare_data.sh to get dataset.
• robust_ood_train.py: the script used to train different models.
• eval.py: the script used to evaluate classification accuracy and robustness of models.
• eval_ood_detection.py: the script used to evaluate OOD detection performance of models.

Example

For CIFAR-10 experiments, you can run the following commands on CIFAR directory to get results.

• train an ALOE model:

python robust_ood_train.py --name ALOE --adv --ood

• train an AOE model:

python robust_ood_train.py --name AOE --adv --adv-only-in --ood

• train an ADV model:

python robust_ood_train.py --name ADV --adv

• train an OE model:

python robust_ood_train.py --name OE --ood

• train an Original model:

python robust_ood_train.py --name Original

• Evaluate classification performance of ALOE model:

python eval.py --name ALOE --adv

• Evaluate the traditional OOD detection performance of MSP and ODIN using ALOE model:

python eval_ood_detection.py --name ALOE --method msp_and_odin

• Evaluate the robust OOD detection performance of MSP and ODIN using ALOE model:

python eval_ood_detection.py --name ALOE --method msp_and_odin --adv

• Evaluate the traditional OOD detection performance of Mahalanobis using Original model:

python eval_ood_detection.py --name Original --method mahalanobis

• Evaluate the robust OOD detection performance of Mahalanobis using Original model:

python eval_ood_detection.py --name Original --method mahalanobis --adv

Citation

Please cite our work if you use the codebase:

@article{chen2020robust,
  title={Robust Out-of-distribution Detection in Neural Networks},
  author={Chen, Jiefeng and Wu, Xi and Liang, Yingyu and Jha, Somesh and others},
  journal={arXiv preprint arXiv:2003.09711},
  year={2020}
}

License

Please refer to the LICENSE.

robust-ood-detection's Issues

Why do you comment out codes which matches Algorithm 1 ?

confidence_pgd_attack.py line 92~102

       ` # one_hot_labels = torch.eye(len(outputs[0]))[y].to(CUDA_DEVICE)
        # other, _ = torch.max((1-one_hot_labels)*outputs, dim=1)
        # correct = torch.masked_select(outputs, one_hot_labels.byte())
        # loss = torch.clamp(other - correct, min=-50.0)
        # if self.in_distribution:
            # outputs = F.log_softmax(outputs, dim=1)
            # uniform_dist = torch.Tensor(x.size(0), self.num_classes).fill_((1./self.num_classes)).cuda()
            # loss = self.loss_func(outputs, uniform_dist)

        # else:
            # loss = self.loss_func(outputs)`

which matches Algorithm 1 in your paper.

Why do you comment the codes out?

Dataset (80 Million Tiny Images) not available

Hello,
The dataset used in OOD training has been withdrawed officially and deleted permanently.
see https://groups.csail.mit.edu/vision/TinyImages/
Will you provide some alternative datasets for 80 Million Tiny Images ?
Thank you!

Pretrained models

Since the 80M tiny images dataset was taken down, it's no longer possible to train the models. Is it possible to provide some pre-trained models instead?

Any specific reasons for where the data normalizer is placed

Hello.

I notice you normalize the data in the forward method instead of dataloader: https://github.com/jfc43/robust-ood-detection/blob/master/CIFAR/robust_ood_train.py#L153. Other than the gradient is not needed to be normalized in Odin and Mahalanobis, may I ask if there are other reasons to do so? For example, in some specific cases, the normalizer may behave differently when it is implemented in the dataloader than the model.forward?