tianyu0207/pebal

[ECCV'22 Oral] Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes

pebal's Issues

Cannot reproduce the paper result and checkpoint result

Hello , Thanks for your work

I followed the pebal config and trained using batch size 16 (as the paper) and 40 epochs
However I cannot reproduce the paper result and check point result
Is there any tips for reproducing the high AUPRC on the FS LS dataset??
Or is there any way to reproduce the result that has similar performance with checkpoint
Could you help??

Thanks in advance
Config :
import os
import numpy
from easydict import EasyDict

C = EasyDict()
config = C
cfg = C

C.seed = 666

"""Root Directory Config"""
C.repo_name = 'pebal'
C.root_dir = os.path.realpath(".")

"""Data Dir and Weight Dir"""
C.city_root_path = '/home/numb7315/PEBAL/code/dataset/city_scape' # path/to/your/city_scape
C.coco_root_path = '/home/numb7315/PEBAL/code/dataset/coco' # path/to/your/coco
C.fishy_root_path = '/home/numb7315/PEBAL/code/dataset/fishyscapes' # path/to/your/fishy

C.pebal_weight_path = os.path.join(C.root_dir, 'ckpts', 'pebal', 'best_ad_ckpt.pth')
C.pretrained_weight_path = os.path.join(C.root_dir, 'ckpts', 'pretrained_ckpts', 'cityscapes_best.pth')

"""Network Config"""
C.fix_bias = True
C.bn_eps = 1e-5
C.bn_momentum = 0.1

"""Image Config"""
C.num_classes = 19+1 # NOTE: 1 more channel for gambler loss
C.image_mean = numpy.array([0.485, 0.456, 0.406]) # 0.485, 0.456, 0.406
C.image_std = numpy.array([0.229, 0.224, 0.225])

C.image_height = 900
C.image_width = 900

C.num_train_imgs = 2975
C.num_eval_imgs = 500

"""Train Config"""
C.lr = 1e-5
C.batch_size = 8
C.lr_power = 0.9
C.momentum = 0.9
C.weight_decay = 1e-4

C.nepochs = 40
C.niters_per_epoch = C.num_train_imgs // C.batch_size
C.num_workers = 8
C.train_scale_array = [0.5, 0.75, 1, 1.5, 1.75, 2.0]
C.void_number = 5
C.warm_up_epoch = 0

"""Eval Config"""
C.eval_epoch = 1
C.eval_stride_rate = 2 / 3
C.eval_scale_array = [1, ] # 0.5, 0.75, 1, 1.5, 1.75
C.eval_flip = False
C.eval_base_size = 800
C.eval_crop_size = 800

"""Display Config"""
C.record_info_iter = 20
C.display_iter = 50

Your project [work_space] name
C.proj_name = "OoD_Segmentation"

Your current experiment name
C.experiment_name = "pebal_baseline"

half pretrained_ckpts-loader upload images; loss upload every iteration
C.upload_image_step = [0, int((C.num_train_imgs / C.batch_size) / 2)]

False for debug; True for visualize
C.wandb_online = True

"""Save Config"""
C.saved_dir = os.path.join(C.root_dir, 'ckpts', C.experiment_name)

if not os.path.exists(C.saved_dir):
os.mkdir(C.saved_dir)

Result:

wandb: Fishyscapes_ls_auprc 0.44404
wandb: Fishyscapes_ls_auroc 0.98498
wandb: Fishyscapes_ls_fpr95 0.06383
wandb: Fishyscapes_static_auprc 0.89534
wandb: Fishyscapes_static_auroc 0.9951
wandb: Fishyscapes_static_fpr95 0.0199
wandb: energy_loss 0.03873
wandb: gambler_loss 0.17864
wandb: global_step 39

Some question about paper

Hi,

So, after carefully read the paper, I am not sure if I got the paper correctly.

The paper proposed a loss which is helpful to find out abnormal class.

steps:
1. formulate D_in and D_out. D_in should not overlap with D_out
2. Train the model with D_IN
3. Retrain model with D_OUT
Question: what do you mean by fine-tune only the final classification block using the loss in (2)

Thanks!

How to generate the city_scape dataset?

Hello and thanks for the awesome work!

I am trying to prepare the data in the format you mentioned in installation page, however I don't understand how to get your structure.
When I download the city_scape gtFine dataset I have the following data structure:

city_scapes
└── gtFine
    ├── test
    │   ├── berlin
    │   ├── bielefeld
    │   ├── bonn
    │   ├── leverkusen
    │   ├── mainz
    │   └── munich
    ├── train
    │   ├── aachen
    │   ├── bochum
    │   ├── bremen
    │   ├── cologne
    │   ├── darmstadt
    │   ├── dusseldorf
    │   ├── erfurt
    │   ├── hamburg
    │   ├── hanover
    │   ├── jena
    │   ├── krefeld
    │   ├── monchengladbach
    │   ├── strasbourg
    │   ├── stuttgart
    │   ├── tubingen
    │   ├── ulm
    │   ├── weimar
    │   └── zurich
    └── val
        ├── frankfurt
        ├── lindau
        └── munster

I succesfully ran the processing step in here.

I also had a look at the synboost preprocessed data and also at the issue #13 but I still don't get how to easily convert the structure to yours.

Also, I saw on the here that you provided the annotation folder. Should I use these files or generate them, if so how?

Hopefully I am not missing something obvious!

Best,
Aldi

Is there a link to download the best checkpoints?

Hi!
I am trying to find out some hard cases in RoadAnomaly. It will be very helpful to provide the best checkpoints.
Could you provide a link to download them?
Thanks!

inlier vs outlier

Hi I am bit confused with the definition of inlier and outlier classes described in your paper.

e.g. For cityscapes, what are the inlier classes and outlier classes?

Thanks

Eq.3 logits or probabilities

In Eq.3, the PAL loss uses logits (f_theta) instead of probabilities (p_theta). When using logits, it seems that the parts inside the parenthesis can become negative. When the parts inside the parenthesis becomes negative, the log operation cannot be performed. When using probabilities, however, the parts inside the parenthesis will always be positive. Why Eq.3 uses logits instead of probabilities?

Typo of "WideResNet34"

Hi, thanks for your excellent work! I am wondering if this is a typo in your paper of the pretrained backbone. In your paper, it is claimed that the seg model is DeepLabv3+ with a WideResNet34 backbone. However, both the baseline "meta-OOD" and your code show "WideResNet38" is actually used. So I guess there must be a typo in your paper.

Thanks again for your awesome work!

Confused about the PAL loss on the outlier pixels.

Hi, thanks for your excellent work! I am a little confused about the PAL loss, especially on how it works on the outlier pixels in D_out.

Main Confusion:
In eq(3) of the paper, as I understand, the label y_w should be Y+1 for outlier pixels. Then the loss term inside the log becomes f(Y+1;x)_w + f(Y+1;x)_w / a_w. However, when I checked the code, I found that the real implementation of the PAL loss on the outlier pixels (inside log) seems to be a sum of the interior class probabilities and f(Y+1;x)_w / a_w. As can be seen in reserve_boosting_energy = torch.add(true_pred, reservation.unsqueeze(1))[mask.unsqueeze(1). repeat(1, 19, 1, 1)].log(). Would you kindly explain this? (p.s. I have also read the paper [33]. It seems that they do not consider an outlier dataset during the training, therefore, I can not find helpful information.)

Others:
I find the code conducts the 'log' operation twice. (Besides the above code, another 'log' operation is conducted on reserve_boosting_energy = torch.clamp(reserve_boosting_energy, min=1e-7).log()). It is really confusing.
Also, there seems to be a minor typo in eq(3) where a softmax operation is lacking on the logit.

Your response would be fully appreciated!

How to get result of Initial Prediction and Final Prediction?

Hi, thank you for your great work.

I've gone through your repo but still had no idea how to get result of Initial Prediction (output of DeepWV3Plus segmentation) and Final Prediction (combined of Init Prediction and Anomaly Prediction ?). Could you please suggest me how to get them out? Thank you so much.

Best regards,

What pixels are predicted as the extra class Y+1?

Hi,

I am interested in which kind of pixels are predicted as the extra class after training? In the paper, it appears me that the anomaly prediction is mainly based on the energy score. Thus, I am very curious about the extra class the model has learned to predict.

Also, how the deep baseline gambler is implemented in the paper to get the numbers in Tab. 1 and 3? Does it use the outlier dataset? what is the reward if not having the energy function?

Thanks! Looking forward to your reply.

Some more guide to data preperation?

Hi, thank you for your great work.

I've been trying to train your model but currently I'm finding it impossible to prepare the datasets (cityscapes, coco) in the form that is required for your code.

If possible can you please provide the preprocessed datasets directly?

Thanks a bunch.

Training results

@tianyu0207 Your work is excellent. When I tried to reproduce the results using the cityscapes_best.pth pre-trained weights that you provided for training, I found that very good performance was already achieved on the validation set in the first epoch. Is this normal?

I also tried other semantic segmentation models, such as DDRNet. However, when using it for training, the initial results were very poor, and it took a complete 40 epochs or more of training to improve. On the other hand, the initial results of the DeepLabv3+ model you used were already very good, and there wasn't a significant improvement after training compared to the first epoch. Is this reasonable?

PAL loss question

Hi

First of all, great paper.

In the paper:

"The minimisation of the loss in (3) will abstain from classifying outlier
pixels into one of the inlier classes, where a pixel is estimated to be an outlier
with aω."

"ℓpal learns to abstain the prediction of outlier pixels from x out containing OE objects (i.e.,
cyan coloured masks) and calibrate the logit of inlier classes (i.e., reduction of the inlier logits) from both inlier image x
in and outlier image xout"

I dont quite understand these two statements.

1. Min_Pal is achieved when a pixel is estimated to be an outlier class
2. if 1 is correct, how ℓpal is helping to abstain the prediction of outlier pixel. From what I understood, we are trying to minimize the PAL loss when model training, minimizing is pushing a pixel to be estimated to be an outlier.

This is confusing to me. Hope somebody could explain.

My key point is your statements is reasonable and correct. I just didnt get the how it works.

Can you offer the fishyscapes and coco (for outlier exposures) datasets baidu netdisk for training?

hello author,
Can you offer the fishyscapes and coco (for outlier exposures) datasets baidu netdisk for training?
I don't know how to download the fishyscapes datasets from the website and what is necessary for training.
Thank you very much!

数据集对不上

说明文档上说的数据集目录结构，和提供的数据集及其构造方法，均对不上，尝试多种方法，一直在报路径不对的错。关于数据集构建这一块，可以提供更详细的步骤和做法吗？

data structure

Thank you for your great job and congratulations for ECCV oral!
I follow your guidance in another issue and download the processed data in synboost but the data structure is different from your code.
For example, when I run python code/test.py and set C.city_root_path as '/GPFS/public/cityscapes_processed', I got an error:
FileNotFoundError: [Errno 2] No such file or directory: '/GPFS/public/cityscapes_processed/images/city_gt_fine/val' .
You know because these are the proceed files, it is harder for us to understand the data structure.
Could you tell us how to reorganize the files downloaded from synboost. Or provide us the correct data structure like this:

./PEBAL
  ├── cityscapes_processed                               # dataset files
  │   ├── original
  │   ├── labels                 
  │   └── semantic

Thank you again and hope for your response!

dataset structure

Hi there,

What is the structure of the datasets used in the code?

• /home/yu/yu_ssd/fishyscapes/
• /home/yu/anomaly_segment/
• /home/yu/yu_ssd/road_anomaly
• /home/yuyuan/work_space/fs_lost_and_found/

Inference on input image

Thank you for your great work.

Is there a way to run inference on single image and get output mask visualization?

Best regards,

RuntimeError: CUDA error: device-side assert triggered

what problem for it?

train log is:
/pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [20430,0,0], thread: [34,0,0] Assertion idx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. epoch (0) | gambler_loss: 10.962 energy_loss: 7.645 : 50%|███████████████████████▌ | 1/2 [00:21<00:21, 21.57s/it] Traceback (most recent call last): File "code/main.py", line 151, in <module> main(-1, 1, config=config, args=args) File "code/main.py", line 107, in main optimizer=optimizer) File "/home/rhdai/workspace/code/PEBAL/code/engine/trainer.py", line 51, in train loss = self.loss1(pred=in_logits, targets=in_target, wrong_sample=False) File "/home/rhdai/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/rhdai/workspace/code/PEBAL/code/losses.py", line 103, in forward gambler_loss = gambler_loss[~mask].log() RuntimeError: CUDA error: device-side assert triggered wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [52,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [53,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [54,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [55,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [56,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"failed. /pytorch/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [19955,0,0], thread: [57,0,0] Assertionidx_dim >= 0 && idx_dim < index_size && "index out of bounds"f ailed. wandb: wandb: wandb: Run history: wandb: energy_loss ▁ wandb: gambler_loss ▁ wandb: global_step ▁▁ wandb: wandb: Run summary: wandb: energy_loss 7.64503 wandb: gambler_loss 10.96237 wandb: global_step 0 wandb: wandb: Synced your_pebal_exp: https://wandb.ai/runist/OoD_Segmentation/runs/359y48y4

training script

Hi,

any plan for releasing your training code?

BR,

Question for Data prepration

Could you provide the detail branch and folder path for the data ??

Also, there is lost and found dataloader class but there is no information about these data

I cannot understand the cityscape dataset preparation and also the fishyscape dataset

And the given processed cityscape and original cityscaped data , they should be in the same branch?

they have different name and branch ( cityscapes_processed , cityscapes) and also the processed data has no name left in file

but original data has the left.jpg then, original cityscape data is not necessary ?

However, dataloader load the named left.jpg I have trouble in data preparation . Could you help ?

Thanks in advance

can't get the good results on the FS datasets

Hi!

I tried to run the test.py , but I can't get the good results on the FS dataset. Here are the results I got:

[pebal][INFO] validating cityscapes dataset ... 
 labeled: 1966976, correct: 1933282: 100%|█████████████████████████████████████████████████████████████| 500/500 [33:56<00:00,  4.07s/it]
[pebal][CRITICAL] current mIoU is 0.895736885447597, mAcc is 0.979122587176078 

100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [02:17<00:00,  1.37s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_ls: 0.5020263181424963                                                                                    
[pebal][CRITICAL] AUPRC score for Fishyscapes_ls: 0.0025106950112928753 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_ls: 0.7372275091455253 

[pebal][INFO] validating Fishyscapes_static dataset ... 
100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:39<00:00,  1.31s/it]
[pebal][CRITICAL] AUROC score for Fishyscapes_static: 0.4023022576107206 
[pebal][CRITICAL] AUPRC score for Fishyscapes_static: 0.012901811279743608 
[pebal][CRITICAL] FPR@TPR95 for Fishyscapes_static: 0.8040210492683544 

The FS datasets is downloaded from the link you provided (synboost GitHub), and the corresponding folder structure is:

fishyscapes/
├── LostAndFound
│ ├── entropy
│ ├── labels
│ ├── labels_with_ROI
│ ├── logit_distance
│ ├── mae_features
│ ├── original
│ ├── semantic
│ └── synthesis
└── Static
     ├── entropy
     ├── labels
     ├── labels_with_ROI
     ├── logit_distance
     ├── mae_features
     ├── original
     └── semantic

In Static and LostAndFound, I use the images in the labels folder and the original folder.

Here is the code in my modified data_loader.py:

def __init__(self, split='Static', root="", transform=None):
    """Load all filenames."""
    self.transform = transform
    self.root = root
    self.split = split  # ['Static', 'LostAndFound']
    self.images = []  # list of all raw input images
    self.targets = []  # list of all ground truth TrainIds images
    # filenames = os.listdir(os.path.join(root, self.split, 'images'))    
    filenames = os.listdir(os.path.join(root, self.split, 'original'))    # final folder
    root = os.path.join(root, self.split)
    for filename in filenames:
        if os.path.splitext(filename)[1] == '.png':
            f_name = os.path.splitext(filename)[0]
            # ======= old ======= #
            # filename_base_img = os.path.join("images", f_name)
            # filename_base_labels = os.path.join("labels", f_name.replace("leftImg8bit", "labels"))
            # ======= final ======= #
            filename_base_img = os.path.join("original", f_name)
            filename_base_labels = os.path.join("labels", f_name)
            # ========================== #
            self.images.append(os.path.join(root, filename_base_img + '.png'))
            self.targets.append(os.path.join(root, filename_base_labels + '.png'))
            self.images = sorted(self.images)
            self.targets = sorted(self.targets)

Thanks! Looking forward to your reply.

Train Custom Data

hi author,
thank you for your great work and open source.
I want to train model in my own custom data, for example: this data have only three inlier classes, and we want use this model to determine outlier. Can you support some training tutorial for custom data?not coco or cityscapes.
Thanks.
.

cannot reproduce the result on "Lost And Found testing set"

Hi,

I use the provided model and could successfully reproduce the results on "cityscapes valid set", "Fishyscapes ls & Static", and "Road Anomaly".

However, when it comes to "Lost And Found testing set", I get the following result:
AUROC score for LostAndFound: 0.9688811261819444
AUPRC score for LostAndFound: 0.7222531717939171
FPR@TPR95 for LostAndFound: 0.1460461138717795

I have exactly the same data structure. I am not sure where goes wrong. Would you have any suggestions?

Thanks!

Fishyscape test result

Thanks for your great work.

I can get the validation result as following instruction.

Fishy scape test result (benchmark result from https://fishyscapes.com/results)

can be achieved by the same checkpoint that you provided ??

Or other checkpoints or something else ??

why larger aω means lower inlier free energy

Thanks for sharing your great work!

In the paper you mentioned that: aω = (−Eθ (x)ω )^2,
but this is same as aω = Eθ (x)ω ^2, but larger aω means lower inlier free energy only happens when Eθ (x)ω <0, where can we confirm this condition ?

Training pebal with custom backbone model and dataset

Hi there! Congrats on such excellent results and thank you so much for you inspiring work!

After reading through the paper and trying to implement PEBAL with the code provided by you (again, thank you so much for opensourcing the code), I am now experimenting the use of custom backbone model and dataset on the performance of PEBAL since the paper discussed the influence of segmentation models. There are a few points that are not mentioned in previous issues and I would like to confirm them before continue working on my experiments.

1. The layers to finetune.
   As stated in the paper, only "the last block of a segmentation model" needs to be fine-tuned. From my understanding, the reason is that there is an extra channel in these layers and the parameter weights for that channel needs to be obtained through fine-tuning. Therefore, only the layers with the the constructed extra channel need to keep requires_grad as True for their parameters. I was wondering if my understanding regarding what layers to freeze and what layers to train is correct.

2. How to pick m_in & m_out values
   I suppose the m_in and m_out need to be repicked as well. I am curious as to what criteria are involved in the selection for m_in and m_out values. In the reply #19 (comment), it is mentioned that the energy is constrained by the two values -12 and -6. During my experiments, I am observing great overlapping between the inlier and outlier energy, I was wondering if you could explain a bit more on how they are constrained and provide some advice on choosing the two values.

3. Other parameters
   I was wondering if you can provide some advice on how to select other hyperparameters such as β_1, β_2 and λ as well?

4. Loss
   When trying to replicate the results from the PEBAL paper, it has caught my attention that both the energy loss and the gambler loss are not converging and keep on fluctuating. However, from the metrics such as AUROC, the performance has actually improved. I was wondering if this behavior is normal for PEBAL.

Thanks!

Which deeplabv3+ checkpoint should I download?

before_start.md states that it is necessary to download a deeplabv3+ checkpoint. However, there are many deeplabv3+ checkpoints. Which one should I download?

How to continue my training

I parse my trainnig and I notice there is a parser in the engine file containing continue args, how can I use this?

EBM loss on D_out formula seems wrong

In the paper, the energy based model to train on D_out is shown as bellow:

However, this formula seems to treat all pixels ω for an image in D_out as ood pixels, but D_out is like this: only the cyan part is ood, the rest are inliers, they should be trained using L_ebm^in instead, right?

what is ood_seg_train2017

what is ood_seg_train2017

Having issues in model training

Hi, thanks for the fantastic work. But I have some issues in model training, I prepared the dataset according to the instructions, but got this error:
0%| | 0/818 [00:02<?, ?it/s]
Traceback (most recent call last):
File "/home/xxxx/xxxx/xxxx/PEBAL/code/main.py", line 158, in
main(-1, 1, config=config, args=args)
File "/home/xxxx/xxxx/xxxx/PEBAL/code/main.py", line 113, in main
trainer.train(model=model, epoch=curr_epoch, train_sampler=train_sampler, train_loader=train_loader,
File "/home/xxxx/xxxx/xxxx/PEBAL/code/engine/trainer.py", line 47, in train
target = minibatch['label'].cuda(non_blocking=True)
KeyError: 'label'

Process finished with exit code 1

---

I print out the minibatch and found it doesn't have the label field:
{'data': tensor([[[[-0.0116, -0.0116, -0.0116, ..., -1.6727, -1.6727, -1.6555],
[-0.0116, -0.0116, -0.0116, ..., -1.6555, -1.6727, -1.6555],
[-0.0287, -0.0116, -0.0116, ..., -1.6555, -1.6555, -1.6384],
...,
...,
[-0.5844, -0.6367, -0.6367, ..., -0.6193, -0.6193, -0.6018],
[-0.5844, -0.6193, -0.6193, ..., -0.6367, -0.6193, -0.6018],
[-0.5844, -0.5844, -0.6018, ..., -0.6367, -0.6367, -0.6193]]]]), 'fn': ['dusseldorf_000206_000019', 'dusseldorf_000144_000019', 'tubingen_000062_000019', 'hamburg_000000_089696', 'dusseldorf_000035_000019_unknown_unknown', 'erfurt_000072_000019', 'hamburg_000000_049558', 'bremen_000118_000019'], 'n': tensor([6545, 6545, 6545, 6545, 6545, 6545, 6545, 6545]), 'is_ood': tensor([False, False, False, False, False, False, False, False])}

---

What files should be include in the annotation files? I put the original files from cityscape and the generated labeltrainids files there. Is there anything I did wrong?

Thanks!
Thanks!

离群点的像素是通过阈值筛选出来的吗？

感谢作者开源的工作。
我看代码中没有对离群点的像素进行特定筛选，只提供了验证精度的代码和可视化的代码。plt.imshow()这个代码的归一化确实符合视觉上的预期，但我想知道你们是否有一个标准的阈值或是其他方法，可以单独获取离群点的像素。
另外我正在做的工作是基于对比检测的方式来识别离群点，不同的是，我训练和测试都采用Y+1类，最后通过Argmax来获取异物。我正在尝试将你们的loss添加到变化检测中，请问作者对这个loss的添加有什么建议么？

Doubt about Fishyscapes

Hi,
thanks for your work! One quick question about the Fishyscapes dataset. Fishyscapes images are only for validation/testing, while training happens on Cityscapes? Or did I get anything wrong?