cvlab-yonsei/joem

An official implementation of "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation" (ICCV 2021) in PyTorch.

Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation

This is an official implementation of the paper "Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation", accepted to ICCV2021.

For more information, please checkout the project site [website] and the paper [arXiv].

Pre-requisites

This repository uses the following libraries:

• Python (3.6)
• Pytorch (1.8.1)

Getting Started

Datasets

VOC

The structure of data path should be organized as follows:

/dataset/PASCALVOC/VOCdevkit/VOC2012/                         % Pascal VOC datasets root
/dataset/PASCALVOC/VOCdevkit/VOC2012/JPEGImages/              % Pascal VOC images
/dataset/PASCALVOC/VOCdevkit/VOC2012/SegmentationClass/       % Pascal VOC segmentation maps
/dataset/PASCALVOC/VOCdevkit/VOC2012/ImageSets/Segmentation/  % Pascal VOC splits

CONTEXT

The structure of data path should be organized as follows:

/dataset/context/                                 % Pascal CONTEXT dataset root
/dataset/context/59_labels.pth                    % Pascal CONTEXT segmentation maps
/dataset/context/pascal_context_train.txt         % Pascal CONTEXT splits
/dataset/context/pascal_context_val.txt           % Pascal CONTEXT splits
/dataset/PASCALVOC/VOCdevkit/VOC2012/JPEGImages/  % Pascal VOC images

Training

We use DeepLabV3+ with ResNet-101 as our visual encoder. Following ZS3Net, ResNet-101 is initialized with the pre-trained weights for ImageNet classification, where training samples of seen classes are used only. (weights here)

VOC

python train_pascal_zs3setting.py -c configs/config_pascal_zs3setting.json -d 0,1,2,3

• Trained visual and semantic encoder weights
  • 2 unseen classes
  • 4 unseen classes
  • 6 unseen classes
  • 8 unseen classes
  • 10 unseen classes

CONTEXT

python train_context_zs3setting.py -c configs/config_context_zs3setting.json -d 0,1,2,3

• Trained visual and semantic encoder weights
  • 2 unseen classes
  • 4 unseen classes
  • 6 unseen classes
  • 8 unseen classes
  • 10 unseen classes

Testing

VOC

python train_pascal_zs3setting.py -c configs/config_pascal_zs3setting.json -d 0,1,2,3 -r <visual encoder>.pth --test

CONTEXT

python train_pascal_zs3setting.py -c configs/config_pascal_zs3setting.json -d 0,1,2,3 -r <visual encoder>.pth --test

Acknowledgements

• This template is borrowed from pytorch-template.

joem's Issues

Code Question

Dear Authors,

For this error, could I only delete the code of self.writer.close()?

dataset split

After introducing the SBD (Semantic Boundary Dataset), what kind of split (train_split and test_split include how many images ) is adopted by this paper?

No such file or directory: 'dataset/PASCALVOC/VOCdevkit/VOC2012/ImageSets/Segmentation/train_aug.txt'

datasets

Thank you for your work～

self._cat_dir = self._base_dir / ("%d_labels.pth" % (self.n_categories))

Could you tell me how to generate the "59_labels.pth" file of the context dataset?

dataset/PASCALVOC/VOCdevkit/VOC2012/SegmentationClassAug

Could you tell me how to get the SegmentationClassAug?

train_aug.txt

Dear Authors,

When I run your code, there is an error:

FileNotFoundError: [Errno 2] No such file or directory: 'dataset/PASCALVOC/VOCdevkit/VOC2012/ImageSets/Segmentation/train_aug.txt'

Could you tell me how to get train_aug.txt?