yurujaja/dginstyle-segmodel

Downstream semantic segmentation evaluation of DGInStyle.

💽 DGInStyle-SegModel: Downstream Tasks Evaluation

This is the downstream tasks evaluation component of DGInStyle. The repository hosts implementations of the off-the-shelf semantic segmentation networks trained on datasets that utilize the DGInStyle data generator.

[Project Page] | [ArXiv] | [Datasets]

By Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, Anton Obukhov

⚙️ Setup Environments

For this project, we used python/3.8.5, cuda/11.0.3.

Repository

git clone https://github.com/yurujaja/DGInStyle-SegModel.git
cd DGInStyle-SegModel

Dependencies

export SegModel_PATH=/path/to/venv/segmodel # change this
python3 -m venv ${SegModel_PATH}
source ${SegModel_PATH}/bin/activate

# use --ignore-installed to override pytorch installation
pip install --ignore-installed -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html
# requires the other packages to be installed first
pip install mmcv-full==1.3.7  

📂 Setup Datasets

In domain generalization scenario, semantic segmentation models are trained exclusively with synthetic datasets and tested on target (real-world) datasets.

Synthetic Datasets:

• DGInStyle Datasets: Please put the DGInStyle produced dataset into data/dginstyle_gen.
• GTA: Please download all image and label packages from here and extract them to data/gta. Then run the following scripts to generate rare classes related files and sample GTA subsets based on rare class sampling.

  cd tools
  python convert_datasets/gta.py data/gta --nproc 8   # generate rare classes related files
  python rcs_sample_gta.py   # sample GTA subsets
  

Real-World Datasets:

Only evaluation datasets are essential for the real-world datasets.

• Cityscapes: Please download leftImg8bit_trainvaltest.zip and gt_trainvaltest.zip from here and extract them to data/cityscapes.

• BDD100K: Please download the 10K Images and Segmentation from here and extract it to data/bdd100k.

• Mapillary: Please download the mapillary-vistas-dataset_public_v1.2.zip from here and extract it to data/mapillary.

• ACDC: Please download rgb_anon_trainvaltest.zip and gt_trainval.zip from here and extract them to data/acdc. Further, please restructure the folders from condition/split/sequence/ to split/ using the following commands:

  rsync -a data/acdc/rgb_anon/*/val/*/* data/acdc/rgb_anon/val/
  rsync -a data/acdc/gt/*/val/*/*_labelTrainIds.png data/acdc/gt/val/

• Dark Zurich: Please download the Dark_Zurich_val_anon.zip from here and extract it to data/dark_zurich.

⚖️ Training

Use the following script to automatically generate and train the configs:

python run_experiments.py --exp <ID>

The training configs, logs and checkpoints are saved in work_dirs/. More information about the available experiments and their assigned IDs, can be found in experiments.py.

🪝 Testing & Predictions

To evaluate the model saved in work_dirs/your_exp/, run the following script:

sh test.sh work_dirs/your_exp/

To specify the dataset or visualizations, please refer to tools/test.py for further instructions.

Provided checkpoints

We provide the checkpoints, training logs and evaluation files for two DG models with MiT-B5 backbone: DAFormer and HRDA.

☘️ Acknowledgements

This repository is based on the following open-source projects. We thank their authors for making the source code publicly available.

• HRDA
• DAFormer
• MMSegmentation
• SegFormer
• DACS
• SHADE

🧾 License

This project is released under the Apache License 2.0, while some specific features in this repository are with other licenses. Please refer to LICENSE for the careful check, if you are using our code for commercial matters.

📃 Citation

@misc{jia2023dginstyle,
      title={DGInStyle: Domain-Generalizable Semantic Segmentation with Image Diffusion Models and Stylized Semantic Control}, 
      author={Yuru Jia, Lukas Hoyer, Shengyu Huang, Tianfu Wang, Luc Van Gool, Konrad Schindler, Anton Obukhov},
      year={2023},
      eprint={2312.03048},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}