wxg0101/unimatch

Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation

UniMatch

This codebase contains a strong re-implementation of FixMatch in the scenario of semi-supervised semantic segmentation, as well as the official PyTorch implementation of our paper:

Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation
Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, Yinghuan Shi
Preprint 2022

Abstract. In this work, we revisit the weak-to-strong consistency framework, popularized by FixMatch from semi-supervised classification, where the prediction of a weakly perturbed image serves as supervision for its strongly perturbed version. Intriguingly, we observe that such a simple pipeline already achieves competitive results against recent advanced works, when transferred to our segmentation scenario. Its success heavily relies on the manual design of strong data augmentations, however, which may be limited and inadequate to explore a broader perturbation space. Motivated by this, we propose an auxiliary feature perturbation stream as a supplement, leading to an expanded perturbation space. On the other, to sufficiently probe original image-level augmentations, we present a dual-stream perturbation technique, enabling two strong views to be simultaneously guided by a common weak view. Consequently, our overall Unified Dual-Stream Perturbations approach (UniMatch) surpasses all existing methods significantly across all evaluation protocols on the Pascal, Cityscapes, and COCO benchmarks. We also demonstrate the superiority of our method in remote sensing interpretation and medical image analysis.

Results

Pascal

Labeled images are sampled from the original high-quality training set. Results are obtained by DeepLabv3+ based on ResNet-101 with training size 321.

Method	1/115 (92)	1/57 (183)	1/28 (366)	1/14 (732)	1/7 (1464)
SupOnly	45.1	55.3	64.8	69.7	73.5
U2PL	68.0	69.2	73.7	76.2	79.5
ST++	65.2	71.0	74.6	77.3	79.1
PS-MT	65.8	69.6	76.6	78.4	80.0
UniMatch (Ours)	75.2	77.2	78.8	79.9	81.2

Cityscapes

Results are obtained by DeepLabv3+ based on ResNet-50/101. We reproduce U²PL results on ResNet-50.

ResNet-50	1/30	1/8	1/4	ResNet-101	1/16	1/8	1/4
SupOnly	56.8	70.8	73.7	SupOnly	67.9	73.5	75.4
U2PL	59.8	73.0	76.3	U2PL	74.9	76.5	78.5
UniMatch (Ours)	64.5	75.6	77.4	UniMatch (Ours)	75.7	77.3	78.7

COCO

Results are obtained by DeepLabv3+ based on Xception-65.

Method	1/512 (232)	1/256 (463)	1/128 (925)	1/64 (1849)	1/32 (3697)
SupOnly	22.9	28.0	33.6	37.8	42.2
PseudoSeg	29.8	37.1	39.1	41.8	43.6
PC2Seg	29.9	37.5	40.1	43.7	46.1
UniMatch (Ours)	32.2	40.4	46.2	48.7	51.2

Getting Started

Installation

cd UniMatch
conda create -n unimatch python=3.6.9
conda activate unimatch
pip install -r requirements.txt
pip install pip install torch==1.8.1+cu102 torchvision==0.9.1+cu102 -f https://download.pytorch.org/whl/torch_stable.html

Pretrained Backbone:

ResNet-50 | ResNet-101 | Xception-65

├── ./pretrained
    ├── resnet50.pth
    ├── resnet101.pth
    └── xception.pth

Dataset:

• Pascal: JPEGImages | SegmentationClass
• Cityscapes: leftImg8bit | gtFine
• COCO: train2017 | val2017 | masks

Please modify the dataset path in configuration files.

The groundtruth mask ids have already been pre-processed. You may use them directly.

├── [Your Pascal Path]
    ├── JPEGImages
    └── SegmentationClass
    
├── [Your Cityscapes Path]
    ├── leftImg8bit
    └── gtFine
    
├── [Your COCO Path]
    ├── train2017
    ├── val2017
    └── masks

Usage

UniMatch

# use torch.distributed.launch
sh tools/train.sh <num_gpu> <port>

# or use slurm
# sh tools/slurm_train.sh <num_gpu> <port> <partition>

To run on different labeled data partitions or different datasets, please modify:

config, labeled_id_path, unlabeled_id_path, and save_path in train.sh.

FixMatch

Modify the unimatch.py to fixmatch.py in train.sh.

Supervised Baseline

Modify the unimatch.py to supervised.py in train.sh, and double the batch_size in configuration file if you use the same number of GPUs as semi-supervised setting (no need to change lr).

Citation

We have another relevant semi-supervised semantic segmentation project: ST++

If you find these projects useful, please consider citing:

@article{unimatch,
  title={Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation},
  author={Yang, Lihe and Qi, Lei and Feng, Litong and Zhang, Wayne and Shi, Yinghuan},
  journal={arXiv:2208.09910},
  year={2022}
}

@inproceedings{st++,
  title={ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation},
  author={Yang, Lihe and Zhuo, Wei and Qi, Lei and Shi, Yinghuan and Gao, Yang},
  booktitle={CVPR},
  year={2022}
}

Acknowledgement

We thank AEL, CPS, CutMix-Seg, DeepLabv3Plus, PseudoSeg, PS-MT, SimpleBaseline, U²PL and other relevant works (see below) for their amazing open-sourced projects!

Semi-Supervised Semantic Segmentation Projects

• [2017 ICCV] Semi Supervised Semantic Segmentation Using Generative Adversarial Network
• [2019 TPAMI] Semi-Supervised Semantic Segmentation with High- and Low-level Consistency [Code]
• [2020 BMVC] Semi-supervised semantic segmentation needs strong, varied perturbations [Code]
• [2020 CVPR] Semi-Supervised Semantic Segmentation with Cross-Consistency Training [Code]
• [2020 CVPR] Semi-Supervised Semantic Image Segmentation with Self-correcting Networks
• [2020 ECCV] Guided Collaborative Training for Pixel-wise Semi-Supervised Learning [Code]
• [2020 ECCV] Semi-Supervised Segmentation based on Error-Correcting Supervision
• [2021 ICLR] PseudoSeg: Designing Pseudo Labels for Semantic Segmentation [Code]
• [2021 WACV] ClassMix: Segmentation-Based Data Augmentation for Semi-Supervised Learning [Code]
• [2021 CVPR] Semi-supervised Semantic Segmentation with Directional Context-aware Consistency [Code]
• [2021 CVPR] Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision [Code]
• [2021 ICCV] Re-distributing Biased Pseudo Labels for Semi-supervised Semantic Segmentation: A Baseline Investigation [Code]
• [2021 ICCV] Semi-Supervised Semantic Segmentation with Pixel-Level Contrastive Learning from a Class-wise Memory Bank [Code]
• [2021 ICCV] A Simple Baseline for Semi-supervised Semantic Segmentation with Strong Data Augmentation [Code]
• [2021 ICCV] C3-SemiSeg: Contrastive Semi-supervised Segmentation via Cross-set Learning and Dynamic Class-balancing [Code]
• [2021 ICCV] Pixel Contrastive-Consistent Semi-Supervised Semantic Segmentation
• [2021 NeurIPS] Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning [Code]
• [2022 PR] DMT: Dynamic Mutual Training for Semi-Supervised Learning [Code]
• [2022 AAAI] GuidedMix-Net: Semi-supervised Semantic Segmentation by Using Labeled Images as Reference
• [2022 ICLR] Bootstrapping Semantic Segmentation with Regional Contrast [Code]
• [2022 CVPR] ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation [Code]
• [2022 CVPR] Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels [Code]
• [2022 CVPR] Perturbed and Strict Mean Teachers for Semi-supervised Semantic Segmentation [Code]
• [2022 CVPR] Unbiased Subclass Regularization for Semi-Supervised Semantic Segmentation [Code]
• [2022 CVPR] Semi-supervised Semantic Segmentation with Error Localization Network [Code]
• [2022 CVPR] UCC: Uncertainty guided Cross-head Co-training for Semi-Supervised Semantic Segmentation