lovemyorange/awesome-weakly-supervised-semantic-segmentation

Awesome weakly-supervised image semantic segmentation；scribble supervision，bounding box supervision, point supervision, image-level class supervision

Awesome Weakly-supervised Semantic Segmentation

Table of Contents

• 1. Performance list
  • 1.1. Results on PASCAL VOC 2012 dataset
    • Image-level supervision with extra data
    • Image-level supervision without extra data
    • Box-level supervision
    • Scribble-level supervision
    • Point-level supervision
  • 1.2. Results on MS-COCO dataset
• 2. Paper List
  • 2.1. supervised by image tags (I)
    • 2022
    • 2021
    • 2020
    • 2019
    • 2018
    • 2017
    • 2016
  • 2.2. Supervised by bounding box (B)
  • 2.3. Supervised by scribble (S)
  • 2.4. Supervised by point (P)
• 3. Dataset
  • PASCAL VOC 2012
  • MS COCO 2014

---

Contact [email protected] if any paper is missed!

---

1. Performance list

1.1. Results on PASCAL VOC 2012 dataset

• For each method, I will provide the name of baseline in brackets if it has.
• Sup.: I-image-level class label, B-bounding box label, S-scribble label, P-point label.
• Bac. C: Method for generating pseudo label, or backbone of the classification network.
• Arc. S: backbone and method of the segmentation network.
• Pre.s : The dataset used to pre-train the segmentation network, "I" denotes ImageNet, "C" denotes COCO. Note that many works use COCO pre-trained DeepLab model but not mentioned in the paper.
• For methods that use multiple backbones, I only reports the results of ResNet101.
• "-" indicates no fully-supervised model is utilized, "?" indicates the corresponding item is not mentioned in the paper.

Image-level supervision with extra data

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	Pre.S	val	test
SEC	ECCV16	VGG16	VGG16 DeepLabv1	I	Saliency	I	50.7	51.7
DSRG (SEC)	CVPR18	VGG16	ResNet101 DeepLabv2	I	Saliency	I	61.4	63.2
AISI	ECCV18	ResNet101	ResNet101 DeepLabv2	I	Saliency	?	63.6	64.5
Ficklenet (DSRG)	CVPR19	VGG16	ResNet101 DeepLabv2	I	Saliency	I	64.9	65.3
AISI	ECCV18	ResNet101	ResNet101 DeepLabv2	I	Saliency 24KImageNet	?	64.5	65.6
OAA	ICCV19	VGG16	ResNet101 DeepLabv1	I	Saliency	I	65.2	66.4
Zhang et al.	ECCV20	ResNet50	ResNet50 DeepLabv2	I	Saliency	?	66.6	66.7
Fan et al.	ECCV20	ResNet38	ResNet101 DeepLabv1	I	Saliency	?	67.2	66.7
MCIS	ECCV20	VGG16	ResNet101 DeepLabv1	I	Saliency	?	66.2	66.9
Lee et al.	ICCV19	VGG16	ResNet101 DeepLabv2	I	Saliency Web	I	66.5	67.4
LIID	PAMI20	ResNet50	ResNet101 DeepLabv2	I	Saliency	?	66.5	67.5
MCIS	ECCV20	VGG16	ResNet101 DeepLabv1	I	Saliency Web	?	67.7	67.5
ICD	CVPR20	VGG16	ResNet101 DeepLabv1	I	Saliency	?	67.8	68.0
LIID	PAMI20	ResNet50	ResNet101 DeepLabv2	I	Saliency 24KImageNet	?	67.8	68.3
Li et al.	AAAI21	ResNet101	ResNet101 DeepLabv2	I	Saliency	?	68.2	68.5
Yao et al.	CVPR21	VGG16	ResNet101 DeepLabv2	I	Saliency	I	68.3	68.5
AuxSegNet	ICCV21	ResNet38	-	I	Saliency	?	69.0	68.6
SPML (Ficklenet)	ICLR21	VGG16	ResNet101 DeepLabv2	I	Saliency	I	69.5	71.6
Yao et al.	CVPR21	VGG16	ResNet101 DeepLabv2	I	Saliency	I+C	70.4	70.2
WegFormer	CVPR22	Deit-B	ResNet101 DeepLabv ?	I	Saliency	I	70.5	70.3
WegFormer	CVPR22	Deit-B	ResNet101 DeepLabv ?	I	Saliency	I+C	70.9	70.5
EDAM	CVPR21	ResNet38	ResNet101 DeepLabv2	I	Saliency	?	70.9	70.6
EPS	CVPR21	ResNet38	ResNet101 DeepLabv2	I	Saliency	I	70.9	70.8
EPS	CVPR21	ResNet38	ResNet101 DeepLabv1	I	Saliency	I	71.0	71.8
DRS	AAAI21	VGG16	ResNet101 DeepLabv2	I	Saliency	I+C	71.2	71.4
L2G	CVPR22	L2G	ResNet101 DeepLabv1	I	Saliency	?	72.0	73.0
L2G	CVPR22	L2G	ResNet101 DeepLabv2	I	Saliency	?	72.1	71.7

Image-level supervision without extra data

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	Pre.S	val	test
AffinityNet	CVPR18	ResNet38	ResNet38	I	-	?	61.7	63.7
ICD	CVPR20	VGG16	ResNet101 DeepLabv1	I	-	?	64.1	64.3
IRN	CVPR19	ResNet50	ResNet50 DeepLabv2	I	-	I	63.5	64.8
IAL	IJCV20	ResNet?	ResNet?	I	-	I	64.3	65.4
SSDD (PSA)	ICCV19	ResNet38	ResNet38	I	-	I	64.9	65.5
SEAM	CVPR20	ResNet38	ResNet38 DeepLabv2	I	-	I	64.5	65.7
Chang et al.	CVPR20	ResNet38	ResNet101 DeepLabv2	I	-	?	66.1	65.9
RRM	AAAI20	ResNet38	ResNet101 DeepLabv2	I	-	?	66.3	66.5
BES	ECCV20	ResNet50	ResNet101 DeepLabv2	I	-	?	65.7	66.6
AFA	CVPR22	MiT-B1	-	I	-	?	66.0	66.3
CONTA (+SEAM)	NeurIPS20	ResNet38	ResNet101 DeepLabv2	I	-	?	66.1	66.7
Ru et al.	IJCAI21	ResNet101	ResNet101 DeepLabv2	I	-	?	67.2	67.3
WSGCN (IRN)	ICME21	ResNet50	ResNet101 DeepLabv2	I	-	I	66.7	68.8
CPN	ICCV21	ResNet38	ResNet38 DeepLabv1	I	-	?	67.8	68.5
RPNet	TMM21	ResNet101	ResNet50 DeepLabv2	I	-	I	68.0	68.2
AdvCAM	CVPR21	ResNet50	ResNet101 DeepLabv2	I	-	I	68.1	68.0
PMM	ICCV21	ResNet38	ResNet38 PSPnet	I	-	?	68.5	69.0
WSGCN (IRN)	ICME21	ResNet50	ResNet101 DeepLabv2	I	-	I+C	68.7	69.3
PMM	ICCV21	Res2Net101	Res2Net101 PSPnet	I	-	?	70.0	70.5
MCTformer	CVPR22	DeiT-S	ResNet38 DeeplabV1	I	-	?	71.9	71.6

Box-level supervision

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	Pre.S	val	test
BBAM	CVPR21	?	ResNet101 DeepLabv2	B	MCG	I	73.7	73.7
WSSL	ICCV15	-	VGG16 DeepLabv1	B	-	I	60.6	62.2
Song et al.	CVPR19	-	ResNet101 DeepLabv1	B	-	I	70.2	-
SPML (Song et al.)	ICLR21	-	ResNet101 DeepLabv2	B	-	I	73.5	74.7
Oh et al.	CVPR21	ResNet101	ResNet101 DeepLabv2	B	-	I+C	74.6	76.1

Scribble-level supervision

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	Pre.S	val	test
Scribblesup				S
NormalCut	CVPR18	-	ResNet101 DeepLabv1	S	Saliency	?	74.5	-
KernelCut	ECCV18	-	ResNet101 DeepLabv1	S	-	?	75.0	-
BPG	IJCAI19	-	ResNet101 DeepLabv2	S	-	?	76.0	-
SPML (KernelCut)	ICLR21	-	ResNet101 DeepLabv2	S	-	I	76.1	-
A2GNN	TPAMI21	-	?	S	-	?	76.2	76.1
DFR	arxiv21	-	UperNet+Swin Transformer	S	22KImageNet	-	82.8	82.9

Point-level supervision

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	Pre.S	val	test
WhatsPoint	ECCV16	-	VGG16 FCN	P	Objectness	I	46.1	-
PCAM	arxiv20	ResNet50	DeepLabv3+	P	-	?	70.5	-

1.2. Results on MS-COCO dataset

Image-level supervision with extra data

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	val	test
AuxSegNet	ICCV21	ResNet38	-	I	Saliency	33.9	-
EPS	CVPR21	ResNet38	ResNet101 DeepLabv2	I	Saliency	35.7	-
L2G	CVPR22	L2G	VGG16 DeepLabv2	I	Saliency	42.7	-
L2G	CVPR22	L2G	ResNet101 DeepLabv2	I	Saliency	44.2	-

Image-level supervision without extra data

Method	Pub.	Bac. C	Arc. S	Sup.	Extra data	val	test
MCTformer	CVPR22	DeiT-S	ResNet38 DeeplabV1	I	-	42.0	-

2. Paper List

2.1. supervised by image tags (I)

2022

• MCTformer: Multi-class Token Transformer for Weakly Supervised Semantic Segmentation CVPR2022
• AFA: Learning Affinity from Attention End-to-End Weakly-Supervised Semantic Segmentation with Transformers CVPR2022
• WegFormer: WegFormer Transformers for Weakly Supervised Semantic Segmentation CVPR2022
• L2G: L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentation CVPR2022

2021

• SPML: "Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning" ICLR2021
• Li et al.: "Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation" AAAI2021
• DRS: "Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation" AAAI2021
• AdvCAM: " Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation" CVPR2021
• **Yao et al. **: "Non-Salient Region Object Mining for Weakly Supervised Semantic Segmentation" CVPR2021
• EDAM: "Embedded Discriminative Attention Mechanism for Weakly Supervised Semantic Segmentation" CVPR2021
• EPS: Railroad is not a Train Saliency as Pseudo-pixel Supervision for Weakly Supervised Semantic Segmentation CVPR2021
• WSGCN: "Weakly-Supervised Image Semantic Segmentation Using Graph Convolutional Networks" ICME2021
• PuzzleCAM: "Puzzle-CAM Improved localization via matching partial and full features" 2021arXiv
• CDA: "Context Decoupling Augmentation for Weakly Supervised Semantic Segmentation" 2021arXiv
• Ru et al.: "Learning Visual Words for Weakly-Supervised Semantic Segmentation" IJCAI2021
• AuxSegNet: "Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation" ICCV2021
• CPN: "Complementary Patch for Weakly Supervised Semantic Segmentation" ICCV2021
• PMM: "Pseudo-mask Matters in Weakly-supervised Semantic Segmentation" ICCV2021
• RPNet: "Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation" TMM2021

2020

• RRM: "Reliability Does Matter An End-to-End Weakly Supervised Semantic Segmentation Approach" AAAI2020
• IAL: "Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning" IJCV2020
• SEAM: "Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation" CVPR2020
• Chang et al.: "Weakly-Supervised Semantic Segmentation via Sub-category Exploration" CVPR2020
• ICD: "Learning Integral Objects with Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation" CVPR2020
• Fan et al.: "Employing multi-estimations for weakly-supervised semantic segmentation" ECCV2020
• MCIS: "Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation" 2020
• BES: "Weakly Supervised Semantic Segmentation with Boundary Exploration" ECCV2020
• CONTA: "Causal intervention for weakly-supervised semantic segmentation" NeurIPS2020
• Method: "Find it if You Can: End-to-End Adversarial Erasing for Weakly-Supervised Semantic Segmentation" 2020arXiv
• Zhang et al.: "Splitting vs. Merging: Mining Object Regions with Discrepancy and Intersection Loss for Weakly Supervised Semantic Segmentation" ECCV2020
• LIID "Leveraging Instance-, Image- and Dataset-Level Information for Weakly Supervised Instance Segmentation" TPAMI2020

2019

• IRN: "Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations" CVPR2019
• Ficklenet: " Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference" CVPR2019
• Lee et al.: "Frame-to-Frame Aggregation of Active Regions in Web Videos for Weakly Supervised Semantic Segmentation" ICCV2019
• OAA: "Integral Object Mining via Online Attention Accumulation" ICCV2019
• SSDD: "Self-supervised difference detection for weakly-supervised semantic segmentation" ICCV2019

2018

• DSRG: "Weakly-supervised semantic segmentation network with deep seeded region growing" CVPR2018
• AffinityNet: "Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation" CVPR2018
• GAIN: " Tell me where to look: Guided attention inference network" CVPR2018
• AISI: "Associating inter-image salient instances for weakly supervised semantic segmentation" ECCV2018
• SeeNet: "Self-Erasing Network for Integral Object Attention" NeurIPS2018
• Method: "" 2018

2017

• CrawlSeg: "Weakly Supervised Semantic Segmentation using Web-Crawled Videos" CVPR2017
• WebS-i2: "Webly supervised semantic segmentation" CVPR2017
• Oh et al.: "Exploiting saliency for object segmentation from image level labels" CVPR2017
• TPL: "Two-phase learning for weakly supervised object localization" ICCV2017

2016

• SEC: "Seed, expand and constrain: Three principles for weakly-supervised image segmentation" ECCV2016
• AF-SS: "Augmented Feedback in Semantic Segmentation under Image Level Supervision" 2016

2.2. Supervised by bounding box (B)

• WSSL: "Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation" ICCV2015
• Boxsup: "Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation" ICCV2015
• Song et al.: "Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation" CVPR2019
• BBAM: "BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation" CVPR2021
• Oh et al.: "Ba ckground-Aware Pooling and Noise-Aware Loss for Weakly-Supervised Semantic Segmentation" CVPR2021
• SPML: "Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning" ICLR2021

2.3. Supervised by scribble (S)

• Scribblesup: "Scribblesup: Scribble-supervised convolutional networks for semantic segmentation" CVPR2016
• NormalCut : "Normalized cut loss for weakly-supervised cnn segmentation" CVPR2018
• KernelCut : "On regularized losses for weakly-supervised cnn segmentation" ECCV2018
• BPG: "Boundary Perception Guidance: A Scribble-Supervised Semantic Segmentation Approach" IJCAI2019
• SPML: "Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning" ICLR2021
• DFR: "Dynamic Feature Regularized Loss for Weakly Supervised Semantic Segmentation" arxiv2021
• A2GNN: "Affinity attention graph neural network for weakly supervised semantic segmentation" TPAMI2021

2.4. Supervised by point (P)

• WhatsPoint: "What’s the Point: Semantic Segmentation with Point Supervision" ECCV2016
• PCAM: "PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision" arxiv2020

3. Dataset

PASCAL VOC 2012

@article{everingham2010pascal,
  title={The pascal visual object classes (voc) challenge},
  author={Everingham, Mark and Van Gool, Luc and Williams, Christopher KI and Winn, John and Zisserman, Andrew},
  journal={International journal of computer vision},
  volume={88},
  number={2},
  pages={303--338},
  year={2010},
  publisher={Springer}
}

MS COCO 2014

@inproceedings{lin2014microsoft,
  title={Microsoft coco: Common objects in context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
  booktitle={European conference on computer vision},
  pages={740--755},
  year={2014},
  organization={Springer}
}