Question about the reported results

[4-0-4-notfound]

Question about the reported results

[LiheYoung]

Hi,

You provide the two results under the setting of 92 labeled images, where the improvement of our ST++ over ST is 3.9% ( 65.2% vs 61.3%), which is a much larger margin than 0.47.

As you point out, the performance gain of our ST++ over ST is less obvious in several settings. You may try other networks (e.g., DeepLabv3+) or settings. Generally speaking, more labeled images, less obvious improvement.

For a stable evaluation, you can even run ST++ and ST on all three data splits under each setting. Limited by time, we only run the first data split several times under certain settings to make sure stability.

[4-0-4-notfound]

However, in more labeled images setting like 732 labeled images in VOC 2012, the gap is even more unignorable comparing to ST++‘s improvement over ST.

Furthermore, in 1464 labeled images setting, ST++'s performance is even worse than ST occasionally.

[LiheYoung]

In the label-scarce setting, when you merely leverage the labeled images, the performance variance can be large. However, when incorporating abundant unlabeled images, the performance will be more stable and harder to improve. So I don't think it is fair or necessary to compare the variance of SupOnly results with the gain of our ST++ over ST.

As for your 1464 results, you may run multiple times and report the average. I want to point out that our ST already surpasses the fully-supervised performance, so the gain of our ST++ is less obvious. But I do observe consistent improvements in my experiments.