CSE embedding for other object categories

[zc-alexfan]

CSE embedding for other object categories

[gengshan-y]

There are two cases I can think of. For one, if the target is occluded by some other objects/background, the reconstructed shape will be (mistakenly) penalized by the silhouette reconstruction loss due to the wrong off-the-shelf segmentations. In this case, banmo would likely fail.

In the second case, if the object is out of the image boundary, banmo should work, since out-of-frame pixels are not sampled to compute losses. However in practice, this may still produce worse results since off-the-shelf segmentation method is usually not robust to partially out-of-frame objects.

[gengshan-y]

Hi, based on the results on a few categories we tried (e.g., penguin, laikago robot, eagle, hands, results on this page), densepose is not that important given good enough initial root pose.

Also note those cases do not contain challenging self-occlusions caused by articulations. I think some form of correspondence like densepose-cse is still needed for more challenging cases.

[zc-alexfan]

Thanks for the quick response!

One last question: How well does the method perform when part of the video contains heavy occlusion? For example, you have a lot of frames in which the object is occluded a little but in some frames the object is heavily occluded. I am curious in how the method handles occlusion in those heavily occluded frames.