Question about CSE and root pose

[facebookresearch]

Question about CSE and root pose

[gengshan-y]

Hi, for sequences beyond quadruped/human category (e.g., penguins, hand, eagle, laikago robot), we use the off-the-shelf quadruped CSE features. A more detailed instruction to reproduce those results are here.

You are right, one can apply banmo to categories like cups/cars without a pre-trained category-specific cse embedding, if good initial root poses are given.

[DummyNodeHead]

Thanks for your response!

So during the registration, the same pre-trained quadruped CSE model is used to get canonical embeddings of the image, and is optimized with the MLP that get the embeddings of the 3D vertex jointly?
And I'm a little confused about this line. What do csenet_feats and dps refer to respectively?

[gengshan-y]

csenet_feats is the 112x112x16 feature map, and dps is the 112x112 integer vertex index to the canonical mesh (used in CSE) for visualization only.

The script for CSE finetune is not published given all the results we've shown in the paper did not fine-tune CSE. But the functions are in this repo. To fine-tune CSE, you may need to pass --ft_cse and --nolineload --batch_size 8 --nsample 128. The later flags makes sure the whole image is loaded to compute CSE features.

[DummyNodeHead]

Thanks for your help! I get it now,