Is it possible to make Kimera Semantics instance-aware?

[mit-spark]

Is it possible to make Kimera Semantics instance-aware?

[ToniRV]

Hi @zhaozhongch, that's right, you can use MaskRCNN, the semantic label corresponds to the last column (number 12 above).

I guess a quick approach to get this working with MaskRCNN would be to connect Kimera-Semantics with the semantic class output of MaskRCNN converted to a grayscale image where the pixel values correspond to the semantic class id.
You may want to reserve as well a special id for non-segmented pixel.
You can also change the covariance matrix here.
I've just set it to be isotropic (constant diagonal matrix).

I had to add the decoding from colors to semantic labels because our segmentation (semantic mask) is given as RGBA colors instead of ids.
The only good thing about this approach is that the semantic images and the 3d reconstruction have the same colors.
For anything else, this is suboptimal because I am constantly mapping semantic ids to colors and vice-versa...